MongoDB LUT - Cache Performance issues

[rich-graylog]

Summary from a customer call:

MongoDB Lookup Table Download and Cache - Like with CSV files, pull the mongodb lookup table on an interval (like every hour) and cache it for in-memory lookups. They had a CIDR table that was 1916 rows and even with a long running cache it saturated a 10GbE link because CIDR queries are filtered at Graylog and not MongoDB so it had to pull the full table each time there was a cache miss.

If possible, we should look at ways of optimizing performance when the cache is not updated and we need to retrieve from Mongo again.