Huge amount of rentained messages and restarting

[skinkie]

The last action upon a restart is saving retained messages to /var/lib/flashmq/retained.db, would it be possible to do this in a different way, since it takes quite long to write it. In addition to it, it looks that flashmq needs a whopping 6GB of RAM, and my expectation is that this is mostly due to these retained messages, any chance to offload this?

[halfgaar]

For both issues, it would probably require integration with some kind of storage specific software, like something SQL based (Sqlite maybe). And then in such a way so that it's not a build-time requirement. Also, some kind of change tracking would have to be implemented.

I'll think about some solutions, perhaps intermediate ones as well.

[ben-auo]

+1

Persistence like https://nanomq.io/docs/en/latest/rule/config-file.html#data-persistence-with-sqlite with the option to republish a topic would be awesome

[bigsmoke]

Just a note: there are a couple of new contenders for in-process databases that might be worth checking out before defaulting to SQLite:

• Column-oriented: https://duckdb.org/
• Key-value: https://rocksdb.org/
• Key-value: https://github.com/google/leveldb
• http://www.lmdb.tech/doc/ is a bit older, but might also be quite sufficient and more performant than SQLite, with the downside of the datafiles being platform-dependent (which may not be a big deal with FlashMQ's singular Linux x64 focus).

This is a preliminary list, without having, for instance, done any license compatibility analysis.

[skinkie]

I am kind of a heavy DuckDB user ;-) The problem with DuckDB is that it is in principle single threaded access. That is not a problem if the inter process communication is handled within the application and the database access can be handled with just one thread. That having said, DuckDB has transparant compression support, which is pretty neat.

[ben-auo]

worth checking out before defaulting to SQLite

Oh yes, definitely. That was just a quick example that came to mind. Some other ideas:

• https://github.com/rqlite/rqlite
• https://github.com/Expensify/Bedrock
• https://github.com/apple/foundationdb/
• https://github.com/bloomberg/comdb2

[bigsmoke]

I've spoke with @halfgaar about this a couple of times recently, and persistence options for retained messages are something that he very much wants to maul over for some time before committing to any particular approach. This is something to get right off the bat, especially because one of the architectural goals for FlashMQ is to keep the number of dependencies to an absolute minimum. (Recently, we even got rid of an external DocBook→groff tool dependency for generating the man pages.)

[halfgaar]

The discussion about different databases aside, it was actually easy to write and read the file in batches and prevent the high memory spike.

Are you able to compile yourself and test it?

https://github.com/halfgaar/FlashMQ/tree/TEST-fix-memory-spike-retained-saving

[skinkie]

Maybe not in production ;-) but certainly doable :-)

[halfgaar]

Do you have an ETA on test results? I would like to make a release soon, with this feature included.

I tested saving and loading with millions of retained messages. It helped the memory use on both ends.

[skinkie]

I cannot commit on it this week, but somewhere within the next two weeks is possible. Please release if you can't wait. Just too much stuff I need to fix on my own development projects.

[halfgaar]

The memory improvement is included in 1.19.0, just released.

You also felt it took too long to save these messages? In my tests with 10538851 messages, I needed double the amount of RAM you reported, so I'm guessing I had more messages, but saving the file (to a slow encrypted magnetic disk) 'only' took about 50 seconds:

[2025-01-04 16:24:59.832] [NOTICE] [main] Saving retained messages to '/mnt/drive/tmp/FlashMQState2/retained.db'
[2025-01-04 16:25:48.833] [NOTICE] [main] Done saving 10538851 retained messages.

Note that saving the file on quit goes faster than saving in the background (because it backs-off on holding locks).

I'm curious how long a restart cycle takes for you. Can you output the relevant log lines for it?

Even when implementing some kind of incremental saving ability with DB back-end, loading the entire DB on start is still wait time. I'm inclined to close the ticket (and the options discussed) at least for now, because the amount of work may not justify the gains.