Let Lyra forget

[WULCAN]

Description

Store.persistToDisk always merges initialState into the payload to be persisted and initialState is not filtered in any way from the previous persisted file.

If a key ever existed somewhere in the structure of Store, it will forever be retained in the persisted file and reloaded. For example, if you ever had the instance configured with a certain project, but then removed that project, the data related to that project will survive forever in this persisted file, be reloaded again on every restart. This might eventually cause performance problems and for some data, it might not be acceptable to retain data for a long time.

Design a system so that irrelevant data can be forgotten eventually.

Estimated size M

I think this issue is Medium because we have no design at all yet, the problem space seems quite large but actual implementation is out-of-scope for this issue. We will create a new issue for the implementation.

Open questions

• For some data, it might be nice to not have it deleted immediately.
• For some data, we might give an admin direct control of the contents, without resorting to editing the file between shutdown and restart.
• How much data is too much data?