Problem: No persistent Quantum State

[maennchen]

Description

Currently quantum does not persist any state. Therefore if an application is restarted, all dynamic jobs are lost. Also in the time that has passed since the restart happened no jobs has been executed and there is no way to let them redo the lost time.

Minimal Requirements

I don''t think that Quantum should choose a storage backend and implement it since this heavily depends on the application where it is integrated in. I propose that we just develop protocols for such a storage and let the implementation to the user of the library. (Where this person can decide if he wants to use redis / postgres / mnesia / whatever to ensure persistence)

• Extendable Storage Protocol
• Config Option to set a Storage Backend Module
• Config Option wether to redo not ran jobs between app stop & start

Implementation

I heavily recommend waiting for PR #230 before implementing any changes.

The storage would have to be inserted in two places:

• JobBroadcaster
  • read on init
    • check if there is a storage present
    • when present read storage instead of config
      • (I see no good way to both read the config and the storage. Since config jobs can be deleted at runtime, there is no safe way to merge storage & config.)
  • write on change
    • after init write config jobs
    • :add
    • :delete
    • :change_state
    • :delete_all
• ExecutionBroadcaster
  • read on init
    • read last execution date from storage if present and config redo is active
  • write on :execute

PR

Since this is a fairly big implementation, we'd be happy to receive a PR. If this is a popular request please leave a 👍 so that we see that this is important to our users and implement it ourselves.

[maennchen]

Replacement of #238

[egeersoz]

Currently there are two popular job scheduling libraries for Elixir: Exq and Quantum-Elixir.

Exq is Redis-backed, so jobs don't get lost if application restarts. However, it cannot be configured to run jobs at regular intervals.

Quantum-Elixir can run jobs at regular intervals, but it has no persistent state for Quantums.

This is a predicament: robust, "real-life" job scheduling functionality is currently split between these two libraries, with neither one providing the full functionality.

That's why I think this feature should be high priority.

[Yamilquery]

I think that you could use ETS and PersistentEts as a storage state system, instead of leave to the user that implementation.

[maennchen]

@Yamilquery I'm aware that there are many options for the storage backend. This is not really a complex feature but requires a decent amount of work. If you'd like to tackle this, you're very welcome to do provide a PR.

As mentioned in the requirements section please do not implement to one specific storage backend but rather write a protocol so that other implementations can be built.

[pyatkov]

Why do you guys insist on this generic mul;ti-backend implementation as if your really reaaly know that it really is a critical for some use cases, while the whole persistence capability (which obviously is critical) is missing. Why not just start off with a feasible DETS/Mnesia implementation and go ahead with it?
I would love to get my hands on implementing a PR, but for a feasible DETS/Mnesia implementation.
I know configurable backend is a common and usefull option, but don't you see it's blocking people from working on a PR, because it's making it too big. Don;t be so demanding when there is nothing in place, ease your requirements and the solution will come to you.

[maennchen]

@pyatkov

My plan was to implement a behaviour for the adapter and one implementation. (not sure yet what I would pick first) The reason this is not long done is that I currently lack the time to do all this.

If you'd like to volunteer, I could define the behaviour / config etc. in a PR and you could provide the first implementation of the storage. Does that sound like an option to you?

Don;t be so demanding

I don't think that defining a behaviour is very demanding. This library is fairly popular by now and I don't see lowering standards as an option.

[maennchen]

@pyatkov
The basis is done on the branch storage. Do you want to implement a storage provider now?

https://github.com/c-rack/quantum-elixir/blob/storage/lib/quantum/storage/adapter.ex

Please tell me if there is something wrong with the interface or if you have ideas for improvements.

[maennchen]

@c-rack Would you prefer if storage providers live in their own package or if we manage them here?

Edit: Depending on the adapter type this could introduce many new optional dependencies like redix etc. which could collide with the installers setup. I think it would be better if the adapters were in its own package.

[pyatkov]

@maennchen
I agree. Just wanted to stresss that sometimes requirements should be less demanding, considering first step is always hard to take.
In fact Im already working on this PR. I was thinking through the options and by now I sticked to the behaviour, so I'm working on a generic imlementation.
I will be open to various comments on something like a code review phase, so details could be adjusted accordingly in the process.
Right now I'm thinking about job execution tracking and the redo feature as well.
Seems like the job should have something like a last_executed_at field holding the date and time the job was last executed at.
Also it seems job.name is a unique field and could be used as a PK or similar for storing the job.
Hopefuly, I will have some draft implementation on hand in the meantime, so it would be a concrete basis to elaborate on this PR/issue.