feat: use a job scheduler to launch workflows when the system is healthy (SBP-421)

[marius-mather]

Pull Request

Summary

SBP-421 move workflow launches to the job scheduler, so that:

• The current /launch endpoint only queues the job in the database, and does not launch directly on Seqera (credits are still deducted at this point)
• The job scheduler checks for pending jobs every 5 minutes, and attempts to launch pending ones
• The job scheduler checks the system health before attempting to submit, to ensure Seqera is ready to accept workflows

As part of this, some refactors have been needed:

• Move caching of system status into the database, so it can be accessed easily by both web service and scheduler

How to Test

Run uv run pytest - large change touching a lot of the code, but I have tried to keep/update unit tests wherever possible

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added or updated documentation where necessary
• I have run linting and unit tests locally
• The code follows the project's style guidelines

[vtnphan]

Thanks for making this big job! I've tested for 4 runs and got them working now. Got few comments regarding the job details for query job using seqera_run_id, and the bulk-prediction workflow missing

[vtnphan]

Looks good to me now, tested locally. I think we should have a docs where to see workflow submission error/debugging as we can't see the error log from UI anymore?