Go back to running bake as individual tasks rather than using rose bunch

[SGallagherMet]

What problem does your feature request solve?

Running a full length trials for a single UKV parameter causes cset bake to run out of wall clock, with a message that the out of memory (OOM) killer has been run multiple times.

There is also the issue of user control. When tasks are grouped in to a bunch it is difficult to see which specific task has caused an issue.

Describe the solution you'd like

I'd like to see the workflow to run the plotting in individual cylc tasks. This will allow more memory/wall clock for individual tasks and will allow users to intervene if a specific task fails.

Describe alternatives you've considered

Changing the number of tasks that rose bunch runs at once. This would ultimately turn a bunch of parallel tasks in to sequential tasks and significantly slow down cset.

[jfrost-mo]

How are you running the longer UKV trials? If it is a long series of short forecasts, then we would only need to change the case aggregation baking.

Baking multiple recipes within a single job leads to a significant speedup over having many hundreds of cylc tasks per cycle, and also helps the cylc gui run smoothly. It is also a precursor to a longer aim of daskifying all of the operators, so we can then use a single python process to bake multiple recipes, which should lead to an even greater speedup/reduced memory usage.

When I've got a bit more time I want to investigate why our memory usage is so high for aggregation. In theory we are collapsing everything down, but I suspect we are realizing all the data before we aggregate it somewhere.

[ukmo-huw-lewis]

To flag I can sympathise with Stephen's initial question here.

Useful Issue to document here a general question I had of James on related theme. Think basically boiled down to "is it possible to have a more task-oriented mode for running workflow e.g. for purposes of debugging". Appreciate answer at time was no in interests of keeping things simpler (which I appreciate).

To illustrate as user I ran a job in recent days with a) released version - think still prior to single job baking? and then b) based on main code (i.e. ran as single task).

Some tasks failed (e.g. variable not found). In the run task-by-task approach, I could get easy handle on how much succeeded (most of the tasks) and ability to dig into failed tasks. In the single job approach, the single job shows as failed (obviously), but....

1. I struggled (maybe user error) to pinpoint what had gone awry and
2. the single job retriggered (when most tasks had succeeded).
3. It required me to be 'brave' as a user to set failed task to 'set' in order that workflow kicks through to finish_website to then be able to see the 90%+ of succeeded task outputs.

Some questions:

• is it worth disabling the auto retrigger from single task mode, given risks just running bulk of succeeding tasks twice if anything fails?
• as a medium-step to individual tasks, how easy/efficient might it be to offer some layering of task families (e.g. as per plot-type) to submit at least a group of different 'single tasks' (i.e. submit histogram, profile, spatial plot, etc tasks separately) - this removes a number of the growth of tasks by variable and number of models, but reduces some of the 'black-box'-ness of all-in task approach.
• are there any further steps that can be taken to improve human readability of failed batch error messaging and organisation in outputs?
• even when tasks fail, is there value to attempt to create webpage? or enhance documentation to encourage set to true, or something else??

Hope further perspectives and user feedback useful food for thought and iteration on this one.

[jfrost-mo]

is it worth disabling the auto retrigger from single task mode, given risks just running bulk of succeeding tasks twice if anything fails?

A retriggered task should only rerun the failed tasks. If it is rerunning the succeeded ones then that is a bug that needs to be fixed.

as a medium-step to individual tasks, how easy/efficient might it be to offer some layering of task families (e.g. as per plot-type) to submit at least a group of different 'single tasks'

I was chatting with the Metomi team last week about possibly chunking things into multiple jobs per cycle. That was more focused on increasing parallelism without getting stuck in queues, but it could be used for this kind of splitting.

are there any further steps that can be taken to improve human readability of failed batch error messaging and organisation in outputs?

We have the possibility of custom outputs, but they may be overwhelming if many tasks failed. We have a lot more flexibility for the content of job.out/job.err. While this will require replacing rose_bunch with our own implementation, we likely want to do that anyway (on the advice of the metomi team) as we can then avoid having to spend 10 seconds starting the python interpreter for each task (on a 30-60 second runtime for non-aggregation tasks), and can also dynamically scale our parallelism to match the memory usage.

even when tasks fail, is there value to attempt to create webpage? or enhance documentation to encourage set to true, or something else??

It would be reasonable to still create the webpage. What I want to avoid however is the workflow finishing, as that makes it harder to debug. It may be that we could run the create webpage task unconditionally, but add an extra dummy task that depends on everything passing that blocks the housekeeping/workflow completion.

[ukmo-huw-lewis]

Will leave potential solutions, options etc with you to consider. Do use Stephen and others as sounding board etc.

On specific, the issue with retriggering is in context of the single task apparently being retriggered. Are you saying that only the failed jobs within the batches submitted actually run again? From user perspective, I just see a red failed task, then it retriggering (assuming it was calling all batches). I have no insight either way on what was being called on 2nd run. If being more efficient behind the scenes, that's great. However, does also reflect risk of "black box" with everything now in 1 task.

Many thanks.

[SGallagherMet]

I have no insight either way on what was being called on 2nd run.

You can see which jobs are submitted in the job.out file, and by which job.<number>.out/err logs are created.

Although I do agree with your point, it's definitely not user friendly, especially as many users are not familiar with suites that use rose bunch. I think we can improve this a little by setting the name option in the rose bunch config.

[SGallagherMet]

@jfrost-mo - To answer your original question I've currently only been looking at UKV trial data which is hourly out to T+48.
I have a branch where I constrain to a single stash code/variable name during the load step (to trigger Iris's shortcut filter-on-load behaviour) and I've separated the baking into separate tasks so I can get some timings.
Looking at just Screen Temperature, you are correct that the per-cycle plots are quite quick. Histograms, min and max plots all had a median run time of about 30 seconds and a max runtime of about 3 minutes.
Aggregation plots had a run time of between 1 and two hours.

MOGREPSUK trials output hourly to T+126

[jfrost-mo]

Sounds like we should split apart the aggregation plots into their own tasks then, while keeping the per-cycle tasks together.

If we assume worst case of 7.5 minutes per MO-GREPS-UK variable ( $ \frac{126}{48} \times 3 $ ), the current setup should allow for over 750 diagnostics in a single task

[SGallagherMet]

I've pushed an example in #1720 of how the split plots can be created without restoring the old include files. It's not especially elegant though so any comments would be appreciated.