Avoid filename collisions for recipes and diagnostic output

[jfrost-mo]

Coming back to this ticket, I think we are actually better off fixing the underlying issue of recipe filename collisions.

In #1713 we display a number of extra points of metadata on each diagnostic in the list. This metadata is needed to be able to filter on, but having it shown removes the need to have it in the title.

However, if we were to remove it from the title currently, then we would run into clashes on the underlying files, which have their names derived from the title.

What I think we should do instead is to add a random component to the filename to avoid collisions. There after the title doesn't need to be unique, and we don't have to avoid any special characters.

This doesn't preclude changing the operator to accept symbols or letters, but such a change should be judged on clarity, rather than what goes into our filenames.

Originally posted by @jfrost-mo in #1692

[SGallagherMet]

In #1707 I mentioned making it easier for users to identify what jobs have failed in the bake step #1707 (comment)

Whatever solution is adopted here should avoid making this worse.

[jfrost-mo]

Indeed. This will mainly (entirely?) affect the parbake part of the workflow. For debugability it would be good to keep it deterministic, so rather than random maybe append a hash of the templated recipe to the filename?

A further point (possibly for a separate issue once defined) would be about whether we show failed diagnostics in the UI. Currently they are shown, but the button will just load a file listing with the log file, instead of the diagnostic. Should we have failed recipes not appear in the listing of diagnostics, or should we instead mark them in some way to indicate they have failed?

[jfrost-mo]

Adding a hash component might also help with #1049 somewhat. Though not entirely unless we put something in the /plots/ folder name.