CI tests not cleaning up temp files properly

[Alex-Vasile]

If a test crashes, e.g. because of OOM, then the temporary files it creates do not get cleaned up properly. This recently caused the storage on our local CI runner to fill up (all 10 TB) and causes all subsequent tests to fail because they couldn't write anything to disk.

The issue came up from flux_test.py But other tests likely do the same.

• Tests need to clean up after themselves, even if they crash.
• Tests need to store their temp artifacts in a better place, currently they are being stored at the top level of the directory, making it difficult to find anything.

[ScottTodd]

We shouldn't assume that tests will be able to clean up after themselves. Runners can go offline in the middle of a job for various reasons, especially if a test fails or crashes.

The usual tricks are:

1. Store temporary files in a designated temporary directory
2. Add a post-job hook (or pre-job hook) that cleans up specific directories, like the runner working directory: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/running-scripts-before-or-after-a-job