Suspend campaign execution to resume later

[arthursloan]

Sub issue for issue #32
For several reasons, I believe it would be beneficial for ARES OS to support suspending and resuming campaigns. This behavior is distinct from the pausing and continuing feature that already exists in that it would allow for a campaign to be suspended, ARES OS closed, and then allow for the campaign to pick up in the same spot after ARES OS is reopened.

I see to major benefits to this feature.

1. Recovery from ARES OS crashes. As various elements move towards being services, it is possible that we could have a crash that doesn't effect the overall operation of the system. In this case, the user could wish to simply pick back up where the campaign was rather than dealing with figuring out how to load the previous campaign's data as seed data to the new campaign. I sort of envision this as a "Ares quit unexpectedly would you like to 1. End the previous campaign. 2. Resume the previous campaign. 3. Resume the previous campaign, and rerun the previous step." Resuming after a crash is potentially fraught with a lot of questions, like do we rerun the setup script, etc. but would overall be valuable.
2. Allow more natural human-in-the-loop operation. If the human is part of the loop for things like experimental execution or data collection there is no guarantee on the timescale that this occurs on. It could be minutes to take a sample over to a balance and weigh it, or it could be days while the researcher waits for a shared instrument to be available. In this case you probably wouldn't want to leave ARES OS open with the campaign running, so some way to save the state and open it back up when you have the data would be invaluable. I see this intersecting with Issue Pause and wait for human approval and/or input #33, where you hit the point where ARES OS is asking for your input, suspend the campaign, go collect your data, resume your campaign, and enter your data in the prompt.

An alternative to this could be some straightforward GUI way to manage any seed dataset the campaign is using. Something like selecting previous campaigns you want to draw from and then being able to further select only specific experiments from those campaigns.

[Babreakfast]

Would it make sense to just keep the suspend/resume on experiment level? Like I see what you mean by expecting to resume on a step/command level, but I don't think that is particularly feasible considering an experiment is designed to be more of an "atomic" unit.
Maybe we could do something like saving the planned values and analyses, and then instead of "resuming" in a traditional sense, we would start a new campaign but provide the aforementioned values as a "seed" which would then in a sense resume the campaign without having to bother keeping track of device states during the previous run.