Improvements for gain selection automation#293

Merged

morcuended merged 86 commits intomainfrom gainsel_automation

Oct 24, 2024

+435-193

Commits on Oct 24, 2024

write a history file for each subrun and check its output

marialainez
authored and
morcuended
committed
move run_sacct_j function to job.py

marialainez
authored and
morcuended
committed
add memory required by job to the cfg file

marialainez
authored and
morcuended
committed
improve gain_selection script

marialainez
authored and
morcuended
committed
relaunch the jobs that finished in TIMEOUT

marialainez
authored and
morcuended
committed
remove unnecessary imports

marialainez
authored and
morcuended
committed
give a warning and exit if run summary is empty

marialainez
authored and
morcuended
committed
solve mistake

marialainez
authored and
morcuended
committed
avoid creating a list with the files of all subruns of a run

marialainez
authored and
morcuended
committed
make the check looking at the history files (instead of logs)

marialainez
authored and
morcuended
committed
remove unnecessary imports and variables

marialainez
authored and
morcuended
committed
add simulate option

marialainez
authored and
morcuended
committed
read all the paths from the cfg file

marialainez
authored and
morcuended
committed
change arg type of simulate option

marialainez
authored and
morcuended
committed
make the check for failed jobs runwise

marialainez
authored and
morcuended
committed
create a history file for each successful run (after check)

marialainez
authored and
morcuended
committed
fix mistake

marialainez
authored and
morcuended
committed
fix mistake

marialainez
authored and
morcuended
committed
solve mistake

marialainez
authored and
morcuended
committed
Update src/osa/scripts/gain_selection.py

marialainez
and
morcuended
committed
remove run_sacct_j function + update run_sacct function with an optional argument

marialainez
authored and
morcuended
committed
put memory requirement for both tools + solve mistake

marialainez
authored and
morcuended
committed
move job_finished_in_timeout function to job.py

marialainez
authored and
morcuended
committed
solve mistake

marialainez
authored and
morcuended
committed
solve mistakes

marialainez
authored and
morcuended
committed
check if the files have been already copied (to be able to launch the script several times)

marialainez
authored and
morcuended
committed
fix issues

marialainez
authored and
morcuended
committed
avoid creating directories with --simulate option + add docstrings

marialainez
authored and
morcuended
committed
Update src/osa/scripts/gain_selection.py

marialainez
and
morcuended
committed
add parameters type

marialainez
authored and
morcuended
committed
consider case where no log files have been created yet

marialainez
authored and
morcuended
committed
add verbose mode

marialainez
authored and
morcuended
committed
avoid creating history and log files for subruns that do not have 4 streams

marialainez
authored and
morcuended
committed
add run_summary_table function

marialainez
authored and
morcuended
committed
rename function run_already_copied

marialainez
authored and
morcuended
committed
remove unnecessary elements

marialainez
authored and
morcuended
committed
use run_summary_table in nightsummary

marialainez
authored and
morcuended
committed
add line to create the directory output_dir

marialainez
authored and
morcuended
committed
avoid creating the gain selection flag file if any run is not completed

marialainez
authored and
morcuended
committed
use the same date format for all the scripts of lstosa (YYYY-MM-DD)

marialainez
authored and
morcuended
committed
add necessary import

marialainez
authored and
morcuended
committed
check if sequencer jobs are running to be able to launch it several times

marialainez
authored and
morcuended
committed
use only the run number to search the job names

marialainez
authored and
morcuended
committed
fix mistake

marialainez
authored and
morcuended
committed
use the same date format

marialainez
authored and
morcuended
committed
remove unused import

marialainez
authored and
morcuended
committed
avoid creating the flag file if any run is still running

marialainez
authored and
morcuended
committed
adapt tests

marialainez
authored and
morcuended
committed
change docstring

marialainez
authored and
morcuended
committed
check if sequencer is already running to allow to launch it several times

marialainez
authored and
morcuended
committed
adapt gain selection to catch warnings in the logs such as the one in 20240626

marialainez
authored and
morcuended
committed
define base_dir

marialainez
authored and
morcuended
committed
exit early if gain sel finished flag exists
morcuended
committed
Apply suggestions from code review
morcuended
committed
rename warnings variable
morcuended
committed
changing some info messages to debug level
morcuended
committed
adapt tests

marialainez
authored and
morcuended
committed
remove unused function

marialainez
authored and
morcuended
committed
remove unused import

marialainez
authored and
morcuended
committed
correct indent

marialainez
authored and
morcuended
committed
remove unused import

marialainez
authored and
morcuended
committed
allow watching the job status when launching sequencer with -s

marialainez
authored and
morcuended
committed
stop gain selection check only in case FF heuristic identification is needed

marialainez
authored and
morcuended
committed
add option in the cfg to decide whether to use or not FF heuristic identification

marialainez
authored and
morcuended
committed
avoid creating the flag file if there is no data

marialainez
authored and
morcuended
committed
solve mistake

marialainez
authored and
morcuended
committed
check if sequencer already finished before submitting the jobs

marialainez
authored and
morcuended
committed
reduce memory requirement in the example cfg

marialainez
authored and
morcuended
committed
adapt tests

marialainez
authored and
morcuended
committed
return False if no jobs were launched yet

marialainez
authored and
morcuended
committed
adapt warning message + create flag file even with warnings

marialainez
authored and
morcuended
committed
rename runwise .history files as .closed

marialainez
authored and
morcuended
committed
update history files only if .closed file does not exist

marialainez
authored and
morcuended
committed
Update src/osa/scripts/sequencer.py

marialainez
and
morcuended
committed
write the run number using always 5 digits

marialainez
authored and
morcuended
committed
add necessary import

marialainez
authored and
morcuended
committed
check if the R0G files can be opened with ctapipe_io_lst

marialainez
authored and
morcuended
committed
remove unused variable

marialainez
authored and
morcuended
committed
remove unnecessary check

marialainez
authored and
morcuended
committed
check the last job id (e.g. for timeouts)

marialainez
authored and
morcuended
committed
rename function + rename force argument

marialainez
authored and
morcuended
committed
Update src/osa/scripts/sequencer.py

marialainez
and
morcuended
committed
Update src/osa/job.py

marialainez
and
morcuended
committed
Update src/osa/job.py

marialainez
and
morcuended
committed
change function name

marialainez
authored and
morcuended
committed
fix name of force option

marialainez
authored and
morcuended
committed