diff --git a/README.md b/README.md index 562f1778..6ef9aee0 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,7 @@ 2. [Data production](#dataprod) 1. [Skimming](#skim) 2. [Data sources](#sources) + 3. [Job Submission](#job-submission) 3. [Reconstruction Chain](#org0bc224d) 1. [Cluster Size Studies](#orgc33e2a6) 4. [Event Visualization](#org44a4071) @@ -99,6 +100,66 @@ This framework relies on photon-, electron- and pion-gun samples produced via CR The `PU0` files above were merged and are stored under `/data_CMS/cms/alves/L1HGCAL/`, accessible to LLR users and under `/eos/user/b/bfontana/FPGAs/new_algos/`, accessible to all lxplus and LLR users. The latter is used since it is well interfaced with CERN services. The `PU200` files were merged and stored under `/eos/user/i/iehle/data/PU200//`. + +## Job Submission + +Job submission to HT Condor is handled through `bye_splits/production/submit_scripts/job_submit.py` using the section of `config.yaml` for its configuration. The configuration should include usual condor variables, i.e `user`, `proxy`, `queue`, and `local`, as well as a path to the `script` you would like to run on condor. The `arguments` sub-section should contain `key/value` pairs matching the expected arguments that `script` accepts. You can also pass arguments directly in the command line, in which case these values will superseed the defaults set in the configuration file. The new `Arguments` class in `bye_splits/utils/job_helpers.py` verifies that the passed arguments are accepted by `script` and that all required arguments have assigned values. For now, this requires that `script` uses `Arguments` to import its arguments, using a dictionary called `arg_dict`; an example can be found in `tests/submission/dummy_submit.py`. The variable that you would like to iterate over should be set in `iterOver` and its value should correspond to a `key` in the `arguments` sub-section whose value is a list containing the values the script should iterate over. It then contains a section for each particle type which should contain a `submit_dir`, i.e. the directory in which to read and write submission related files, and `args_per_batch` which can be any number between 1 and `len(arguments[])`. An example of the `job` configuration settings is as such: + +```yaml +job: + user: iehle + proxy: ~/.t3/proxy.cert + queue: short + local: False + script: /grid_mnt/vol_home/llr/cms/ehle/NewRepos/bye_splits/tests/submission/dummy_submit.py + iterOver: gen_arg + arguments: + float_arg: 0.11 + str_arg: a_string + gen_arg: [gen, 3.14, work, broke, 9, False, 12.9, hello] + test: + submit_dir: /home/llr/cms/ehle/NewRepos/bye_splits/tests/submission/ + args_per_batch: 2 +``` + +After setting the configuration variables, the jobs are created and launched via + + python bye_splits/production/submit_scripts/job_submit.py + +while will produce the executable `.sh` file in `/subs/` that looks like: + + #!/usr/bin/env bash + export VO_CMS_SW_DIR=/cvmfs/cms.cern.ch + export SITECONFIG_PATH=$VO_CMS_SW_DIR/SITECONF/T2_FR_GRIF_LLR/GRIF-LLR/ + source $VO_CMS_SW_DIR/cmsset_default.sh + list=$1 + cleaned_list=$(echo $list | tr -d '[]' | tr ';' ' + ') + while IFS=";" read -r val; do + python /grid_mnt/vol_home/llr/cms/ehle/NewRepos/bye_splits/tests/submission/dummy_submit.py --gen_arg "$val" --float_arg 0.11 --str_arg a_string + done <<< "$cleaned_list" + +and the `.sub` file submitted to HT Condor in `/jobs/` that looks like: + + executable = /home/llr/cms/ehle/NewRepos/bye_splits/tests/submission/subs/dummy_submit_exec_v5.sh + Universe = vanilla + Arguments = $(gen_arg) $(float_arg) $(str_arg) + output = /home/llr/cms/ehle/NewRepos/bye_splits/tests/submission/logs/dummy_submit_C$(Cluster)P$(Process).out + error = /home/llr/cms/ehle/NewRepos/bye_splits/tests/submission/logs/dummy_submit_C$(Cluster)P$(Process).err + log = /home/llr/cms/ehle/NewRepos/bye_splits/tests/submission/logs/dummy_submit_C$(Cluster)P$(Process).log + getenv = true + T3Queue = short + WNTag = el7 + +SingularityCmd = "" + include: /opt/exp_soft/cms/t3/t3queue | + queue gen_arg, float_arg, str_arg from ( + ['gen';3.14], 0.11, a_string + ['work';'broke'], 0.11, a_string + [9;False], 0.11, a_string + [12.9;'hello'], 0.11, a_string + ) + +All logs, outputs, and errors are written to their respective files in `/logs/`. Some primary uses of `job_submit.py` include running the [skimming procedure](#skimming), iterating over each particle type, and running the [cluster studies](#cluster-size-studies) over a list of radii. # Reconstruction Chain @@ -120,33 +181,51 @@ The above will create `html` files with interactive outputs. ## Cluster Size Studies -The script `bye_splits/scripts/cluster_size.py` reads a configuration file `bye_splits/scripts/cl_size_params.yaml` and runs the Reconstruction Chain on the `.root` inside corresponding to the chosen particle, where the clustering step is repeated for a range of cluster radii that is specified in the parameter file under `cl_size: Coeffs`. - -The most convenient way of running the study is to do: - - bash run_cluster_size.sh - -where `` is your lxplus username, creating `.hdf5` files containing Pandas DFs containing cluster properties (notably energy, eta, phi) and associated gen-level particle information for each radius. The bash script acts as a wrapper for the python script, setting a few options that are convenient for the cluster size studies that are not the default options for the general reconstruction chain. As of now, the output `.hdf5` files will be written to your local directory using the structure: - - ├── / - │ ├── out - │ ├── data - │ │ ├──new_algos +The optimization of the clustering radius is done via the scripts in `bye_splits/scripts/cluster_size/`. The configuration is done in the `config.yaml` file under `clusterStudies`. +The initial steps of the reconstruction chain (fill, smooth, seed) are run via + + python run_init_tasks.py --pileup + +which will produce the files required for `bye_splits/scripts/cluster_size/condor/run_cluster.py` (default value for `pileup==PU0`). One can run the script on a single radius: + + python run_cluster.py --radius --particles --pileup + +As the directory name suggests, `run_cluster.py` can and should be run as a `script` passed to an HTCondor job as described by [Job Submission](#job-submission) if you wish +to run over all radii. The configuration would look something like this: + +```yaml +job: +user: iehle +proxy: ~/.t3/proxy.cert +queue: short +local: False +script: /grid_mnt/vol_home/llr/cms/ehle/NewRepos/bye_splits/bye_splits/scripts/cluster_size/condor/run_cluster.py +iterOver: radius +arguments: + radius: [0.001, 0.002, 0.003, 0.004, 0.005, 0.006, 0.007, 0.008, 0.009, + 0.01 , 0.011, 0.012, 0.013, 0.014, 0.015, 0.016, 0.017, 0.018, + 0.019, 0.02 , 0.021, 0.022, 0.023, 0.024, 0.025, 0.026, 0.027, + 0.028, 0.029, 0.03 , 0.031, 0.032, 0.033, 0.034, 0.035, 0.036, + 0.037, 0.038, 0.039, 0.04 , 0.041, 0.042, 0.043, 0.044, 0.045, + 0.046, 0.047, 0.048, 0.049, 0.05] + particles: pions + pileup: PU0 +photons: + submit_dir: /data_CMS/cms/ehle/L1HGCAL/PU0/photons/ + args_per_batch: 10 +electrons: + submit_dir: /data_CMS/cms/ehle/L1HGCAL/PU0/electrons/ + args_per_batch: 10 +pions: + submit_dir: /data_CMS/cms/ehle/L1HGCAL/PU0/pions/ + args_per_batch: 10 +``` -with the files ending up in `new_algos/`. Currently working on implementing an option to send the files directly to your `eos/` directory, assuming the structure: +This will produce the output of `cluster.cluster_default()` for each radius. These files are then combined into one larger `.hdf5` file whose keys correspond to the various radii, and combined and normalized with the gen-level data via: - ├── /eos/user// - │ ├── out - │ ├── data - │ │ ├──PU0 - │ │ │ ├──electrons - │ │ │ ├──photons - │ │ │ ├──pions - │ │ ├──PU200 - │ │ │ ├──electrons - │ │ │ ├──photons - │ │ │ ├──pions + python run_combine.py +The optional `--file` argument performs the combination and normalization with the gen-level data on only ``. diff --git a/bye_splits/production/produce.cc b/bye_splits/production/produce.cc deleted file mode 100644 index cef61f1f..00000000 --- a/bye_splits/production/produce.cc +++ /dev/null @@ -1,94 +0,0 @@ -#include -#include "include/skim.h" - -#include // for printf() -#include // for strtol() -#include // for errno -#include // for INT_MIN and INT_MAX -#include // for strlen - -int convert_to_int(char** argv, int idx) { - char* p; - errno = 0; // not 'int errno', because the '#include' already defined it - long arg = strtol(argv[idx], &p, 10); - if (*p != '\0' || errno != 0) { - return 1; // In main(), returning non-zero means failure - } - - if (arg < INT_MIN || arg > INT_MAX) { - return 1; - } - int arg_int = arg; - - // Everything went well, print it as a regular number plus a newline - return arg_int; -} - -void show_help(const po::options_description&, const std::string&); -po::variables_map process_program_options(int argc, char **argv); - -void validate(po::variables_map args) { - std::string particles = args["particles"].as(); - if(!(particles=="photons" || particles=="electrons" || particles=="pions")) { - throw po::validation_error(po::validation_error::invalid_option_value, "particles"); - } -} - -void show_help(const po::options_description& desc, - const std::string& topic = "") { - std::cout << desc << '\n'; - if (topic != "") { - std::cout << "You asked for help on: " << topic << '\n'; - } -} - -po::variables_map process_program_options(int argc, char **argv) -{ - po::options_description desc("Usage"); - desc.add_options() - ("help,h", - po::value()->implicit_value("") - ->notifier([&desc](const std::string &topic) {show_help(desc, topic);}), - "Show help. If given, show help on the specified topic.") - ("nevents", po::value()->default_value(-1), - "number of entries to consider, useful for debugging (-1 means all)") - ("particles", po::value()->required(), - "type of particle"); - - if (argc <= 1) { - show_help(desc); // does not return - exit( EXIT_SUCCESS ); - } - - po::variables_map args; - try { - po::store(po::parse_command_line(argc, argv, desc), args); - } - catch (po::error const& e) { - std::cerr << e.what() << '\n'; - exit( EXIT_FAILURE ); - } - po::notify(args); - validate(args); - return args; -} - -//Run with ./produce.exe photons -int main(int argc, char **argv) { - std::string dir = "/eos/user/b/bfontana/FPGAs/new_algos/"; - std::string tree_name = "FloatingpointMixedbcstcrealsig4DummyHistomaxxydr015GenmatchGenclustersntuple/HGCalTriggerNtuple"; - - po::variables_map args = process_program_options(argc, argv); - if (args.count("help")) { - return 1; - } - - string particles = args["particles"].as(); - int nevents = args["nevents"].as(); - - std::string infile = particles + "_0PU_bc_stc_hadd.root"; - std::string events_str = nevents > 0 ? std::to_string(nevents) + "events_" : ""; - std::string outfile = "skim_" + events_str + infile; - skim(tree_name, dir + infile, dir + outfile, particles, nevents); - return 0; -} diff --git a/bye_splits/production/submit_scripts/job_submit.py b/bye_splits/production/submit_scripts/job_submit.py new file mode 100644 index 00000000..ddc114a9 --- /dev/null +++ b/bye_splits/production/submit_scripts/job_submit.py @@ -0,0 +1,270 @@ +#!/usr/bin/env python + +import os +import sys + +parent_dir = os.path.abspath(__file__ + 5 * "../") +sys.path.insert(0, parent_dir) + +from bye_splits.utils import params, common, job_helpers +from bye_splits.utils.job_helpers import Arguments + +from datetime import datetime +import re +import subprocess +import argparse +import yaml + +class JobBatches: + """Class for setting up job batches and setting configuration + variables. The function setup_batches() will take the list in + config[arguments[]] and return a list of lists containing + values in each sublist. Example for five total values + with = 2: + [0.01, 0.02, 0.03, 0.04, 0.05] --> [[0.01, 0.02], [0.03, 0.04], [0.05]]""" + + def __init__(self, particle, config): + self.particle = particle + self.config = config + self.iterOver = config["job"]["iterOver"] + self.particle_var = lambda part, var: config["job"][part][var] + + def setup_batches(self): + total_vals = self.config["job"]["arguments"][self.iterOver] + + vals_per_batch = self.particle_var(self.particle, "args_per_batch") + + batches = [total_vals[i: i + vals_per_batch] for i in range(0, len(total_vals), vals_per_batch)] + + return batches + +class CondJobBase: + def __init__(self, particle, config): + self.script = config["job"]["script"] + self.queue = config["job"]["queue"] + self.proxy = config["job"]["proxy"] + self.local = config["job"]["local"] + self.user = config["job"]["user"] + + self.true_args = Arguments(self.script) + self.default_args = {} + for arg, val in config["job"]["arguments"].items(): + self.default_args["--"+arg] = val + self.combined_args = self.true_args.verify_args(self.default_args) + + if "--particles" in self.combined_args: + self.particle = self.combined_args["--particles"] + else: + self.particle = particle + + self.batch = JobBatches(particle, config) + self.particle_dir = self.batch.particle_var(self.particle, "submit_dir") + self.iterOver = "--"+config["job"]["iterOver"] + self.batches = self.batch.setup_batches() + + def _get_condor_args(self): + condor_args = [] + for arg in self.combined_args: + if "action" in self.true_args.accepted_args[arg] and self.true_args.accepted_args[arg]["action"]=="store_true": + continue + else: + condor_args.append(arg.replace("--", "")) + + return condor_args + + def _write_arg_values(self, current_version): + """Adds the argument values, where the batch lists are converted + to strings as [val_1, val_2, ...] --> "[val_1;val_2]". + The choice of a semicolon as the delimiter is arbitrary but it + cannot be a comma because this is the delimeter condor itself uses. + + Example: + + queue radius, particle from ( + [0.01, 0.02], photon + ) + incorrectly assigns radius="[0.01", particle="0.02]" + + queue radius, particle from ( + [0.01;0.02], photon + ) + correctly assigns radius="[0.01, 0.02]", particle="photon" + """ + + condor_args = self._get_condor_args() + + arg_keys = "queue " + ", ".join(condor_args) + " from (\n" + current_version.append(arg_keys) + + batch_strs = [str(batch).replace(", ", ";") for batch in self.batches] + + for batch in batch_strs: + inner_test = [] + for key, val in self.combined_args.items(): + if "action" in self.true_args.accepted_args[key] and self.true_args.accepted_args[key]["action"]=="store_true": + continue + else: + if key != self.iterOver: + inner_test.append(val) + else: + inner_test.append(batch) + inner_test = ", ".join(map(str, inner_test)) + "\n" + current_version.append(inner_test) + + current_version.append(")") + + def write_exec_file(self): + """Writes the .sh file that the condor .sub file runs as the + . This constitutes a few common exports and sourcing, + and follows by writing the Python call to