dev3 updates

.flake8

@@ -1,10 +1,4 @@
 [flake8]
-
-# line length of 100 is recommended, but set it to a forgiving value
-max-line-length = 120
-
-# codes of errors to ignore
+max-line-length = 119
 ignore = E128, E306, E402, E722, E731, W504, Q003
-
-# enforce double quotes
-inline-quotes = double
+inline-quotes = double

.gitignore

@@ -31,7 +31,7 @@ docs/_build
 tmp
 store
 software
-data
+columnflow_venv
 .data
 .law
 .setups

.markdownlint

@@ -0,0 +1 @@
+modules/columnflow/.markdownlint

.setups/template.sh

@@ -0,0 +1,36 @@
+export CF_CERN_USER="kjaffel"
+export CF_CERN_USER_FIRSTCHAR="${CF_CERN_USER:0:1}"
+export CF_DATA="$CF_REPO_BASE/columnflow_venv"
+export CF_SOFTWARE_BASE="$CF_DATA/software"
+export CF_VENV_BASE="$CF_SOFTWARE_BASE/venvs"
+export CF_STORE_NAME="cf_store"
+export CF_WLCG_USE_CACHE="true"
+export CF_WLCG_CACHE_CLEANUP="false"
+export CF_VENV_SETUP_MODE_UPDATE="false"
+export CF_VENV_SETUP_MODE="update"
+export CF_INTERACTIVE_VENV_FILE=""
+export CF_LOCAL_SCHEDULER="true"
+export CF_SCHEDULER_HOST="127.0.0.1"
+export CF_SCHEDULER_PORT="8082"
+export CF_FLAVOR="cms"
+export LAW_CMS_VO="cms"
+
+# on manivald
+export CF_CRAB_STORAGE_ELEMENT="T2_EE_Estonia"
+export CF_SLURM_FLAVOR="manivald"
+export CF_SLURM_PARTITION="main"
+export CF_CLUSTER_LOCAL_PATH="/local/$CF_CERN_USER/HHMultilepton_Run3/"
+export CF_CRAB_BASE_DIRECTORY="/store/user/$CF_CERN_USER/HHMultilepton_Run3/cf_crab_outputs"
+export TMPDIR="/scratch/local/$CF_CERN_USER"
+
+# on lxplus
+#export CF_CRAB_STORAGE_ELEMENT="T2_CH_CERN"
+#export CF_HTCONDOR_FLAVOR="cern_el9"   # or "cern" for older versions of lxplus not using ELMA9
+#export CF_CLUSTER_LOCAL_PATH="/eos/user/$CF_CERN_USER_FIRSTCHAR/$CF_CERN_USER/HHMultilepton_Run3/"
+#export CF_CRAB_BASE_DIRECTORY="$CF_CLUSTER_LOCAL_PATH/cf_crab_outputs"
+#export TMPDIR="/tmp/$CF_CERN_USER"
+
+
+export CF_STORE_LOCAL="$CF_CLUSTER_LOCAL_PATH/$CF_STORE_NAME"
+export CF_WLCG_CACHE_ROOT="$CF_CLUSTER_LOCAL_PATH/cf_scratch"
+export CF_JOB_BASE="$CF_CLUSTER_LOCAL_PATH/cf_jobs"

LICENSE

@@ -1,3 +1,3 @@
 All rights reserved.
 
-The UHH HH -> bbtautau developers.
+The HH -> Multilepton developers.

README.md

@@ -1,64 +1,125 @@
-# HH → bb𝜏𝜏
+# HH (H → WW/ZZ/𝜏𝜏) → Multi-Leptons Analysis
 
-[![Lint and test](https://github.com/uhh-cms/hh2bbtautau/actions/workflows/lint_and_test.yaml/badge.svg)](https://github.com/uhh-cms/hh2bbtautau/actions/workflows/lint_and_test.yaml)
+**Table of contents**
+- [Introduction](#introduction)
+- [Installation (first time)](#first-time-setup)
+- [Usage](#usage)
+- [Useful links](#useful-links)
+- [Contributors](#contributors)
+- [Development](#development)
 
-## Quickstart
 
-A couple test tasks are listed below.
-They might require a **valid voms proxy** for accessing input data.
+## Introduction
+
+This is the code base for the Run2+Run3 iteration of the CMS HH Multileptons analysis.
+
+The code is forked and for now heavily based on the UHH bersion of the [HH → bb𝜏𝜏 analysis](https://github.com/uhh-cms/hh2bbtautau)
+and still very much WIP. Expect remnants from the bb𝜏𝜏 analysis, crashes and bugs, you have been warned!
+
+Please make sure you are subscribed to our e-group: [email protected]
+It controls the acess to our indico etc. and is a good way to get updates for our meetings.
+
+Also join our channel on [mattermost](https://mattermost.web.cern.ch/cms-exp/channels/hh-multilepton-run3).
+(You will need to join the CMS team first if not done so).
+
+The code is currently developed with the Tallinn T2 (and lxplus) in mind.
+For further questions please, contact t\*\*\*\*.l\*\*\*\*@no-spam-cern.ch.
+
+## First time setup
 
 ```shell
-# clone the project
-git clone --recursive [email protected]:uhh-cms/hh2bbtautau.git
-cd hh2bbtautau
-
-# source the setup and store decisions in .setups/dev.sh (arbitrary name)
-source setup.sh dev
-
-# index existing tasks once to enable auto-completion for "law run"
-law index --verbose
-
-# run your first task
-# (they are all shipped with columnflow and thus have the "cf." prefix)
-law run cf.ReduceEvents \
-    --version v1 \
-    --dataset hh_ggf_bbtautau_madgraph \
-    --branch 0
-
-# create a plot
-law run cf.PlotVariables1D \
-    --version v1 \
-    --datasets hh_ggf_bbtautau_madgraph \
-    --producers default \
-    --variables jet1_pt \
-    --categories incl \
-    --branch 0
+# 1. clone the project
+git clone --recursive [email protected]:HEP-KBFI/hhmultilepton.git
+cd hhmultilepton
 
-# create a (test) datacard (CMS-style)
-law run cf.CreateDatacards \
-    --version v1 \
-    --producers default \
-    --inference-model test \
-    --workers 3
+# 2. get a voms token
+voms-proxy-init -voms cms -rfc -valid 196:00
+
+# 3. copy the provided template to a new file (you can choose any <setup_name>):
+cp .setups/template.sh .setups/mydev.sh
+
+# 4. open .setups/mydev.sh in your editor and adjust any environment variables or paths as needed for your local setup.
+# then source the main setup script with your custom setup name:
+source setup.sh <setup_name> [sandbox_type]
+```
+```bash 
+source setup.sh --help
+Arguments:
+  <setup_name>     Name of the setup (random name of your choice)
+  [sandbox_type]   Optional: choose between 'minimal' (default) or 'full'
+Examples:
+  source setup.sh mydev            # uses 'minimal' environment from (sandboxes/venv_multilepton.sh)
+  source setup.sh mydev full       # uses 'full' environment from (sandboxes/venv_multilepton_dev.sh) 
 ```
 
-## Useful commands
+Note: If you prefer not to use the provided template, you can still activate the environment manually by running:
+source setup.sh `<setup_name>`
+In this case, `<setup_name>` should not already exist under the `.setups/` directory.
+When you run the command, the setup script will guide you interactively, prompting you to enter the required environment variables (as `export` commands). Once completed, these settings will be automatically saved to `.setups/<setup_name>.sh`.
+
+
+<img width="1336" height="506" alt="image" src="img.png" />
+
+
+Code can now be run but first storage locations for the tasks outputs should be checked as configured [here](https://github.com/HEP-KBFI/hhmultilepton/blob/master/law_outputs.cfg#L26-L90). Currently outputs point to the user store of the `T2_EE_Estonia on manivald` so that outputs are also accessible remotely, but we will likely adapt this over time depending on the output.
+I.e large outputs available in a remote reachable location, smaller ones on local stores. Larger ones likely also split by user/cluster so that central versions can be reused.
+
+**For development on lxplus "i strongly" advise to change `wlcg_fs_manivald` to `wlcg_fs_cernbox` in the beginning.**
+
+## Usage 
 
-### Full reduction
+1. Setup your enviorement (**always**):
 
 ```shell
-law run cf.ReduceEventsWrapper \
-    --version prod1 \
-    --configs run3_2022_preEE \
-    --datasets "*" \
-    --shifts "nominal,{tune,hdamp,mtop}_{up,down}" \
-    --cf.ReduceEvents-workflow htcondor \
-    --cf.ReduceEvents-pilot \
-    --cf.ReduceEvents-tasks-per-job 3 \
-    --local-scheduler False \
-    --workers 6
+voms-proxy-init -voms cms -rfc -valid 196:00
+
+# source the setup and export env in the sorted file " .setups/mydev.sh " in this case
+source setup.sh mydev
+```
+
+2. Try to run on 1 signal, 1 backgound and 1 data locally:
+
+```shell
+law run cf.PlotVariables1D \
+    --version test \
+    --producers default \
+    --variables nmu \
+    --datasets hh_ggf_htt_hvv_kl1_kt1_powheg,zz_pythia,data_e_c  \
+```
+
+3. And if the above run sucessfully, you can proceed to submit jobs via slurm/condor adding 
+
+```shell
+    --workflow slurm \     # or
+    --workflow htcondor \  # or
+    --workflow crab \      # to be tested!?
 ```
 
+## Documentation
+
+- Lives here: https://gitlab.cern.ch/hh-multileptons-full-analysis/hh-multileptons-doc
+- Talks:
+    - slides: https://indico.cern.ch/event/1580193/contributions/6660044/attachments/3121091/5534653/multilep%20framework.pdf
+
+## 🙏 Contributors
+
+<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
+<!-- prettier-ignore-start -->
+<!-- markdownlint-disable -->
+
+<table>
+  <tbody>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/tolange"><img src="https://avatars.githubusercontent.com/u/11850680?s=96&v=4" width="100px;" alt="`Torben Lange`"/><br /><sub><b>Torben Lange</b></sub></a><br /><a href="https://github.com/HEP-KBFI/hhmultilepton/commits/master/?author=tolange" title="Code">💻</a> </td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/MatheuspCoelho"><img src="https://avatars.githubusercontent.com/u/85200761?v=4" width="100px;" alt="`Matheus Coelho`"/><br /><sub><b>Matheus Coelho</b></sub></a><br /><a href="https://github.com/HEP-KBFI/hhmultilepton/commits/master/?author=MatheuspCoelho" title="Code">💻</a> </td>
+    </tr>
+  </tbody>
+</table>
+
+<!-- markdownlint-restore -->
+<!-- prettier-ignore-end -->
+<!-- ALL-CONTRIBUTORS-LIST:END -->
+
 ## Useful links
 
 - [columnflow documentation](https://columnflow.readthedocs.io/en/latest/index.html)
@@ -68,19 +129,20 @@ law run cf.ReduceEventsWrapper \
   - [GrASP](https://cms-pdmv-prod.web.cern.ch/grasp/)
   - [XSDB](https://xsdb-temp.app.cern.ch)
   - [DAS](https://cmsweb.cern.ch/das)
-NanoAOD:
+NanoAOD
   - [Nano documentation](https://gitlab.cern.ch/cms-nanoAOD/nanoaod-doc)
   - [Correctionlib files](https://gitlab.cern.ch/cms-nanoAOD/jsonpog-integration)
-- JME
-  - [Docs](https://cms-jerc.web.cern.ch)
-- BTV
-  - [Docs](https://btv-wiki.docs.cern.ch)
+- [JME](https://cms-jerc.web.cern.ch)
+- [BTV](https://btv-wiki.docs.cern.ch)
 - TAU
   - [Run 2 Twiki](https://twiki.cern.ch/twiki/bin/viewauth/CMS/TauIDRecommendationForRun2)
   - [Run 3 Twiki](https://twiki.cern.ch/twiki/bin/viewauth/CMS/TauIDRecommendationForRun3)
   - [Correctionlib files](https://gitlab.cern.ch/cms-tau-pog/jsonpog-integration/-/tree/TauPOG_v2_deepTauV2p5/POG/TAU?ref_type=heads)
 
 ## Development
 
-- Source hosted at [GitHub](https://github.com/uhh-cms/hh2bbtautau)
-- Report issues, questions, feature requests on [GitHub Issues](https://github.com/uhh-cms/hh2bbtautau/issues)
+- Source hosted at [GitHub](https://github.com/HEP-KBFI/hhmultilepton)
+- Report issues, questions, feature requests on [GitHub Issues](https://github.com/HEP-KBFI/hhmultilepton/issues)
+- Ideally also ping us on [mattermost](https://mattermost.web.cern.ch/cms-exp/channels/hh-multilepton-run3).
+- For new features open a new branch before merging into master, ask for a code review by a felllow contributor and dont forget linting!
+- Happy coding 😊