Restructure and comment the tau embedding method

[winterchristian]

PR description:

This PR restructures and cleans the tau embedding method (TauAnalysis/MCEmbeddingTools) so that it is possible to produce RUN 3 tau embedding samples with CMS submission workflows.
One goal was to remove the need of --customise or --customise_commands in the tau embedding cmsDriver.py commands.
This was possible due to modifiers and the possibility to add python config fragments to some cmsDriver.py steps.
Only the HLT step required additional modification of the cmsDriver.py config builder.

The other goal of this PR is to have more understandable, better commented code and therefore sustainable code.

Tau embedding cmsDriver.py commands

With the changes introduced in this PR, tau embedding samples can be produced using the following cmsDriver.py commands.

Selection

cmsDriver.py \
--step RAW2DIGI,L1Reco,RECO,PAT,FILTER:TauAnalysis/MCEmbeddingTools/Selection_FILTER_cff.makePatMuonsZmumuSelection \
--processName SELECT \
--data \
--scenario pp \
--conditions auto:run3_data \
--era Run3_2024 \
--eventcontent TauEmbeddingSelection \
--datatier RAWRECO \
--filein "root://cmsdcache-kit-disk.gridka.de:1094//store/data/Run2024C/Muon0/RAW/v1/000/380/115/00000/00979445-916c-42e2-8038-428d7bd4f176.root" \
--fileout ...

LHE and Cleaning

cmsDriver.py \
--step USER:TauAnalysis/MCEmbeddingTools/LHE_USER_cff.embeddingLHEProducerTask,RAW2DIGI,RECO \
--processName LHEembeddingCLEAN \
--data \
--scenario pp \
--conditions auto:run3_data \
--era Run3_2024 \
--eventcontent TauEmbeddingCleaning \
--datatier RAWRECO \
--procModifiers tau_embedding_cleaning,tau_embedding_mu_to_mu \
--filein ... \
--fileout ...

Simulation Gen

cmsDriver.py TauAnalysis/MCEmbeddingTools/python/Simulation_GEN_cfi.py \
--step GEN,SIM,DIGI,L1,DIGI2RAW \
--processName SIMembeddingpreHLT \
--mc \
--beamspot DBrealistic \
--geometry DB:Extended \
--era Run3_2024 \
--conditions auto:phase1_2024_realistic \
--eventcontent TauEmbeddingSimGen \
--datatier RAWSIM \
--procModifiers tau_embedding_sim,tau_embedding_mu_to_mu \
--filein ... \
--fileout ...

Simulation HLT

cmsDriver.py \
--step HLT:Fake2+TauAnalysis/MCEmbeddingTools/Simulation_HLT_customiser_cff.embeddingHLTCustomiser \
--processName SIMembeddingHLT \
--mc \
--beamspot DBrealistic \
--geometry DB:Extended \
--era Run3_2024 \
--conditions auto:phase1_2024_realistic \
--eventcontent TauEmbeddingSimHLT \
--datatier RAWSIM \
--filein ... \
--fileout ...

Simulation Reco

cmsDriver.py \
--step RAW2DIGI,L1Reco,RECO,RECOSIM \
--processName SIMembedding \
--mc \
--beamspot DBrealistic \
--geometry DB:Extended \
--era Run3_2024 \
--conditions auto:phase1_2024_realistic \
--eventcontent TauEmbeddingSimReco \
--datatier RAW-RECO-SIM \
--procModifiers tau_embedding_sim \
--filein ... \
--fileout ...

Merging

cmsDriver.py \
--step USER:TauAnalysis/MCEmbeddingTools/Merging_USER_cff.merge_step,PAT \
--processName MERGE \
--data \
--scenario pp \
--conditions auto:run3_data \
--era Run3_2024 \
--eventcontent TauEmbeddingMergeMINIAOD \
--datatier USER \
--procModifiers tau_embedding_merging \
--inputCommands 'keep *_*_*_*' \
--filein ... \
--fileout ...

NanoAOD

cmsDriver.py \
--step NANO:@TauEmbedding \
--data \
--scenario pp \
--conditions auto:run3_data \
--era Run3_2024 \
--eventcontent TauEmbeddingNANOAOD \
--datatier NANOAODSIM \
--filein ... \
--fileout ...

PR validation:

I had to disable the tests, as I first have to get RAW samples from tape to create a small set of samples I can use. The test .root files right now are only useable for the old structure.
I will deliver tests in the next pull request.

UPDATE
I reactivated the unit tests as I realized I can use the RAW samples from the release validation tests.
Both, the unit tests as the release validation tests worked locally.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48408/45306

• Found files with invalid states:

  • TauAnalysis/MCEmbeddingTools/python/Merging_cff.py:
    • Added: 4276cf0
    • Deleted: 0928b23

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): Use RNTuple output module for NanoAOD #48162

[cmsbuild]

A new Pull Request was created by @winterchristian for master.

It involves the following packages:

• Configuration/Applications (operations)
• Configuration/ProcessModifiers (operations)
• TauAnalysis/MCEmbeddingTools (simulation)

@antoniovilela, @civanch, @cmsbuild, @davidlange6, @fabiocos, @kpedro88, @mandrenguyen, @mdhildreth, @rappoccio can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @mmusich this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[kpedro88]

@smuzaffar will RAW files used as input for unit tests in TauAnalysis/MCEmbeddingTools/test automatically get cached?

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48408/45308

• Found files with invalid states:

  • TauAnalysis/MCEmbeddingTools/python/Merging_cff.py:
    • Added: 4276cf0
    • Deleted: 0928b23

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): Use RNTuple output module for NanoAOD #48162

[cmsbuild]

Pull request #48408 was updated. @antoniovilela, @civanch, @cmsbuild, @davidlange6, @fabiocos, @kpedro88, @mandrenguyen, @mdhildreth, @rappoccio can you please check and sign again.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-48408/45348

• Found files with invalid states:

  • TauAnalysis/MCEmbeddingTools/python/Merging_cff.py:
    • Added: 4276cf0
    • Deleted: 0928b23

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): Use RNTuple output module for NanoAOD #48162

[cmsbuild]

Pull request #48408 was updated. @antoniovilela, @civanch, @cmsbuild, @davidlange6, @fabiocos, @kpedro88, @mandrenguyen, @mdhildreth, @rappoccio can you please check and sign again.

[civanch]

please test

[Moanwar]

+Upgrade

[civanch]

+simulation

[ftorrresd]

+1

[winterchristian]

@ftenchini, @mandrenguyen, and @AdrianoDee can you please approve this, too. I think only your approval is missing.

[AdrianoDee]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

[ftenchini]

+1