Skip to content

Create datasets with snakemake #452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jpata wants to merge 30 commits into
base: main
Choose a base branch
from
Open

Create datasets with snakemake #452

jpata wants to merge 30 commits into from

Conversation

@jpata
Copy link
Owner

@jpata jpata commented Feb 4, 2026 •
edited
Loading

Unifies the sample production (gen, postprocessing, tfds steps) and training for three detectors: CMS Run 3, CLIC, CLD in a common workflow, so it is easier to run the sample generation and training from scratch.

CMS, 21k events per sample:

gen:
3.1G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/gen/nopu/QCDForPF_13p6TeV_TuneCUETP8M1_cfi
3.1G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/gen/nopu/TTbar_13p6TeV_TuneCUETP8M1_cfi
2.0G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/gen/nopu/ZTT_All_hadronic_13p6TeV_TuneCUETP8M1_cfi
143G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/gen/pu55to75/QCDForPF_13p6TeV_TuneCUETP8M1_cfi
72G		/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/gen/pu55to75/TTbar_13p6TeV_TuneCUETP8M1_cfi
813G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/gen/pu55to75_val/QCDForPF_13p6TeV_TuneCUETP8M1_cfi
142G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/gen/pu55to75/ZTT_All_hadronic_13p6TeV_TuneCUETP8M1_cfi

postprocessing:
948M	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/post/nopu/QCDForPF_13p6TeV_TuneCUETP8M1_cfi
996M	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/post/nopu/TTbar_13p6TeV_TuneCUETP8M1_cfi
623M	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/post/nopu/ZTT_All_hadronic_13p6TeV_TuneCUETP8M1_cfi
7.5G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/post/pu55to75/QCDForPF_13p6TeV_TuneCUETP8M1_cfi
3.8G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/post/pu55to75/TTbar_13p6TeV_TuneCUETP8M1_cfi
7.2G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/post/pu55to75/ZTT_All_hadronic_13p6TeV_TuneCUETP8M1_cfi

TFDS:
11G		/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/tfds/cms_pf_qcd
1.4G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/tfds/cms_pf_qcd_nopu
5.2G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/tfds/cms_pf_ttbar
1.4G	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/tfds/cms_pf_ttbar_nopu
11G		/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/tfds/cms_pf_ztt
954M	/local/joosep/mlpf/cms/20260204_cmssw_15_0_5_117d32/tfds/cms_pf_ztt_nopu

CLIC, ~100k events per sample:

gen:
250G	/local/joosep/mlpf/clic/v1.2.4_key4hep_2025-05-29_CLIC_819e4e/gen/p8_ee_qq_ecm380
302G	/local/joosep/mlpf/clic/v1.2.4_key4hep_2025-05-29_CLIC_819e4e/gen/p8_ee_ttbar_ecm380
334G	/local/joosep/mlpf/clic/v1.2.4_key4hep_2025-05-29_CLIC_819e4e/gen/p8_ee_WW_fullhad_ecm380

postprocessing:
869M	/local/joosep/mlpf/clic/v1.2.4_key4hep_2025-05-29_CLIC_819e4e/post/p8_ee_qq_ecm380
1.8G	/local/joosep/mlpf/clic/v1.2.4_key4hep_2025-05-29_CLIC_819e4e/post/p8_ee_ttbar_ecm380
1.4G	/local/joosep/mlpf/clic/v1.2.4_key4hep_2025-05-29_CLIC_819e4e/post/p8_ee_WW_fullhad_ecm380

TFDS:
587M	/local/joosep/mlpf/clic/v1.2.4_key4hep_2025-05-29_CLIC_819e4e/tfds/clic_edm_qq_pf
1.2G	/local/joosep/mlpf/clic/v1.2.4_key4hep_2025-05-29_CLIC_819e4e/tfds/clic_edm_ttbar_pf
924M	/local/joosep/mlpf/clic/v1.2.4_key4hep_2025-05-29_CLIC_819e4e/tfds/clic_edm_ww_fullhad_pf

CLD, ~100k events per sample:

gen:
220G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_qq_ecm365
285G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_ttbar_ecm365
312G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_WW_ecm365
131G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_ZH_Htautau_ecm240
192G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_ZH_Htautau_ecm365
87G		/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_Z_qq_ecm91
41G		/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_Z_tautau_ecm91
234G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_ZZ_ecm365
99G		/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_ZZ_tautau_ecm240
140G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/gen/p8_ee_ZZ_tautau_ecm365

postprocessing:
898M	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/post/p8_ee_qq_ecm365
1.8G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/post/p8_ee_ttbar_ecm365
1.4G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/post/p8_ee_WW_ecm365
652M	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/post/p8_ee_ZH_Htautau_ecm240
683M	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/post/p8_ee_ZH_Htautau_ecm365
1.1G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/post/p8_ee_ZZ_ecm365
264M	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/post/p8_ee_ZZ_tautau_ecm240
272M	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/post/p8_ee_ZZ_tautau_ecm365

TFDS:
607M	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/tfds/cld_edm_qq_pf
1.2G	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/tfds/cld_edm_ttbar_pf
959M	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/tfds/cld_edm_ww_pf
710M	/local/joosep/mlpf/cld/v1.2.3_key4hep_2025-05-29_CLD_f1e8f9/tfds/cld_edm_zz_pf

jpata and others added 30 commits January 21, 2026 18:26
@jpata
set num splits
5224dc3
@jpata
snakemake
c707786
@jpata
clean up key4hep postprocessing
b0b11fe
@jpata
added simulation submodule
537aa59
@jpata
add cld to spec
139cecd
@jpata
updated key4hep-sim scripts
196a72f
@jpata
update paths
cd43516
@jpata
use apptainer
56f4ba2
@jpata
first run works
bd8d0de
@jpata
added snakemake files
1ba945c
@jpata
add mem for jobs
d04af3e
@jpata
refactor config
f27d805
@jpata
v1.2.3_key4hep_2025-05-29_CLD_f1e8f9
ed3d38e
@jpata
training works
c7de5bc
@jpata
batch mult
2a3d9da
@jpata
set tfds version
5f3f09f
@jpata
clean up datasets
75ea015
@jpata
added ww and zz datasets
b74e302
@jpata
added ww, zz datasets
4912db1
fixes for running the cld workflow
e9bc2a9
@jpata
unify and clean up cms gen scripts
d6de1f1
cms gen,post,tfds seems to run
e8eb29e
fixes for CMS workflow, update cld config to have Z 91
9789556
@jpata
cms and cld both work
9848b30
@jpata
grouping
754a17f
@jpata
add CLIC training
dc73b2d
improve runtime
c436bc9
added snakemake instructions
0f406c1
fix test path
12befb2
check paths
b95fdac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jpata