Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1022 commits
Select commit Hold shift + click to select a range
a85ac24
update hypertuning for tfds, add dqm plots
jpata Sep 9, 2021
49a1829
up
jpata Sep 10, 2021
f197db0
separate cms datasets
jpata Sep 10, 2021
791c068
propagate epochs
jpata Sep 10, 2021
f2b7806
fix hypertune, gun sample training
jpata Sep 10, 2021
f83e5d2
up
jpata Sep 10, 2021
5f04dde
update instructions
jpata Sep 10, 2021
2faa5f3
gen training update
jpata Sep 10, 2021
3a24b89
add timing script for ONNX
jpata Sep 11, 2021
dd20de2
update adv training
jpata Sep 12, 2021
541bad0
support for multiple datasets
jpata Sep 13, 2021
28ce8bc
clean outdated parameters
jpata Sep 13, 2021
f2d1531
fixes for delphes and cms pipeline
jpata Sep 13, 2021
06f3523
fix
jpata Sep 13, 2021
72ddab1
fix
jpata Sep 13, 2021
89741a3
fix
jpata Sep 13, 2021
ff9eb20
fix
jpata Sep 13, 2021
e32be49
update dataset
jpata Sep 14, 2021
b7c49b7
added additional variables
jpata Sep 14, 2021
63d582f
fixes to job gen
jpata Sep 14, 2021
164a0f7
add singletau
jpata Sep 14, 2021
a651523
configure pipeline
jpata Sep 14, 2021
2f6a8b9
fixes
jpata Sep 14, 2021
83c0c6d
Merge branch 'jpata_gen_training' of https://github.com/jpata/particl…
jpata Sep 14, 2021
639212e
up
jpata Sep 14, 2021
f9e26fb
script to run a small genration
jpata Sep 14, 2021
c6256d6
added tfds script
jpata Sep 14, 2021
94d1967
up
jpata Sep 14, 2021
ad85b52
up
jpata Sep 15, 2021
e0e9c61
up
jpata Sep 15, 2021
42ba326
baseline
jpata Sep 15, 2021
1a686e1
update radme
jpata Sep 15, 2021
5a9dafb
Merge pull request #79 from jpata/jpata_gen_training
jpata Sep 15, 2021
b19e054
Update README_cms.md
jpata Sep 15, 2021
84ea8b4
updated schematics
jpata Sep 20, 2021
3ecf965
Merge pull request #83 from jpata/schematis
jpata Sep 21, 2021
51c9137
Baseline MLPF model for CMS, PF group status 24.09.21 [TF] (#81)
jpata Sep 22, 2021
8350d92
Update README_cms.md
jpata Sep 22, 2021
41d4a6c
add license (Apache 2.0) (#86)
jmduarte Sep 30, 2021
0f7549f
hep_tfds support in raytune (#82)
erwulff Sep 30, 2021
547ce44
update with larger dataset 1.2.0 [TF] (#84)
jpata Oct 6, 2021
48724ac
Add raytune algorithms (#85)
erwulff Oct 7, 2021
a3b9be5
Minor raytune fixes (#87)
erwulff Oct 12, 2021
41640cd
Minor raytune fixes (#87)
erwulff Oct 12, 2021
c378b92
Load best weights before saving model after end on training (#88)
erwulff Oct 26, 2021
873002b
Load best weights before saving model after end on training (#88)
erwulff Oct 26, 2021
ffa4017
Add Bayesian Optimization to raytune command, using nevergrad or scik…
erwulff Oct 27, 2021
bb40804
Add Bayesian Optimization to raytune command, using nevergrad or scik…
erwulff Oct 27, 2021
6d34c39
ACAT2021 benchmark (#92)
jpata Nov 9, 2021
59336b6
ACAT2021 benchmark (#92)
jpata Nov 9, 2021
0b523da
New best hyperparameter config (#90)
erwulff Nov 9, 2021
e3d1922
New best hyperparameter config (#90)
erwulff Nov 9, 2021
4906868
rewrite einsum -> multiply (#93)
jmduarte Nov 16, 2021
d2edc95
rewrite einsum -> multiply (#93)
jmduarte Nov 16, 2021
cbdf030
merge (#94)
jpata Nov 16, 2021
a0310d1
merge (#94)
jpata Nov 16, 2021
f14be95
update Ray Tune version in github tests (#99)
erwulff Dec 7, 2021
2fcee73
update Ray Tune version in github tests (#99)
erwulff Dec 7, 2021
790ccdc
up
jpata Feb 15, 2022
ac4a533
up
jpata Feb 15, 2022
d2bc8ba
Merge branch 'master' of https://github.com/jpata/particleflow
jpata Feb 15, 2022
0ea1460
Merge branch 'master' of https://github.com/jpata/particleflow
jpata Feb 15, 2022
1d35056
Gen/Sim training dataset (#100)
jpata Apr 28, 2022
28626be
Gen/Sim training dataset (#100)
jpata Apr 28, 2022
3823800
Fix raytune imports (#106)
erwulff Apr 29, 2022
054a5f4
Fix raytune imports (#106)
erwulff Apr 29, 2022
48d8e2a
fix f16, transformer, cms dataset (#105)
jpata May 2, 2022
d473280
fix f16, transformer, cms dataset (#105)
jpata May 2, 2022
2e3bfb6
Fix output decoding for PFNetDense (#107)
jpata May 3, 2022
b57922d
Fix output decoding for PFNetDense (#107)
jpata May 3, 2022
3838c5b
up
jpata May 4, 2022
4df58a1
up
jpata May 4, 2022
002fee5
feat: Ray Tune analysis on JUWELS (#111)
erwulff May 6, 2022
d2cb12c
feat: Ray Tune analysis on JUWELS (#111)
erwulff May 6, 2022
ce83fba
Up flatiron modules (#112)
erwulff May 6, 2022
b0cd098
Up flatiron modules (#112)
erwulff May 6, 2022
e536b0f
fix optimizer save/load, add mpnn config, get f16 training to work (#…
jpata May 6, 2022
d51439b
fix optimizer save/load, add mpnn config, get f16 training to work (#…
jpata May 6, 2022
cf80664
updated/documented lrp and pytorch pipeline (#110)
farakiko May 6, 2022
9226111
updated/documented lrp and pytorch pipeline (#110)
farakiko May 6, 2022
3ef15ba
Fix PCgrad loading, trainable weights (#113)
jpata May 20, 2022
77ad63a
Fix PCgrad loading, trainable weights (#113)
jpata May 20, 2022
db9c37b
fix: broken quickstart notebook (#116)
erwulff Jun 16, 2022
d01d758
fix: broken quickstart notebook (#116)
erwulff Jun 16, 2022
2b21fa2
Comet-ml offline logging (#115)
erwulff Jun 16, 2022
49b94aa
Comet-ml offline logging (#115)
erwulff Jun 16, 2022
950ce04
optimized pytorch geometric pipeline using DDP (#118)
farakiko Jul 1, 2022
e940242
optimized pytorch geometric pipeline using DDP (#118)
farakiko Jul 1, 2022
4a58e17
Fix bug in CustomCallback class (#119)
erwulff Aug 9, 2022
6740faa
Fix bug in CustomCallback class (#119)
erwulff Aug 9, 2022
0de3bdf
Hypertuning development (#120)
erwulff Aug 9, 2022
fa8cc2f
Hypertuning development (#120)
erwulff Aug 9, 2022
06c6b4f
fix: remove unused Ray Tune code and update Ray Tune analysis (#121)
erwulff Aug 15, 2022
5f122b9
fix: remove unused Ray Tune code and update Ray Tune analysis (#121)
erwulff Aug 15, 2022
eb7dae4
June 2022 update: new datasets, jet/MET level validation, additional …
jpata Aug 17, 2022
4c6788f
June 2022 update: new datasets, jet/MET level validation, additional …
jpata Aug 17, 2022
de0aca5
Add multinode training using Horovod (#104)
MaPoKen Aug 22, 2022
be35760
Add multinode training using Horovod (#104)
MaPoKen Aug 22, 2022
51fd4a5
log jet/met reso, make event loss configurable, add sliced Wasserstei…
jpata Aug 22, 2022
4f87058
log jet/met reso, make event loss configurable, add sliced Wasserstei…
jpata Aug 22, 2022
2e595ec
fix small bug in eval (#127)
jpata Aug 23, 2022
91688f4
fix small bug in eval (#127)
jpata Aug 23, 2022
ac15a9d
Gen jet loss (#126)
jmduarte Aug 24, 2022
ef9ea15
Gen jet loss (#126)
jmduarte Aug 24, 2022
b1cb537
Faster test, pre-commit formatting, general cleanup (#129)
jpata Aug 24, 2022
5ab55a7
Faster test, pre-commit formatting, general cleanup (#129)
jpata Aug 24, 2022
f0f861e
Pre commit fixes (#131)
farakiko Aug 25, 2022
84e5c23
Pre commit fixes (#131)
farakiko Aug 25, 2022
3b972dc
Comparison job for different event losses (#132)
jpata Aug 29, 2022
cb66adf
Comparison job for different event losses (#132)
jpata Aug 29, 2022
ec3d0e3
fix: LR logging in CustomTensorBoard with PCGrad (#137)
erwulff Sep 2, 2022
296edaa
fix: LR logging in CustomTensorBoard with PCGrad (#137)
erwulff Sep 2, 2022
fb89d79
integrate hep_tfds, September 2022 benchmark training (#136)
jpata Sep 2, 2022
05e14e8
integrate hep_tfds, September 2022 benchmark training (#136)
jpata Sep 2, 2022
0a350bc
MET loss as an option (#138)
jpata Sep 12, 2022
4429a9e
MET loss as an option (#138)
jpata Sep 12, 2022
4e4ce6f
added MET file (#139)
jpata Sep 13, 2022
25dd3d0
added MET file (#139)
jpata Sep 13, 2022
34d2c44
Fix MET loss, validation in CMSSW (#141)
jpata Sep 13, 2022
b0990c5
Fix MET loss, validation in CMSSW (#141)
jpata Sep 13, 2022
5780727
Bump tensorflow from 2.9 to 2.9.1 (#143)
dependabot[bot] Sep 17, 2022
ba0013d
Bump tensorflow from 2.9 to 2.9.1 (#143)
dependabot[bot] Sep 17, 2022
deb05ea
Ray Tune checkpointing fix, allow LR schedules for non-PCGrad opt, an…
erwulff Sep 20, 2022
49cba23
Ray Tune checkpointing fix, allow LR schedules for non-PCGrad opt, an…
erwulff Sep 20, 2022
fc0e482
PCGrad with LR schedules, resume from checkpoint with LR schedules (#…
erwulff Sep 22, 2022
b2434b9
PCGrad with LR schedules, resume from checkpoint with LR schedules (#…
erwulff Sep 22, 2022
1d05100
Add ability to train on Habana Gaudi (#135)
jmduarte Oct 17, 2022
fd89e3e
Add ability to train on Habana Gaudi (#135)
jmduarte Oct 17, 2022
5d09ac3
high-pT gun samples (#144)
jpata Oct 22, 2022
45593c5
high-pT gun samples (#144)
jpata Oct 22, 2022
aae4f55
Create faq.md
jpata Oct 28, 2022
c6907d4
Create faq.md
jpata Oct 28, 2022
6bdcc33
Update faq.md
jpata Oct 28, 2022
579f343
Update faq.md
jpata Oct 28, 2022
61516f2
Move padding from dataset to model, change response plot definition, …
jpata Nov 17, 2022
e9cab1b
Move padding from dataset to model, change response plot definition, …
jpata Nov 17, 2022
111c52c
format
jpata Nov 18, 2022
a0d8cbe
format
jpata Nov 18, 2022
73803b6
Add benchmarking utilities (#147)
erwulff Nov 18, 2022
ec060cc
Add benchmarking utilities (#147)
erwulff Nov 18, 2022
ae32561
added clic pipeline from parquet (#149)
jpata Nov 18, 2022
8f5c515
added clic pipeline from parquet (#149)
jpata Nov 18, 2022
f74c421
added acat2022 model (#148)
jpata Nov 18, 2022
1089c9a
added acat2022 model (#148)
jpata Nov 18, 2022
2d0ea76
fix num_cpus flag (#150)
jpata Nov 21, 2022
266fa00
fix num_cpus flag (#150)
jpata Nov 21, 2022
3388f5c
Bump tensorflow from 2.10.0 to 2.10.1 (#151)
dependabot[bot] Nov 22, 2022
27fa930
Bump tensorflow from 2.10.0 to 2.10.1 (#151)
dependabot[bot] Nov 22, 2022
c143129
training on LUMI HPC (#152)
jpata Nov 22, 2022
1082c25
training on LUMI HPC (#152)
jpata Nov 22, 2022
7588419
Update README.md
jpata Nov 23, 2022
cf6285c
Update README.md
jpata Nov 23, 2022
24a1ad7
Refactoring, CLIC datasets (#153)
jpata Jan 6, 2023
84e3537
Refactoring, CLIC datasets (#153)
jpata Jan 6, 2023
716e9e3
format black (#154)
jpata Jan 11, 2023
8cca110
format black (#154)
jpata Jan 11, 2023
3e6e03b
ssl-based mlpf first iteration (#158)
farakiko Jan 12, 2023
5a84666
ssl-based mlpf first iteration (#158)
farakiko Jan 12, 2023
9f24bca
Fix legacy CLIC dataset pdgid (#160)
jpata Jan 16, 2023
fa2bd56
Fix legacy CLIC dataset pdgid (#160)
jpata Jan 16, 2023
9ec3162
edm4hep postprocessing (#159)
jpata Jan 24, 2023
a4e9792
edm4hep postprocessing (#159)
jpata Jan 24, 2023
81a72c6
SSL updates: pipeline, new datasets and jet clustering (#161)
farakiko Jan 25, 2023
516011c
SSL updates: pipeline, new datasets and jet clustering (#161)
farakiko Jan 25, 2023
0d2d997
tune the pytorch mlpf model to be more similar to tf (#165)
jpata Jan 26, 2023
0153d77
tune the pytorch mlpf model to be more similar to tf (#165)
jpata Jan 26, 2023
57395b2
Raytune updates, LR-finder bug fix (#164)
erwulff Feb 6, 2023
9def3e1
Raytune updates, LR-finder bug fix (#164)
erwulff Feb 6, 2023
4846e25
Feb 2023: tuning the downstream MLPF model [pyg] (#166)
jpata Feb 6, 2023
12f6c02
Feb 2023: tuning the downstream MLPF model [pyg] (#166)
jpata Feb 6, 2023
aaa1146
Ssl finetuning (#167)
farakiko Feb 6, 2023
d5883b9
Ssl finetuning (#167)
farakiko Feb 6, 2023
0f5e42e
Update data split mode for SSL studies [pytorch] (#168)
farakiko Feb 8, 2023
8449e60
Update data split mode for SSL studies [pytorch] (#168)
farakiko Feb 8, 2023
3d2e5dd
fix: error in raytune search space
erwulff Feb 14, 2023
1ff6c15
few fixes and cleanups to pytorch, update sim scripts (#169)
jpata Feb 15, 2023
087e9f3
few fixes and cleanups to pytorch, update sim scripts (#169)
jpata Feb 15, 2023
f9e6251
update clic plots (#171)
jpata Feb 15, 2023
5762af0
update clic plots (#171)
jpata Feb 15, 2023
67c9b32
fix: error in raytune search space (#170)
erwulff Feb 15, 2023
6ca85cb
fix: error in raytune search space (#170)
erwulff Feb 15, 2023
fc75328
optimizing VICReg (#173)
farakiko Mar 1, 2023
daf1292
optimizing VICReg (#173)
farakiko Mar 1, 2023
6ec93cf
update CLIC dataset, retrain MLPF (#172)
jpata Mar 6, 2023
7a7c96d
update CLIC dataset, retrain MLPF (#172)
jpata Mar 6, 2023
efc7bb4
update README (#175)
jpata Mar 6, 2023
e3a2ecb
update README (#175)
jpata Mar 6, 2023
e22f770
Refactoring (#174)
farakiko Mar 6, 2023
96a299b
Refactoring (#174)
farakiko Mar 6, 2023
0b556d2
clean up dataset prep (#176)
jpata Mar 7, 2023
8fb6b14
clean up dataset prep (#176)
jpata Mar 7, 2023
8f02bf3
Merge branch 'jpata:main' into master
erwulff Mar 8, 2023
56ca1b7
fix loader, readd tensorboard (#177)
jpata Mar 8, 2023
dfba0f6
fix loader, readd tensorboard (#177)
jpata Mar 8, 2023
1e24b91
Additional small fixes to pytorch (#178)
jpata Mar 8, 2023
07f17b0
Additional small fixes to pytorch (#178)
jpata Mar 8, 2023
295bcc8
Merge branch 'jpata:main' into master
erwulff Mar 9, 2023
627b2a8
standardize input features, re-enable fp16 [TF], unify plotting [pyto…
jpata Mar 15, 2023
12020c4
standardize input features, re-enable fp16 [TF], unify plotting [pyto…
jpata Mar 15, 2023
9b4d452
Update test.yml (#180)
jpata Mar 16, 2023
cad6997
Update test.yml (#180)
jpata Mar 16, 2023
a281e7d
Merge branch 'jpata:main' into master
erwulff Mar 23, 2023
2feca8b
CLIC new samples with 1M events (#181)
jpata Mar 24, 2023
f033066
CLIC new samples with 1M events (#181)
jpata Mar 24, 2023
4203bad
Merge branch 'jpata:main' into master
erwulff Mar 29, 2023
b0bd7d6
CLIC new datasets, hit based training option (#182)
jpata May 2, 2023
b07650c
CLIC new datasets, hit based training option (#182)
jpata May 2, 2023
5af04d4
Merge branch 'jpata:main' into master
erwulff May 8, 2023
13dd72e
update hit-based training (#184)
jpata Jul 6, 2023
df727ff
update hit-based training (#184)
jpata Jul 6, 2023
8b32e25
Merge branch 'jpata:main' into master
erwulff Jul 19, 2023
afcf0fe
TF perf tuning, CLIC benchmarks, flatiron scripts (#185)
erwulff Jul 28, 2023
e302298
TF perf tuning, CLIC benchmarks, flatiron scripts (#185)
erwulff Jul 28, 2023
4d270bd
Add inference command (#187)
erwulff Jul 28, 2023
d707877
Add inference command (#187)
erwulff Jul 28, 2023
4f05d54
Merge branch 'jpata:main' into master
erwulff Aug 2, 2023
0e12a52
clean up repo (#188)
jpata Aug 4, 2023
057d594
clean up repo (#188)
jpata Aug 4, 2023
0f95ad5
Merge branch 'jpata:main' into master
erwulff Aug 8, 2023
f10197d
scale test on lumi, fix horovod (#189)
jpata Aug 10, 2023
78a564b
scale test on lumi, fix horovod (#189)
jpata Aug 10, 2023
235e991
Merge branch 'jpata:main' into master
erwulff Aug 11, 2023
ac4c6f3
switch to tfds array_record, improve visualization, dataset descripti…
jpata Aug 28, 2023
279f6ae
switch to tfds array_record, improve visualization, dataset descripti…
jpata Aug 28, 2023
410cb99
new scaling test on LUMI (#193)
jpata Aug 30, 2023
31ed5b0
new scaling test on LUMI (#193)
jpata Aug 30, 2023
c63febf
updates for habana v1.9.0 image and multi-card training (#186)
jmduarte Aug 30, 2023
6cf14bc
updates for habana v1.9.0 image and multi-card training (#186)
jmduarte Aug 30, 2023
f5631fc
Merge branch 'jpata:main' into master
erwulff Sep 1, 2023
5776798
add option to pad everything to a constant size (#195)
jpata Sep 1, 2023
e2464ef
add option to pad everything to a constant size (#195)
jpata Sep 1, 2023
4ddc37e
integer batch multiplier (#196)
jpata Sep 2, 2023
3e94e2c
integer batch multiplier (#196)
jpata Sep 2, 2023
816d7e1
Merge branch 'jpata:main' into master
erwulff Sep 4, 2023
3aa6c2d
update habana reqs for new tensorflow datasets (#194)
jmduarte Sep 8, 2023
fd259df
update habana reqs for new tensorflow datasets (#194)
jmduarte Sep 8, 2023
8a66699
Clic visualization (#198)
farakiko Sep 11, 2023
b1d6eb5
Clic visualization (#198)
farakiko Sep 11, 2023
dc35629
fix keynotes (#199)
jpata Sep 12, 2023
51d3476
fix keynotes (#199)
jpata Sep 12, 2023
c3ed0ed
Merge branch 'jpata:main' into master
erwulff Sep 13, 2023
687673f
Optimize and cache data steps counter (#197)
jpata Sep 13, 2023
11b2a4a
remove models dir
jpata Sep 15, 2023
981cf97
pin matplotlib due to breaking changes in 3.8.0 (#210)
jpata Sep 15, 2023
5a8f685
Update README_tf.md (#209)
jpata Sep 15, 2023
2dc9241
Update README.md (#213)
jpata Sep 19, 2023
de4dcca
WIP on torch GNN-LSH model (#211)
jpata Sep 22, 2023
bab3205
small fixes for CMS training, torch onnx export (#215)
jpata Sep 22, 2023
f06f5c1
Merge branch 'main' of github.com:jpata/particleflow into jpata-main
erwulff Sep 22, 2023
69e3d51
Merge pull request #11 from erwulff/jpata-main
erwulff Sep 22, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
27 changes: 27 additions & 0 deletions .github/workflows/pre-commit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: Run pre-commit

on:
pull_request:
branches: [ main ]
push:
branches: [ main ]

jobs:
lint:
name: Lint PR or Push to main
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: [3.8.10]

steps:
- name: Checkout
uses: actions/checkout@v2

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}

- name: Run Lint
uses: pre-commit/action@v2.0.0
206 changes: 149 additions & 57 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,70 +2,162 @@ name: CI

on:
push:
branches: [ master ]
branches: [ main ]
pull_request:
branches: [ master ]
branches: [ main ]

workflow_dispatch:

jobs:
delphes-pipeline:
runs-on: ubuntu-latest
deps:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2
- name: Install python deps
run: |
sudo apt install python3 python3-pip wget
sudo python3 -m pip install --upgrade pip
sudo python3 -m pip install --upgrade setuptools
sudo python3 -m pip install tensorflow==2.6 setGPU \
sklearn matplotlib mplhep pandas scipy uproot3 uproot3-methods \
awkward0 keras-tuner networkx \
tensorflow-probability==0.12.2 tensorflow-addons==0.13.0 \
tqdm click tensorflow-datasets 'ray[default]' 'ray[tune]' \
tf-models-official tensorflow-text \
tf2onnx onnxruntime zenodo_get seaborn
git submodule init
git submodule update
- name: Run delphes TF model
run: ./scripts/local_test_delphes_pipeline.sh
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
deps-pyg:
runs-on: ubuntu-20.04
needs: [deps]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: pip3 install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
- run: pip install pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-2.0.1+cpu.html

tf-unittest:
runs-on: ubuntu-22.04
needs: [deps]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: ./scripts/local_test_cms_pipeline.sh
- run: PYTHONPATH=. python3 -m unittest tests/test_tf.py

tf-clic-pipeline:
runs-on: ubuntu-22.04
needs: [tf-unittest]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: ./scripts/local_test_clic_pipeline.sh

tf-clic-hits-pipeline:
runs-on: ubuntu-22.04
needs: [tf-unittest]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: ./scripts/local_test_clic_hits_pipeline.sh

tf-delphes-pipeline:
runs-on: ubuntu-22.04
needs: [tf-unittest]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: ./scripts/local_test_delphes_pipeline.sh

tf-cms-pipeline:
runs-on: ubuntu-22.04
needs: [tf-unittest]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: ./scripts/local_test_cms_pipeline.sh

pyg-unittests:
runs-on: ubuntu-22.04
needs: [deps-pyg]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: pip3 install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
- run: pip install pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-2.0.1+cpu.html
- run: PYTHONPATH=. python3 -m unittest tests/test_torch_and_tf.py

pyg-cms-pipeline:
runs-on: ubuntu-22.04
needs: [pyg-unittests]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: pip3 install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
- run: pip install pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-2.0.1+cpu.html
- run: ./scripts/local_test_pyg_cms.sh

pyg-delphes-pipeline:
runs-on: ubuntu-22.04
needs: [pyg-unittests]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: pip3 install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
- run: pip install pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-2.0.1+cpu.html
- run: ./scripts/local_test_pyg_delphes.sh

cms-pipeline:
runs-on: ubuntu-latest
pyg-clic-pipeline:
runs-on: ubuntu-22.04
needs: [pyg-unittests]
steps:
- uses: actions/checkout@v2
- name: Install python deps
run: |
sudo apt install python3 python3-pip wget
sudo python3 -m pip install --upgrade pip
sudo python3 -m pip install --upgrade setuptools
sudo python3 -m pip install tensorflow==2.6 setGPU \
sklearn matplotlib mplhep pandas scipy uproot3 uproot3-methods \
awkward0 keras-tuner networkx \
tensorflow-probability==0.12.2 tensorflow-addons==0.13.0 \
tqdm click tensorflow-datasets 'ray[default]' 'ray[tune]' \
tf-models-official tensorflow-text \
tf2onnx onnxruntime zenodo_get seaborn
git submodule init
git submodule update
- name: Run CMS TF model using the pipeline
run: ./scripts/local_test_cms_pipeline.sh
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: pip3 install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
- run: pip install pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-2.0.1+cpu.html
- run: ./scripts/local_test_pyg_clic.sh

delphes-pytorch:
runs-on: ubuntu-latest
pyg-ssl-pipeline:
runs-on: ubuntu-22.04
needs: [pyg-unittests]
steps:
- uses: actions/checkout@v2
- name: Install python deps
run: |
sudo apt install python3 python3-pip wget
sudo python3 -m pip install --upgrade pip
sudo python3 -m pip install --upgrade setuptools
sudo python3 -m pip install setGPU sklearn matplotlib mplhep pandas scipy uproot3 uproot3-methods awkward0 networkx
sudo python3 -m pip install torch==1.6.0 torchvision==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
sudo python3 -m pip install --no-index torch-scatter -f https://pytorch-geometric.com/whl/torch-1.6.0+cpu.html
sudo python3 -m pip install --no-index torch-sparse -f https://pytorch-geometric.com/whl/torch-1.6.0+cpu.html
sudo python3 -m pip install --no-index torch-cluster -f https://pytorch-geometric.com/whl/torch-1.6.0+cpu.html
sudo python3 -m pip install --no-index torch-spline-conv -f https://pytorch-geometric.com/whl/torch-1.6.0+cpu.html
sudo python3 -m pip install torch-geometric comet_ml
- name: Run Delphes Pytorch model
run: ./scripts/local_test_delphes_pytorch.sh
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip'
- run: pip install -r requirements.txt
- run: pip3 install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
- run: pip install pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-2.0.1+cpu.html
- run: ./scripts/local_test_pyg_ssl.sh
32 changes: 22 additions & 10 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,34 @@
*.npz
*.pt
*.pdf
*.png
data/*
experiments/*
prp/*
*.pth
test/__pycache__/

mlpf/pytorch/__pycache__/*
mlpf/plotting/__pycache__/*
mlpf/pytorch/data
test_tmp/
test_tmp_delphes/
*/__pycache__/*
.DS_Store

prp
*.pyc
*.pyo

mlpf/updated/LRP/pid*
mlpf/updated/LRP/class*

*.ipynb_checkpoints

*playground.py
*playground.ipynb
nohup.out

*.pkl
*.pkl.bz2
*.parquet

slurm-*.out

.vscode

models

*.root

logs
3 changes: 0 additions & 3 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +0,0 @@
[submodule "hep_tfds"]
path = hep_tfds
url = https://github.com/jpata/hep_tfds
49 changes: 49 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
default_language_version:
python: python3
exclude: ^(scripts/delphes/tev14_pythia8_*.)|^(images)|^(scripts/fcc/clicRec_e4h_input.py)|^(scripts/fcc/clic_steer.py)|^(scripts/fcc/pythia.py)|^(scripts/fcc/PandoraSettings/.*)

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: trailing-whitespace
- id: check-added-large-files
- id: check-ast
- id: check-json
- id: check-merge-conflict
- id: check-xml
- id: check-yaml
- id: debug-statements
- id: end-of-file-fixer
- id: requirements-txt-fixer
- id: mixed-line-ending
args: ['--fix=no']

#This is giving different results on github CI vs. locally and I can't understand why
#- repo: https://github.com/PyCQA/isort
# rev: 5.11.2
# hooks:
# - id: isort
# args: ['--profile', 'black', '--filter-files']

- repo: https://github.com/psf/black
rev: 22.12.0
hooks:
- id: black
# It is recommended to specify the latest version of Python
# supported by your project here, or alternatively use
# pre-commit's default_language_version, see
# https://pre-commit.com/#top_level-default_language_version
language_version: python3
args: [--line-length=125]

- repo: https://github.com/PyCQA/flake8
rev: 6.0.0
hooks:
- id: flake8
# black-compatible flake-8 config

# E203 is not PEP8 compliant
# E402 due to logging.basicConfig in pipeline.py
args: ['--max-line-length=125', # github viewer width
'--extend-ignore=E203,E402,W605']
40 changes: 24 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,36 @@
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4559587.svg)](https://doi.org/10.5281/zenodo.4559587)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4559324.svg)](https://doi.org/10.5281/zenodo.4559324)
[![CI](https://github.com/jpata/particleflow/workflows/CI/badge.svg)](https://github.com/jpata/particleflow/actions)

# Overview
MLPF focuses on developing full event reconstruction based on computationally scalable and flexible end-to-end ML models.

<p float="left">
<img src="notebooks/plots/event.png" alt="Simulated event" width="600"/>
<img src="images/schematic.png" alt="High-level overview" width="600"/>
</p>

## MLPF development in CMS

<p float="left">
<img src="notebooks/plots/num_particles.png" alt="Particle multiplicity" width="300"/>
<img src="notebooks/plots/res_pid2.png" alt="Neutral hadron resolution" width="300"/>
<img src="images/cms/ev_pf.png" alt="PF reconstruction" width="300"/>
<img src="images/cms/ev_mlpf.png" alt="MLPF reconstruction" width="300"/>
</p>

## MLPF with Delphes

Short instructions with a single test file in [notebooks/delphes-tf-mlpf-quickstart.ipynb](notebooks/delphes-tf-mlpf-quickstart.ipynb).
<p float="left">
<img src="images/cms/ak4jet_puppi_pt_ttbar.png" alt="PUPPI jets in ttbar" width="300"/>
</p>

Long instructions for reproducing the full training from scratch in [README_delphes.md](README_delphes.md).
The plots can be generated using the notebook [delphes/delphes_model_analysis.ipynb](delphes/delphes_model_analysis.ipynb).
- ACAT 2022:
- CERN-CMS-DP-2022-061, http://cds.cern.ch/record/2842375
- ACAT 2021:
- J. Phys. Conf. Ser. 2438 012100, http://dx.doi.org/10.1088/1742-6596/2438/1/012100
- CERN-CMS-DP-2021-030, https://cds.cern.ch/record/2792320

### Delphes dataset
The dataset is available from zenodo: https://doi.org/10.5281/zenodo.4559324.
## Initial development with Delphes

### Software setup
The software setup for the ML training is available in the singularity spec file [scripts/base.singularity](scripts/base.singularity).
<p float="left">
<img src="images/delphes/num_particles.png" alt="Number of reconstructed particles" width="250"/>
<img src="images/delphes/inference_time.png" alt="Scaling of the inference time" width="300"/>
</p>

## Acknowledgements
This project is supported by the Mobilitas Pluss Returning Researcher Grant MOBTP187 of the Estonian Research Council. Part of this work was conducted at **iBanks**, the AI GPU cluster at Caltech. We acknowledge NVIDIA, SuperMicro and the Kavli Foundation for their support of **iBanks**.
- paper: https://doi.org/10.1140/epjc/s10052-021-09158-w
- code: https://doi.org/10.5281/zenodo.4559587
- dataset: https://doi.org/10.5281/zenodo.4559324
Loading