Updating EMTF NN to v1.0.1

[pallabidas]

EMTF NN model v1.0.1

model release: https://github.com/cms-hls4ml/EMTF_NN/releases/tag/v1.0.1

[cmsbuild]

A new Pull Request was created by @pallabidas for branch IB/CMSSW_16_0_X/master.

@akritkbehera, @cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-339024/49422/summary.html
COMMIT: 250f072
CMSSW: CMSSW_16_0_X_2025-11-12-1200/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10191/49422/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 96 lines to the logs
• Reco comparison results: 7 differences found in the comparisons
• Reco comparison had 2 failed jobs
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3880468
• DQMHistoTests: Total failures: 16002
• DQMHistoTests: Total nulls: 14
• DQMHistoTests: Total successes: 3864432
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 218 log files, 188 edm output root files, 51 DQM output files
• TriggerResults: found differences in 4 / 49 workflows

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-339024/49440/summary.html
COMMIT: 250f072
CMSSW: CMSSW_16_0_X_2025-11-13-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10191/49440/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 60 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 13 differences found in the comparisons
• Reco comparison had 2 failed jobs
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3880468
• DQMHistoTests: Total failures: 15331
• DQMHistoTests: Total nulls: 10
• DQMHistoTests: Total successes: 3865107
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 218 log files, 188 edm output root files, 51 DQM output files
• TriggerResults: found differences in 4 / 49 workflows

[smuzaffar]

@pallabidas , any idea why we get over 15K DQM comparison failures (https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-13-1100+339024/71714/validateJR.html )

[pallabidas]

hi @smuzaffar is the failure from this PR? I only added the namespace as suggested in cms-sw/cmssw#49325 , you can see the changes here: cms-hls4ml/EMTF_NN@279f6b5

[smuzaffar]

please test

@pallabidas , I ran PR tests twice and botjh times there were over 15K DQM differences. I have started tests here again and also at #10193 to see if I get these difference in both tests or not

[pallabidas]

ok, also asking @thesps if you have seen this type of behaviour before?

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-339024/49457/summary.html
COMMIT: 250f072
CMSSW: CMSSW_16_0_X_2025-11-13-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10191/49457/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 57 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 8 differences found in the comparisons
• Reco comparison had 2 failed jobs
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3880468
• DQMHistoTests: Total failures: 15320
• DQMHistoTests: Total nulls: 10
• DQMHistoTests: Total successes: 3865118
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 218 log files, 188 edm output root files, 51 DQM output files
• TriggerResults: found differences in 4 / 49 workflows

[smuzaffar]

@pallabidas , I again see DQM differences here while tests for #10193 (comment) show no differences I am afraid the DQM changes are related to this chage

[artlbv]

the good news though is that the Phase-2 Tau P2GT results seem to be getting fixed:

e.g. in https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_16_0_X_2025-11-12-1200+339024/71692/triggerResults/29834.999_TTbar_14TeV+Run4D110PU_PMXS1S2PR/HLT.log

Found 10 matching events, out of which 6 have different HLT results

      Events    Accepted      Gained        Lost       Other  Trigger
          10           8           -          -6           -  pDoublePuppiTau52_52
          10           2           -           -          ~6  HLT_DoubleMediumChargedIsoPFTauHPS40_eta2p1
          10           1           -           -          ~6  HLT_DoubleMediumDeepTauPFTauHPS35_eta2p1

[cmsbuild]

Pull request #10191 was updated.

[pallabidas]

I've updated the package tag to v1.0.2 now.

[mmusich]

@cmsbuild, please test

[artlbv]

The same kind of changes (perhaps reversed) were observed also in PR

ok that explains it! thank you Marco!

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-339024/49465/summary.html
COMMIT: c7d5716
CMSSW: CMSSW_16_0_X_2025-11-14-1100/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10191/49465/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @cms-sw Update tag for PhysicsTools-NanoAOD to V01-07-00 #10197

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-339024/49465/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-339024/49465/git-merge-result

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test TestIOPoolInputNoParentDictionary had ERRORS

Comparison Summary

Summary:

• You potentially added 1637 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 59 differences found in the comparisons
• Reco comparison had 2 failed jobs
• DQMHistoTests: Total files compared: 68
• DQMHistoTests: Total histograms compared: 4352785
• DQMHistoTests: Total failures: 262232
• DQMHistoTests: Total nulls: 144
• DQMHistoTests: Total successes: 4090389
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 11975.365 KiB( 67 files compared)
• DQMHistoSizes: changed ( 16834.0,... ): 2993.495 KiB HLT/ScoutingOffline
• DQMHistoSizes: changed ( 29634.0,... ): 0.277 KiB MTD/ETL
• Checked 275 log files, 231 edm output root files, 68 DQM output files
• TriggerResults: found differences in 21 / 66 workflows

[mmusich]

-1

failures are unrelated.

Now why that particular workflow 29634.75_TTbar_14TeV+Run4D110_HLT75e33Timing seems to be very affected while others don't its not clear to me at the moment.

looking at the matrix of DQM bin-by-bin comparisons it is clear that all the workflows that use HLT:75e33_timing as opposed HLT:75e33 are much more affected.
This is understood because the "full" menu HLT:75e33 employs also "MC" paths that do not depend on the L1T accepts, but get executed at every event:

https://github.com/cms-sw/cmssw/blob/8e2bf45689eebdeb9232cfc63b5b24cb343aba7d/HLTrigger/Configuration/python/HLT_75e33_cff.py#L370-L374

whereas the timing menu does not link, and so the effective list of event products available in the event as coming from the HLT reconstruction strongly depends on the L1T accepts.

[mmusich]

urgent

• to fix broken Phase-2 L1T in IBs (@rovere @VourMa FYI)

[mmusich]

@smuzaffar please integrate this bug-fix at earliest convenience

[mmusich]

type bug-fix

[smuzaffar]

+externals

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_16_0_X/master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @ftenchini, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)