update MkFit DNN

[minxiyang]

Update MkFit DNN for the newest update of the MkFit reconstruction: cms-sw/cmssw#36246.

The performance of the new DNN: https://jschulte.web.cern.ch/jschulte/DNN_Nov29th/run3finalv4_TTbar/

The test is ran on the dataset /RelValTTbar_14TeV/CMSSW_12_1_0_pre5-PU_121X_mcRun3_2021_realistic_v15-v2/GEN-SIM-DIGI-RAW with the CMSSW _12_2_X. The performance of the MkFit DNN on the MkFit tracks is compared to the CKF DNN on the CKF dataset.

[cmsbuild]

A new Pull Request was created by @minxiyang (Minxi Yang) for branch master.

@smuzaffar, @iarspider, @cmsbuild, @slava77, @jpata, @ddaina can you please review it and eventually sign? Thanks.
@felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @ebrondol, @gpetruc, @mmusich, @mtosi, @dgulhan this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

[mmusich]

@minxiyang

The performance of the new DNN: https://jschulte.web.cern.ch/jschulte/DNN_Nov29th/run3finalv4_TTbar/

do you have also the comparison with the current mkFit + DNN tuning?

[minxiyang]

@minxiyang

The performance of the new DNN: https://jschulte.web.cern.ch/jschulte/DNN_Nov29th/run3finalv4_TTbar/

do you have also the comparison with the current mkFit + DNN tuning?

I don't have have the comparison of this update and the previous mkFIt + DNN tuning yet. Do you want the add a new result of the previous MkFit DNN with the previous MkFit reconstruction on the current comparison?
cms-sw/cmssw#36285. In here I update the new working point.

[mmasciov]

@mmusich, @minxiyang: to clarify, the plots that I have shown in today's TRK POG presentation already used the updated WPs on top of this updated DNN, with only one difference for pixelLessStep, where we used slightly adjusted cuts wrt. cms-sw/cmssw#36285.
Will reproduce the results with pixelLess WPs as in cms-sw/cmssw#36285, and compare to our adjusted ones.

[mmasciov]

@mmusich, @minxiyang: to clarify, the plots that I have shown in today's TRK POG presentation already used the updated WPs on top of this updated DNN, with only one difference for pixelLessStep, where we used slightly adjusted cuts wrt. cms-sw/cmssw#36285. Will reproduce the results with pixelLess WPs as in cms-sw/cmssw#36285, and compare to our adjusted ones.

@minxiyang, @mmusich, @slava77:
here are MTV results comparing the WP on top of the updated DNN for pixelLessStep, in TTbar events with =50 (same as shown in today's TRK POG presentation):
https://mmasciovecchio.web.cern.ch/BTV_Nov2021/MTV_TTbarPU50_mkFit_for1220pre3_DNNWP/

Legend:

• Blue = Run3_noMkFit (i.e., CKF);
• Red = last mkFit (as in Update mkFit for CMSSW_12_2_0_pre3 cms-sw/cmssw#36246) with old DNN;
• Black = last mkFit with new DNN as in this PR and WPs as in update working point for new DNN cms-sw/cmssw#36285;
• Orange = same as black, with better tuned WP for pixelLessStep.

Note: in this context, we mostly care of orange vs. black.

Difference is visible in high-purity tracks: https://mmasciovecchio.web.cern.ch/BTV_Nov2021/MTV_TTbarPU50_mkFit_for1220pre3_DNNWP/plots_highPurity/effandfakePtEtaPhi.pdf
-> Orange allows to retain the same efficiency, while reducing fakes+duplicates (in pixelLessStep).
The only suggested change would be, at https://github.com/cms-sw/cmssw/pull/36285/files#diff-143a81f7a4341ac338a7dabe13c4a49a4510ca142d099ba9960be3fb3b96762fR12:
PixelLessStep = cms.vdouble(-0.6, -0.5, -0.4)
-->
PixelLessStep = cms.vdouble(-0.60, -0.40, 0.02)

[minxiyang]

@mmusich, @minxiyang: to clarify, the plots that I have shown in today's TRK POG presentation already used the updated WPs on top of this updated DNN, with only one difference for pixelLessStep, where we used slightly adjusted cuts wrt. cms-sw/cmssw#36285. Will reproduce the results with pixelLess WPs as in cms-sw/cmssw#36285, and compare to our adjusted ones.

@minxiyang, @mmusich, @slava77: here are MTV results comparing the WP on top of the updated DNN for pixelLessStep, in TTbar events with =50 (same as shown in today's TRK POG presentation): https://mmasciovecchio.web.cern.ch/BTV_Nov2021/MTV_TTbarPU50_mkFit_for1220pre3_DNNWP/

Legend:

• Blue = Run3_noMkFit (i.e., CKF);
• Red = last mkFit (as in Update mkFit for CMSSW_12_2_0_pre3 cms-sw/cmssw#36246) with old DNN;
• Black = last mkFit with new DNN as in this PR and WPs as in update working point for new DNN cms-sw/cmssw#36285;
• Orange = same as black, with better tuned WP for pixelLessStep.

Note: in this context, we mostly care of orange vs. black.

Difference is visible in high-purity tracks: https://mmasciovecchio.web.cern.ch/BTV_Nov2021/MTV_TTbarPU50_mkFit_for1220pre3_DNNWP/plots_highPurity/effandfakePtEtaPhi.pdf -> Orange allows to retain the same efficiency, while reducing fakes+duplicates (in pixelLessStep). The only suggested change would be, at https://github.com/cms-sw/cmssw/pull/36285/files#diff-143a81f7a4341ac338a7dabe13c4a49a4510ca142d099ba9960be3fb3b96762fR12: PixelLessStep = cms.vdouble(-0.6, -0.5, -0.4) --> PixelLessStep = cms.vdouble(-0.60, -0.40, 0.02)

Hi Slava, thanks. I update working point for pixelLess track.

[slava77]

+reconstruction

tested with cms-sw/cmssw#36285

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-34efa2/20917/summary.html
COMMIT: d42fa12
CMSSW: CMSSW_12_2_X_2021-12-01-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-data/RecoTracker-FinalTrackSelectors/11/20917/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 17257 differences found in the comparisons
• DQMHistoTests: Total files compared: 41
• DQMHistoTests: Total histograms compared: 3041955
• DQMHistoTests: Total failures: 23380
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3018553
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 40 files compared)
• Checked 175 log files, 37 edm output root files, 41 DQM output files
• TriggerResults: no differences found

[perrotta]

+1

• It has to go together with #36285

[perrotta]

merge