[15.1.X] Pixel GPU Online DQM Client changes for CMSHLT-3147

[mmusich]

backport of #49237

PR description:

This PR is opened in preparation of the integration of the ticket CMSHLT-3147.
In a nutshell that ticket suggests that the DQM plugins which are currently in the Path DQM_PixelReconstruction_v (Sequence: HLTDQMPixelReconstruction) and in the Path DQM_PixelReconstruction_v (Sequece HLTDQMPixelReconstruction) can be removed from the HLT combined table, as long as the Tracker-DPG and/or Tracking-POG experts move the corresponding monitoring sequence into the appropriate online-DQM client (which reads the DQMGPUvsCPU streamer files).
This latter point is proposed in this PR, in which we change the client DQM/Integration/python/clients/pixelgpu_dqm_sourceclient-live_cfg.py to include directly the modules responsible of monitoring of the GPU vs CPU comparisons, as the input data to feed them will be sent by the input DQMGPUvsCPU stream.

PR validation:

In addition to the tests in #49237 (comment) I created a suitable set of streamer files using the following script to test the Heavy Ion menu:

#!/bin/bash -ex                                                                                       
RUNNUMBER=362321
LUMISECTION=231
NEVENTS=500

# cmsrel CMSSW_15_1_0_patch2
# cd CMSSW_15_1_0_patch2/src/
# cmsenv                                                                                                                                                                                                   
# scram b                                                                                                                                   

INPUTFILE=root://eoscms.cern.ch//eos/cms//store/user/cmsbuild/store/hidata/HIRun2022A/HITestRaw0/RAW/v1/000/362/321/00000/f467ee64-fc64-47a6-9d8a-7ca73ebca2bd.root 
rm -rf run${RUNNUMBER}*

# run on 100 events of LS 131, with 100 events per input file                                                                                             
convertToRaw -f ${NEVENTS} -l ${NEVENTS} -r ${RUNNUMBER}:${LUMISECTION} -s rawDataRepacker -o . -- "${INPUTFILE}"

tmpfile=$(mktemp)
#hltConfigFromDB --configName /users/musich/tests/dev/CMSSW_15_1_0/CMSHLT-3147/HIon > "${tmpfile}"
hltConfigFromDB --configName /dev/CMSSW_15_1_0/HIon/V14 > "${tmpfile}"
sed -i 's|process = cms.Process( "HLT" )|from Configuration.Eras.Era_Run3_cff import Run3\nprocess = cms.Process( "HLT", Run3 )|g' "${tmpfile}"
cat <<@EOF >> "${tmpfile}"
process.load("run${RUNNUMBER}_cff")

# to run without any HLT prescales                                                  
del process.PrescaleService
del process.MessageLogger
process.load('FWCore.MessageLogger.MessageLogger_cfi')

# override the GlobalTag, connection string and pfnPrefix
from Configuration.AlCa.GlobalTag import GlobalTag as customiseGlobalTag
process.GlobalTag = customiseGlobalTag(
    process.GlobalTag,
    globaltag = "150X_dataRun3_HLT_v1",
    conditions = "L1Menu_CollisionsHeavyIons2024_v1_0_6_xml,L1TUtmTriggerMenuRcd,frontier://FrontierProd/CMS_CONDITIONS,,9999-12-31 23:59:59.000"
)

# run the Full L1T emulator, then repack the data into a new RAW collection, to be used by the HLT
from HLTrigger.Configuration.CustomConfigs import L1REPACK
process = L1REPACK(process, "uGT")

process.options.numberOfThreads = 32
process.options.numberOfStreams = 32

process.options.wantSummary = True
# process.PrescaleService.forceDefault = True
@EOF
edmConfigDump "${tmpfile}" > hlt.py

cmsRun hlt.py &> hlt.log

bash -c 'echo $$ > cmsrun.pid; exec cmsRun hlt.py &> hlt.log'
job_pid=$(cat cmsrun.pid)
echo "cmsRun is running with PID: $job_pid"

# remove input files to save space
rm -f run362321/run362321_ls0*_index*.*

# prepare the files by concatenating the .ini and .dat files
mkdir -p preparedHIon

cat run362321/run362321_ls0000_streamHIDQMGPUvsCPU_pid${job_pid}.ini run362321/run362321_ls0231_streamHIDQMGPUvsCPU_pid${job_pid}.dat > preparedHIon/run362321_ls0231_streamHIDQMGPUvsCPU_pid${job_pid}.dat
cp run362321/run362321_ls0231_streamHIDQMGPUvsCPU_pid${job_pid}.jsn preparedHIon/run362321_ls0231_streamHIDQMGPUvsCPU_pid${job_pid}_prep.jsn

# now remove the extra 0

input="preparedHIon/run362321_ls0231_streamHIDQMGPUvsCPU_pid${job_pid}_prep.jsn"
output="preparedHIon/run362321_ls0231_streamHIDQMGPUvsCPU_pid${job_pid}.jsn"

jq '
  .data as $d |
  .data = (
    reduce range(0; $d|length) as $i ([]; 
      if ($i > 0 and .[-1] == "0" and $d[$i] == "0") 
      then . 
      else . + [$d[$i]] 
      end
    )
  )
' "$input" > "$output"

rm -fr preparedHIon/run362321_ls0231_streamHIDQMGPUvsCPU_pid${job_pid}_prep.jsn
rm -fr preparedHIon/run362321_ls0231_streamHIDQMGPUvsCPU_pid${job_pid}.ini

and then placed the resulting files under the path $CMSSW_BASE/src/DQM/Integration/data/run362321.
After applying the following extra changes to the package:

diff --git a/DQM/Integration/test/BuildFile.xml b/DQM/Integration/test/BuildFile.xml
index c5f5763d8d2..3e80cc4769f 100644
--- a/DQM/Integration/test/BuildFile.xml
+++ b/DQM/Integration/test/BuildFile.xml
@@ -29,7 +29,7 @@
 <test name="TestDQMOnlineClient-onlinebeammonitor_dqm_sourceclient" command="runtest.sh onlinebeammonitor_dqm_sourceclient-live_cfg.py 381594 pp_run"/>
 <test name="TestDQMOnlineClient-ecalgpu_dqm_sourceclient" command="runtest.sh ecalgpu_dqm_sourceclient-live_cfg.py 381594"/>
 <test name="TestDQMOnlineClient-hcalgpu_dqm_sourceclient" command="runtest.sh hcalgpu_dqm_sourceclient-live_cfg.py 381594"/>
-<test name="TestDQMOnlineClient-pixelgpu_dqm_sourceclient" command="runtest.sh pixelgpu_dqm_sourceclient-live_cfg.py 381594"/>
+<test name="TestDQMOnlineClient-pixelgpu_dqm_sourceclient" command="runtest.sh pixelgpu_dqm_sourceclient-live_cfg.py 362321 hi_run"/>
 <test name="TestDQMOnlineClient-pfgpu_dqm_sourceclient" command="runtest.sh pfgpu_dqm_sourceclient-live_cfg.py 381594"/>
 <!-- streamDQMCalibration is required -->
 <!-- <test name="TestDQMOnlineClient-ecalcalib_dqm_sourceclient" command="runtest.sh ecalcalib_dqm_sourceclient-live_cfg.py" /> -->

I then run the unit test of the package via:

 scram b runtests_TestDQMOnlineClient-pixelgpu_dqm_sourceclient

and I did not observe any issues.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Verbatim backport of #49237 for 2025 PbPb operations.

[mmusich]

test parameters:

• pull_request = Reorganization due to CMSHLT-3147 cms-data/DQM-Integration#14

[cmsbuild]

A new Pull Request was created by @mmusich for CMSSW_15_1_X.

It involves the following packages:

• DQM/Integration (dqm)

@cmsbuild, @ctarricone, @gabrielmscampos, @nothingface0, @rseidita can you please review it and eventually sign? Thanks.
@batinkov, @francescobrivio, @threus this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

• Backported from Pixel GPU Online DQM Client changes for CMSHLT-3147 #49237

[cmsbuild]

cms-bot internal usage

[mmusich]

@cmsbuild, please test

[cmsbuild]

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3f1128/49207/summary.html
COMMIT: 3bfcdf3
CMSSW: CMSSW_15_1_X_2025-11-03-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49296/49207/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 14 lines to the logs
• Reco comparison results: 6 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4124047
• DQMHistoTests: Total failures: 23
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4124004
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 215 log files, 184 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[smuzaffar]

test parameters:

• pull_request = [15.1.X] Update DQM-Integration version to V00-12-00 cmsdist#10169

[smuzaffar]

please test

[cmsbuild]

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3f1128/49217/summary.html
COMMIT: 3bfcdf3
CMSSW: CMSSW_15_1_X_2025-11-03-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49296/49217/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @alaperto Cosmics Strip clusters DQM [15_1_X] #49246

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3f1128/49217/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3f1128/49217/git-merge-result

Comparison Summary

Summary:

• You potentially added 13 lines to the logs
• Reco comparison results: 10 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4124047
• DQMHistoTests: Total failures: 22
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4124005
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 215 log files, 184 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[mandrenguyen]

urgent
Assuming we want this in the next release for heavy-ion data taking to be build ~ today.

[mtosi]

thanks for the prompt action, can you confirm this will be integrated in CMSSW_15_1_0_patch3 ?

[mmusich]

thanks for the prompt action, can you confirm this will be integrated in CMSSW_15_1_0_patch3 ?

while it is highly desireable to have a release including it, DQM online can still pick it up on the fly (@gabrielmscampos @nothingface0 kind ping)

[mandrenguyen]

@mtosi @mmusich This would require a full release. So if we get confirmation from @cms-sw/dqm-l2 that this can be added directly online, we could just build a patch without this PR, and then build a full release in the not too distant future.

[mmusich]

and then build a full release in the not too distant future.

to be clear, with the menu we are going to put online, it's absolutely mandatory to have this PR included in the online DQM.

[nothingface0]

if we get confirmation from @cms-sw/dqm-l2 that this can be added directly online

Of course, no problem

[nothingface0]

+dqm

• P5 tests OK

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_15_1_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_16_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-sw/cmsdist#10169

[cmsbuild]

REMINDER @sextonkennedy, @ftenchini, @mandrenguyen: This PR was tested with cms-sw/cmsdist#10169, please check if they should be merged together

[ftenchini]

+1

[ftenchini]

Merging post-patch3 targeting next full build.