Fix era for DQM BeamSpot clients #35898

francescobrivio · 2021-10-29T12:30:51Z

PR description:

Fix the era modifier for the DQM BeamSpot clients.

PR validation:

Compiles - 12_0_X version should be tested in Playback at P5.

Backport

Will provide backport to 12_1_X and 12_0_X

FYI: @gennai @mtosi @mmusich @dzuolo @vmariani

cmsbuild · 2021-10-29T12:38:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35898/26300

This PR adds an extra 20KB to repository

cmsbuild · 2021-10-29T12:38:33Z

A new Pull Request was created by @francescobrivio for master.

It involves the following packages:

DQM/Integration (dqm)

@emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @pbo0, @rvenditti can you please review it and eventually sign? Thanks.
@threus, @batinkov, @battibass this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

francescobrivio · 2021-10-29T12:41:33Z

@cmsbuild please test

cmsbuild · 2021-10-29T16:02:06Z

-1

Failed Tests: UnitTests RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-88cfbb/20052/summary.html
COMMIT: fbeeca4
CMSSW: CMSSW_12_2_X_2021-10-28-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35898/20052/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS

RelVals-INPUT

138.3138.3_RunMinimumBias2021Splash+RunMinimumBias2021Splash+RECODR3Splash+HARVESTDR3/step2_RunMinimumBias2021Splash+RunMinimumBias2021Splash+RECODR3Splash+HARVESTDR3.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2901412
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

francescobrivio · 2021-10-30T06:06:36Z

please test with #35905

cmsbuild · 2021-10-30T11:51:23Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-88cfbb/20099/summary.html
COMMIT: fbeeca4
CMSSW: CMSSW_12_2_X_2021-10-29-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35898/20099/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2901412
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

mmusich · 2021-10-31T13:43:29Z

please test with #35921

cmsbuild · 2021-10-31T15:05:38Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-88cfbb/20119/summary.html
COMMIT: fbeeca4
CMSSW: CMSSW_12_2_X_2021-10-31-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35898/20119/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOnlineClient-ctpps_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-ecal_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-csc_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-pixel_dqm_sourceclient had ERRORS
and more ...

RelVals

135.4

----- Begin Fatal Exception 31-Oct-2021 15:16:01 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

136.8311

----- Begin Fatal Exception 31-Oct-2021 15:16:34 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

136.7611

----- Begin Fatal Exception 31-Oct-2021 15:16:34 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

4.64.6_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1/step2_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1.log
136.72411136.72411_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM.log
136.72412136.72412_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM.log

Expand to see more relval errors ...

AddOn Tests

fastsim1

----- Begin Fatal Exception 31-Oct-2021 15:11:39 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

fastsim2

----- Begin Fatal Exception 31-Oct-2021 15:11:40 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

hlt_mc_Fake1

----- Begin Fatal Exception 31-Oct-2021 15:15:58 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

francescobrivio · 2021-10-31T18:14:16Z

please test with #35921,#35766

cmsbuild · 2021-10-31T20:24:02Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-88cfbb/20123/summary.html
COMMIT: fbeeca4
CMSSW: CMSSW_12_2_X_2021-10-31-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35898/20123/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
135.4 step 1
136.8311 step 2
136.7611 step 2
136.88811 step 2
4.22 step 3
140.56 step 2
4.53 step 3
136.731 step 3
136.793 step 3
136.874 step 3
2018.1 step 1
1001.0 step 2
1000.0 step 2
1306.0 step 3
1330.0 step 3
312.0 step 3
10042.0 step 3
10024.0 step 3
10824.0 step 3
25202.0 step 3
11634.0 step 3
11634.7 step 3
11634.914 step 3
10224.0 step 3
11634.911 step 3
12434.0 step 3
23234.0 step 3
34634.0 step 3
38634.0 step 3
28234.0 step 3
250202.181 step 4
34834.999 step 4
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially added 22571 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 9
DQMHistoTests: Total histograms compared: 304467
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 304459
DQMHistoTests: Total skipped: 8
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 8 files compared)
Checked 118 log files, 16 edm output root files, 9 DQM output files
TriggerResults: no differences found

francescobrivio · 2021-11-01T09:07:04Z

@qliphy @perrotta I don't understand the errors in the baseline...
Anyway the 12_0_X PR is already merged.

qliphy · 2021-11-01T09:10:35Z

@francescobrivio
This is because IB already having errors, and has just been fixed by #35766
Let's retest with a cleaner IB: CMSSW_12_2_X_2021-10-31-2300

qliphy · 2021-11-01T09:11:01Z

please test with #35921

francescobrivio · 2021-11-01T09:11:59Z

please test with #35921

@qliphy I already tested this PR including that PR, see #35898 (comment)

qliphy · 2021-11-01T09:14:50Z

@qliphy I already tested this PR including that PR, see #35898 (comment)

Yes, but this time we will test against new IB: CMSSW_12_2_X_2021-10-31-2300
which doesn't have so many errors as last IB.

cmsbuild · 2021-11-01T12:46:40Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-88cfbb/20142/summary.html
COMMIT: fbeeca4
CMSSW: CMSSW_12_2_X_2021-10-31-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35898/20142/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901890
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2901862
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 41 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

jfernan2 · 2021-11-02T15:29:27Z

+1
Backport tested in Online DQM at P5

cmsbuild · 2021-11-02T15:29:59Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2021-11-02T15:38:43Z

+1

Move to Run3 era

Fix era for DQM BeamSpot clients

fbeeca4

cmsbuild added this to the CMSSW_12_2_X milestone Oct 29, 2021

cmsbuild added code-checks-pending dqm-pending orp-pending pending-signatures tests-pending labels Oct 29, 2021

This was referenced Oct 29, 2021

Fix era for DQM BeamSpot clients [12_0_X] #35899

Merged

Fix era for DQM BeamSpot clients [12_1_X] #35900

Merged

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 29, 2021

cmsbuild added tests-started and removed tests-pending labels Oct 29, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 29, 2021

cmsbuild added requires-external tests-started and removed tests-rejected labels Oct 30, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 30, 2021

cmsbuild added tests-started and removed tests-rejected labels Oct 31, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 31, 2021

cmsbuild added tests-started and removed tests-rejected labels Oct 31, 2021

cmsbuild added tests-approved and removed tests-started labels Oct 31, 2021

cmsbuild added tests-started and removed tests-approved labels Nov 1, 2021

cmsbuild mentioned this pull request Nov 1, 2021

Improve DQM beamspot clients #35932

Merged

cmsbuild added tests-approved and removed tests-started labels Nov 1, 2021

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Nov 2, 2021

cmsbuild added orp-approved and removed orp-pending labels Nov 2, 2021

cmsbuild merged commit b72bfe4 into cms-sw:master Nov 2, 2021

francescobrivio deleted the alca-BS_eraModifier_12_2_X branch November 3, 2021 10:16

Fix era for DQM BeamSpot clients #35898

Fix era for DQM BeamSpot clients #35898

Uh oh!

Conversation

francescobrivio commented Oct 29, 2021

PR description:

PR validation:

Backport

Uh oh!

cmsbuild commented Oct 29, 2021

Uh oh!

cmsbuild commented Oct 29, 2021

Uh oh!

francescobrivio commented Oct 29, 2021

Uh oh!

cmsbuild commented Oct 29, 2021

Unit Tests

RelVals-INPUT

Comparison Summary

Uh oh!

francescobrivio commented Oct 30, 2021

Uh oh!

cmsbuild commented Oct 30, 2021

Unit Tests

Comparison Summary

Uh oh!

mmusich commented Oct 31, 2021

Uh oh!

cmsbuild commented Oct 31, 2021

Unit Tests

RelVals

RelVals-INPUT

AddOn Tests

Uh oh!

francescobrivio commented Oct 31, 2021

Uh oh!

cmsbuild commented Oct 31, 2021

Comparison Summary

Uh oh!

francescobrivio commented Nov 1, 2021

Uh oh!

qliphy commented Nov 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qliphy commented Nov 1, 2021

Uh oh!

francescobrivio commented Nov 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qliphy commented Nov 1, 2021

Uh oh!

cmsbuild commented Nov 1, 2021

Comparison Summary

Uh oh!

jfernan2 commented Nov 2, 2021

Uh oh!

cmsbuild commented Nov 2, 2021

Uh oh!

perrotta commented Nov 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

qliphy commented Nov 1, 2021 •

edited

Loading

francescobrivio commented Nov 1, 2021 •

edited

Loading