Adding Proxy Support to hltGetConfiguration and fixing converter/db specification bugs with hltIntegrationTests

[Sam-Harper]

PR description:

The PR updates hltGetConfiguration and friends to be able to use a socks proxy thus simplifying hlt work outside the cern network.

It also fixes a bug in hltIntegrationTests reported by Andres Leonardo Cabrera Mora which has to do with how the database/converter is selected for the "setup menu" download.

--dbproxy will now activate this and --dbproxyhost/--dbproxyport will allow hosts/ports other than localhost:8080

Note this PR is a work in progress, the main v3 converter is not updated (only v3-test) and Andrea just made a good suggestion about alternatives to sha512 hashes which I would like explore. However I thought it would be nice to have the rest of the PR available to also receive comments rather than waiting to explore that suggestion.

PR validation:

 hltIntegrationTests v3-test/run3:/dev/CMSSW_12_0_0/Fake -s /dev/CMSSW_12_0_0/GRun -i /store/data/Run2018D/EphemeralHLTPhysics1/RAW/v1/000/323/790/00000/15FD6507-9B9A-DE49-973B-B63C4DEDC226.root -x  "--globaltag auto:run3_hlt" --dbproxy 
hltGetConfiguration v3-test/run3:/dev/CMSSW_12_0_0/Fake --dbproxy
hltConfigFromDB --configName /dev/CMSSW_12_0_0/GRun/V1  --dbproxy --v3-test --run3

commands all work as expected at RAL (ie very much off the cern network)

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35987/26416

• This PR adds an extra 40KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File HLTrigger/Configuration/python/Tools/confdb.py modified in PR(s): improved element-removal in HLTProcess (hltGetConfiguration) #35917

[cmsbuild]

A new Pull Request was created by @Sam-Harper (Sam Harper) for master.

It involves the following packages:

• HLTrigger/Configuration (hlt)

@cmsbuild, @missirol, @Martin-Grunewald can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @silviodonato this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[missirol]

Related updates in ConfDB are in cms-sw/hlt-confdb#34.

In my understanding, the PR tests in CMSSW won't be sensitive to any of these updates (except possible python-syntax errors, which I checked are absent); if so, the tests can be triggered later on, once the PR has converged.

Also verified that the following now works outside the CERN network. Thanks, Sam!

ssh -f -N -D 8081 [email protected]
hltGetConfiguration v3-test/run3:/dev/CMSSW_12_0_0/Fake --dbproxy --dbproxyport 8081

[missirol]

Minor comments, mainly on output of --help.

[Sam-Harper]

thanks Marino for the review and independent test, I'll implement those fixes. I've pushed a slight rework along Andreas suggestion to cms-sw/hlt-confdb#34 so it uses the git hash so I'll update this PR to work with that as well.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35987/26447

• This PR adds an extra 40KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File HLTrigger/Configuration/python/Tools/confdb.py modified in PR(s): improved element-removal in HLTProcess (hltGetConfiguration) #35917

[cmsbuild]

Pull request #35987 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

[Sam-Harper]

okay I adjusted it to read the version and also the v3 is now at V03-00-08 so this is ready to go there

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35987/26455

• This PR adds an extra 16KB to repository

[missirol]

please test with #36021

looks like this other unrelated PR is necessary to fix some of the PR tests

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4c0be7/20324/summary.html
COMMIT: cab0c62
CMSSW: CMSSW_12_2_X_2021-11-07-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35987/20324/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
11634.911 step 1
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 41
• DQMHistoTests: Total histograms compared: 2694541
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2694513
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 40 files compared)
• Checked 173 log files, 37 edm output root files, 41 DQM output files
• TriggerResults: no differences found

[missirol]

+hlt

• related to ConfDB updates in updates for V03-00-08 hlt-confdb#34

• despite the label, this PR did not require any externals
  (another PR was needed to work around an unrelated issue in PR tests)

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1

[qliphy]

@Sam-Harper The HLT Validation shows some error in IB CMSSW_12_2_X_2021-11-09-1100
https://cmssdt.cern.ch/SDT/jenkins-artifacts/HLT-Validation/CMSSW_12_2_X_2021-11-09-1100/slc7_amd64_gcc900/HLT_Integration_GRun_DATA.log

stty: standard input: Inappropriate ioctl for device
Traceback (most recent call last):
File "/pool/condor/dir_13577/jenkins/workspace/ib-run-HLT/CMSSW_12_2_X_2021-11-09-1100/bin/slc7_amd64_gcc900/hltListPaths", line 108, in
paths = getPathList(config)
File "/pool/condor/dir_13577/jenkins/workspace/ib-run-HLT/CMSSW_12_2_X_2021-11-09-1100/bin/slc7_amd64_gcc900/hltListPaths", line 23, in getPathList
if config.proxy:
AttributeError: 'Namespace' object has no attribute 'proxy'

It seems related to this PR? Would you please have a look?

[qliphy]

Ok, I noticed there is already a fix merged: #36059
Let's see the outcome in next IB.

[missirol]

Hi @qliphy, unfortunately the fix in #36059 is unrelated to this.
I have a fix for this error, I can make a PR shortly. Also, we'll need to backport this latest fix to 12_1 and 12_0.

[qliphy]

Hi @qliphy, unfortunately the fix in #36059 is unrelated to this. I have a fix for this error, I can make a PR shortly. Also, we'll need to backport this latest fix to 12_1 and 12_0.

Thanks @missirol