CTPPS related issue in several IB workflows

[qliphy]

Although the issues mentioned in #35927 should have been fixed mostly by #35766
There appears several CTPPS related issues in IB:

https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/relVal/CMSSW_12_2/2021-10-31-2300?selectedArchs=slc7_amd64_gcc900&selectedFlavors=X&selectedStatus=failed

For example, workflow 136.8311
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_2_X_2021-10-31-2300/pyRelValMatrixLogs/run/136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017/step2_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017.log#/115-115

----- Begin Fatal Exception 01-Nov-2021 02:51:10 CET-----------------------
An exception of category 'FatalRootError' occurred while
[0] Processing Event run: 305064 lumi: 36 event: 55020723 stream: 1
[1] Running path 'MINIAODoutput_step'
[2] Prefetching for module PoolOutputModule/'MINIAODoutput'
[3] Calling method for module CTPPSProtonProducer/'ctppsProtons'
Additional Info:
[a] Fatal Root Error: @sub=TFormula::Eval
Formula is invalid and not ready to execute

and workflow 136.796

Module: CTPPSProtonProducer:ctppsProtons (crashed)
Module: StandAloneMuonProducer:displacedStandAloneMuons
Module: LXXXCorrectorProducer:ak4CaloResidualCorrector
Module: PreshowerPhiClusterProducer:multi5x5SuperClustersWithPreshower

A fatal system signal has occurred: segmentation violation

[cmsbuild]

A new Issue was created by @qliphy Qiang Li.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[qliphy]

The issues should be related to #35766 or #35914

@malbouis @CTPPS

[qliphy]

assign dqm, db, alca

[cmsbuild]

New categories assigned: dqm,db,alca

@jfernan2,@ahmad3213,@yuanchao,@rvenditti,@emanueleusai,@ggovi,@francescobrivio,@francescobrivio,@pbo0,@malbouis,@malbouis,@tvami,@tvami,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

[tvami]

@jan-kaspar can you please have a look?

[tvami]

I'm somewhat puzzled how these didnt come up in #35766 ' test,
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c5d209/20110/runTheMatrixINPUT-results/136.796_RunMET2017C+RunMET2017C+HLTDR2_2017+RECODR2_2017reHLT_skimMET_Prompt+HARVEST2017/
and
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c5d209/20110/runTheMatrixINPUT-results/136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017/
dont show this error, right?

[malbouis]

I'm somewhat puzzled how these didnt come up in #35766 ' test, https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c5d209/20110/runTheMatrixINPUT-results/136.796_RunMET2017C+RunMET2017C+HLTDR2_2017+RECODR2_2017reHLT_skimMET_Prompt+HARVEST2017/ and https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c5d209/20110/runTheMatrixINPUT-results/136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017/ dont show this error, right?

In addition to those two above, also wf 136.8642 was tested in #35914 without crashes.

[qliphy]

@tvami @malbouis Thanks. Yes, it is curious. Let's wait for next IB results. In the meantime, I will do some local tests.

[makortel]

If it problem is a threading issue (e.g. a race condition), it would not show up in PR tests (unless multithreaded tests were enabled explicitly), but would show up IB tests.