Inaccessiable EventSetup in ExternalGeneratorFilter

[colizz]

Hi experts,

An error occurs running the GEN step with ExternalGeneratorFilter, which acts on Pythia8GeneratorFilter with Tauola set as the external decayer:

----- Begin Fatal Exception 14-Jun-2021 18:15:07 CEST-----------------------
An exception of category 'NoRecord' occurred while
   [0] Processing global begin LuminosityBlock run: 1 luminosityBlock: 1
   [1] Calling method for module Pythia8GeneratorFilter/'generator'
Exception Message:
No "PDTRecord" record found in the EventSetup.n
 Please add an ESSource or ESProducer that delivers such a record.
----- End Fatal Exception -------------------------------------------------
caught exception
An exception of category 'NoRecord' occurred while
   [0] Processing global begin LuminosityBlock run: 1 luminosityBlock: 1
   [1] Calling method for module Pythia8GeneratorFilter/'generator'
Exception Message:
No "PDTRecord" record found in the EventSetup.n
 Please add an ESSource or ESProducer that delivers such a record.

----- Begin Fatal Exception 14-Jun-2021 18:16:07 CEST-----------------------
An exception of category 'ExternalFailed' occurred while
   [0] Processing global begin LuminosityBlock run: 1 luminosityBlock: 1
   [1] Calling method for module ExternalGeneratorFilter/'generator'
Exception Message:
failed waiting for external process. Timed out after 60 seconds.
----- End Fatal Exception -------------------------------------------------

By a rough check, the reason may be related to the transfer of EventSetup between the worker and controller instance. A quick example is to run on the process TAU-RunIISummer19UL16GEN-00003 with ExternalGeneratorFilter, shown as follows:

export SCRAM_ARCH=slc7_amd64_gcc900
cmsrel CMSSW_12_0_X_2021-06-14-1100
cd CMSSW_12_0_X_2021-06-14-1100/src
cmsenv

curl -s -k https://cms-pdmv.cern.ch/mcm/public/restapi/requests/get_fragment/TAU-RunIISummer19UL16GEN-00003 --retry 3 --create-dirs -o Configuration/GenProduction/python/TAU-RunIISummer19UL16GEN-00003-fragment.py
scram b -j4
cd ../..

cmsDriver.py Configuration/GenProduction/python/TAU-RunIISummer19UL16GEN-00003-fragment.py --python_filename TAU-RunIISummer19UL16GEN-00003_1_cfg.py --eventcontent RAWSIM --customise Configuration/DataProcessing/Utils.addMonitoring --datatier GEN --fileout file:TAU-RunIISummer19UL16GEN-00003.root --conditions 112X_mcRun2_asymptotic_v2 --beamspot Realistic25ns13TeV2016Collision \
 --customise_commands='from GeneratorInterface.Core.ExternalGeneratorFilter import ExternalGeneratorFilter; process.generator = ExternalGeneratorFilter(process.generator)' \
 --step GEN --geometry DB:Extended --era Run2_2016 --no_exec --mc -n 1000

cmsRun TAU-RunIISummer19UL16GEN-00003_1_cfg.py

Many thanks for your investigation into this.

[cmsbuild]

A new Issue was created by @colizz Congqiao Li.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign core, generators

[cmsbuild]

New categories assigned: core,generators

@Dr15Jones,@smuzaffar,@mkirsano,@SiewYan,@alberto-sanchez,@makortel,@agrohsje,@GurpreetSinghChahal you have been requested to review this Pull request/Issue and eventually sign? Thanks

[agrohsje]

Hi @mkirsano @SiewYan , are you available to follow-up? The problem appears when running tauola as external decayer, there is no problem with evtgen.

[Dr15Jones]

The ExternalGeneratorFilter would need a way to pass the configuration of the needed ES modules to the external processes. That is, ES data is not passed from the main process to the worker processes, each worker process would have their own copies of the ES modules.

The main idea would be to extend the following string to contain the configuration info for the needed ES modules

cmssw/GeneratorInterface/Core/plugins/ExternalGeneratorFilter.cc

Lines 239 to 243 in 7b66901

	config += "process."s + label + "=" + config_ + "\n";
	config += "process.moduleToTest(process."s + label + ")\n";
	config += R"_(
	process.add_(cms.Service("InitRootHandlers", UnloadRootSigHandler=cms.untracked.bool(True)))
	)_";

That would probably be done by adding additional special parameters to

https://github.com/cms-sw/cmssw/blob/master/GeneratorInterface/Core/python/ExternalGeneratorFilter.py

[colizz]

Thanks Chris! While studying the implementation of the external cmsRun instance, our concern is that whether the TestProcessor that the "external cmsRun" is based on is different from the default cmsRun. As we are not familiar with the TestProcessor, not sure if EventSetup was loaded smoothly in this event routine (the EventSetup part looks pretty at the core and we haven't figured out how it interplays with the event processing). Would you mind helping with the fix if available?

Many thanks;-) Also adding @PhyM73 who was studying the code.

The ExternalGeneratorFilter would need a way to pass the configuration of the needed ES modules to the external processes. That is, ES data is not passed from the main process to the worker processes, each worker process would have their own copies of the ES modules.

The main idea would be to extend the following string to contain the configuration info for the needed ES modules

cmssw/GeneratorInterface/Core/plugins/ExternalGeneratorFilter.cc

Lines 239 to 243 in 7b66901

	config += "process."s + label + "=" + config_ + "\n";
	config += "process.moduleToTest(process."s + label + ")\n";
	config += R"_(
	process.add_(cms.Service("InitRootHandlers", UnloadRootSigHandler=cms.untracked.bool(True)))
	)_";

That would probably be done by adding additional special parameters to

https://github.com/cms-sw/cmssw/blob/master/GeneratorInterface/Core/python/ExternalGeneratorFilter.py

[Dr15Jones]

The documentation on the TestProcessor can be found here
https://github.com/cms-sw/cmssw/tree/master/FWCore/TestProcessor

[colizz]

Ok thanks @Dr15Jones ! With your previous comments, do you mean we can send in the ES module name manually (i.e. for this above case we specify 'PDTRecord' manually and somehow pass it to the code block you just show)? If I understand your points correctly then it can be a nice fix.

[PhyM73]

The ExternalGeneratorFilter would need a way to pass the configuration of the needed ES modules to the external processes. That is, ES data is not passed from the main process to the worker processes, each worker process would have their own copies of the ES modules.

The main idea would be to extend the following string to contain the configuration info for the needed ES modules

cmssw/GeneratorInterface/Core/plugins/ExternalGeneratorFilter.cc

Lines 239 to 243 in 7b66901

	config += "process."s + label + "=" + config_ + "\n";
	config += "process.moduleToTest(process."s + label + ")\n";
	config += R"_(
	process.add_(cms.Service("InitRootHandlers", UnloadRootSigHandler=cms.untracked.bool(True)))
	)_";

That would probably be done by adding additional special parameters to

https://github.com/cms-sw/cmssw/blob/master/GeneratorInterface/Core/python/ExternalGeneratorFilter.py

I have found a discussion on hypernews about adding empty ESSource. I have tried to add such a patch in ExternalGeneratorFilter.cc in a similar way as this discussion:

  std::string config = R"_(from FWCore.TestProcessor.TestProcess import *
process = TestProcess()
)_";
  config += "process."s + label + "=" + config_ + "\n";
  std::string emptypdtrecord = R"_(process.PDTRecord = cms.ESSource('EmptyESSource',
    recordName = cms.string('PDTRecord'),
    iovIsRunNotTime = cms.bool(True),
    firstValid = cms.vuint32(1)
)
)_";
  config += emptypdtrecord ;
  config += "process.moduleToTest(process."s + label + ")\n";
  config += R"_(
process.add_(cms.Service("InitRootHandlers", UnloadRootSigHandler=cms.untracked.bool(True)))
  )_";

However, this still does not work. But it does take some effects, as the error message changed to the below:

----- Begin Fatal Exception 15-Jun-2021 18:20:51 CEST-----------------------
An exception of category 'NoProxyException' occurred while
   [0] Processing global begin LuminosityBlock run: 1 luminosityBlock: 1
   [1] Calling method for module Pythia8GeneratorFilter/'generator'
Exception Message:
No data of type "HepPDT::ParticleDataTable" with label "" in record "PDTRecord"
 Please add an ESSource or ESProducer to your job which can deliver this data.
----- End Fatal Exception -------------------------------------------------
globalWaitTask waitted
exception in gloobalWaitTask
caught exception
An exception of category 'NoProxyException' occurred while
   [0] Processing global begin LuminosityBlock run: 1 luminosityBlock: 1
   [1] Calling method for module Pythia8GeneratorFilter/'generator'
Exception Message:
No data of type "HepPDT::ParticleDataTable" with label "" in record "PDTRecord"
 Please add an ESSource or ESProducer to your job which can deliver this data.

0 waiting FAILED
----- Begin Fatal Exception 15-Jun-2021 18:21:46 CEST-----------------------
An exception of category 'ExternalFailed' occurred while
   [0] Processing global begin LuminosityBlock run: 1 luminosityBlock: 1
   [1] Calling method for module ExternalGeneratorFilter/'generator'
Exception Message:
failed waiting for external process
----- End Fatal Exception -------------------------------------------------