Segfault in Herwig7GeneratorFilter

[makortel]

Workflow 511.0 step 1 segfaults in CMSSW_12_0_X_2021-07-16-1100

Thread 5 (Thread 0x2b8eacc00700 (LWP 30961)):
#2  0x00002b8e72f0c590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b8e6d690d19 in syscall () from /lib64/libc.so.6
#5  0x00002b8e6c574f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b8e71b2f02c) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 4 (Thread 0x2b8eabc9e700 (LWP 30960)):
#3  0x00002b8e72f1095b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b8e9b1986d0 in (anonymous namespace)::HerwigGenericRun(Herwig::HerwigUI const&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libHerwigAPI.so.2
#6  0x00002b8e9b19a757 in Herwig::API::prepareRun(Herwig::HerwigUI const&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/external/slc7_amd64_gcc900/lib/libHerwigAPI.so.2
#7  0x00002b8e9ab7babd in Herwig7Interface::callHerwigGenerator() () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libGeneratorInterfaceHerwig7Interface.so
#8  0x00002b8e9ab7bcb1 in Herwig7Interface::initGenerator() () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libGeneratorInterfaceHerwig7Interface.so
#9  0x00002b8e9ab3c074 in Herwig7Hadronizer::initializeForInternalPartons() () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginGeneratorInterfaceHerwig7HadronizerPlugins.so
#10 0x00002b8e9ab48e84 in edm::GeneratorFilter<Herwig7Hadronizer, gen::ExternalDecayDriver>::beginLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginGeneratorInterfaceHerwig7HadronizerPlugins.so
#11 0x00002b8e6ae62ecd in edm::one::EDFilterBase::doBeginLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#12 0x00002b8e6ae40b50 in edm::WorkerT<edm::one::EDFilterBase>::implDoBegin(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#13 0x00002b8e6ad49ed6 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#14 0x00002b8e6ad4a0dd in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#15 0x00002b8e6ad4a386 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#16 0x00002b8e6ad4a8f0 in void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#17 0x00002b8e6ad4aaa1 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)0> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#18 0x00002b8e6ab192c9 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/libFWCoreConcurrency.so

Thread 3 (Thread 0x2b8eab29d700 (LWP 30959)):
#2  0x00002b8e72f0c590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b8e6d690d19 in syscall () from /lib64/libc.so.6
#5  0x00002b8e6c574f17 in tbb::detail::r1::futex_wait (comparand=2, futex=0x2b8e71b2f0ac) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Thread 1 (Thread 0x2b8e6f711a40 (LWP 15763)):
#2  0x00002b8e72f0c590 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_X_2021-07-16-1100/lib/slc7_amd64_gcc900/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b8e6d690d19 in syscall () from /lib64/libc.so.6
#5  0x00002b8e6c57b948 in tbb::detail::r1::futex_wait (comparand=2, futex=0x7ffdc525cd40) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc900/external/tbb/v2021.2.0-b31097a1f21d118698ff6fa80cccd3e7/tbb-v2021.2.0/src/tbb/semaphore.h:289

Current Modules:
Module: Herwig7GeneratorFilter:generator (crashed)
Module: none
Module: none
Module: none

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_0_X_2021-07-16-1100/pyRelValMatrixLogs/run/511.0_QCD_Pt-30_13TeV_aMCatNLO_herwig7+QCD_Pt-30_13TeV_aMCatNLO_herwig7+HARVESTGEN/step1_QCD_Pt-30_13TeV_aMCatNLO_herwig7+QCD_Pt-30_13TeV_aMCatNLO_herwig7+HARVESTGEN.log#/

The log has

>>> Compiling MadGraph amplitudes. This may take some time -- please be patient.
>>> In case of problems see ./Herwig-cache/Build/MadGraphAmplitudes/MG.log for details.

*** MadGraph build failed, check logfile for details ***Error: Failed to initialize amplitude 'MadGraph'
Error: The object '/Herwig/EventHandlers/Luminosity' was not created as another object with that name already exists.
Error: The object '/Herwig/Analysis/HepMC' was not created as another object with that name already exists.

but I don't know if that is relevant or not (sounds plausible at least)

[makortel]

assign generators

[cmsbuild]

New categories assigned: generators

@Saptaparna,@mkirsano,@SiewYan,@alberto-sanchez,@agrohsje,@GurpreetSinghChahal you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[qliphy]

adding @theofil @colizz @Dominic-Stafford

[qliphy]

Herwig-cache/Build/MadGraphAmplitudes/MG.log from local test shows:

madgraph requires the six module. The easiest way to install it is to run "pip install six --user"
in case of problem with pip, you can download the file at https://pypi.org/project/six/ . It has a single python file that you just need to put inside a directory of your $PYTHONPATH environment variable.
Traceback (most recent call last):
File "/cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/external/herwig7/7.2.2-2f073c2aa813b8c514b4ec563f97e619/bin/mg2herwig", line 377, in
"MG_tmp","ident_card.dat"))
File "/cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/external/herwig7/7.2.2-2f073c2aa813b8c514b4ec563f97e619/bin/mg2herwig", line 368, in replacetext
with open(filename, "r") as f:
IOError: [Errno 2] No such file or directory: 'MG5/Source/MODEL/lha_read.f'

Could this be related to cms-sw/cmsdist#7112 ? @davidlange6 @smuzaffar

[theofil]

Error: The object '/Herwig/EventHandlers/Luminosity' was not created as another object with that name already exists.
Error: The object '/Herwig/Analysis/HepMC' was not created as another object with that name already exists.

but I don't know if that is relevant or not (sounds plausible at least)

@makortel These error messages are known issues of the cmssw Herwig interface and appear (always) when Herwig is used as hadronizer of an external LHE. we are on the way fixing this printing issue (related to duplicate lines appended in the Herwig.in file). Bottomline, alone these Error lines in the printout don't indicate a problem and should be ignored. That is not precluding that the error is on the Herwig interface, but the specific error messages are ~normal

[Dominic-Stafford]

As @qliphy suggested, I think the problem is the script that makes the Madgraph directory used by the Herwig matchbox fails almost silently, just giving this message about six being missing. Then the actual traceback in MG.log is just from Herwig trying to patch one of the MG fortran files, and throwing the exception because it's not there. Unfortunately I'm not sure where the python script that creates the Madgraph directory is, so we can't easily change that to run with python3. Will the default python be changed from 2 to 3 fairly soon? If not I can contact the authors to ask where this troublesome script is and patch it to run with python3.

[Dominic-Stafford]

Actually, I think I've found the line- I'll patch this, check it fixes the issue then make a PR

[Dominic-Stafford]

I'm having trouble building cmsdist, as the python script that compiles madgraph fails when trying to import six, despite the fact this is running with python3, and the py3-six package successfully installed. Is there anything one needs to do to ensure this is on the PYTHONPATH during the install, or something?

[davidlange6]

Add a requires statement on py3-six in your spec file? if not, could you put your spec file someplace?

[Dominic-Stafford]

Yes, I require py3-six in the spec file. I haven't changed the madgraph spec file from the one here: https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_12_0_X/master/madgraph5amcatnlo.spec, which was why I was confused it didn't work. However I think my version of pkgtools may have been out of date, so I've updated that and started the build again

[agrohsje]

+generators