Relvals 537.0 and 538.0 failing in GCC10 IB

[mrodozov]

Two relvals are failing in gcc10 with the same stacktrace suggesting the same issue in ThePEG:

#4  <signal handler called>
#5  0x00002b94bf6c5305 in ThePEG::CFileLineReader::operator>> (this=0x2b94c1987de0, d=<error reading variable>) at CFileLineReader.cc:115
#6  0x00002b94c42e8b45 in ThePEG::FxFxFileReader::doReadEvent() () from /cvmfs/cms-ib.cern.ch/nweek-02694/slc7_amd64_gcc10/external/herwig7/7.2.2-39af8eacc71a05fc1859229135845ef8/lib/Herwig/HwFxFx.so
#7  0x00002b94c4317a8e in ThePEG::FxFxReader::readEvent() () from /cvmfs/cms-ib.cern.ch/nweek-02694/slc7_amd64_gcc10/external/herwig7/7.2.2-39af8eacc71a05fc1859229135845ef8/lib/Herwig/HwFxFx.so
#8  0x00002b94c43108a1 in ThePEG::FxFxReader::scan() () from /cvmfs/cms-ib.cern.ch/nweek-02694/slc7_amd64_gcc10/external/herwig7/7.2.2-39af8eacc71a05fc1859229135845ef8/lib/Herwig/HwFxFx.so
#9  0x00002b94c4315610 in ThePEG::FxFxReader::initialize(ThePEG::FxFxEventHandler&) () from /cvmfs/cms-ib.cern.ch/nweek-02694/slc7_amd64_gcc10/external/herwig7/7.2.2-39af8eacc71a05fc1859229135845ef8/lib/Herwig/HwFxFx.so
#10 0x00002b94c42ed6c9 in ThePEG::FxFxFileReader::initialize(ThePEG::FxFxEventHandler&) () from /cvmfs/cms-ib.cern.ch/nweek-02694/slc7_amd64_gcc10/external/herwig7/7.2.2-39af8eacc71a05fc1859229135845ef8/lib/Herwig/HwFxFx.so
#11 0x00002b94c42fb11a in ThePEG::FxFxEventHandler::initialize() () from /cvmfs/cms-ib.cern.ch/nweek-02694/slc7_amd64_gcc10/external/herwig7/7.2.2-39af8eacc71a05fc1859229135845ef8/lib/Herwig/HwFxFx.so
#12 0x00002b94bf72f55b in ThePEG::EventGenerator::doinit (this=0x2b9490563200) at EventGenerator.cc:262
#13 0x00002b94bf732be4 in ThePEG::InterfacedBase::init (this=0x2b9490563200) at ../include/ThePEG/Interface/InterfacedBase.h:246
#14 ThePEG::EventGenerator::setup (this=this@entry=0x2b9490563200, newRunName=..., newObjects=..., newParticles=..., newMatchers=...) at EventGenerator.cc:175
#15 0x00002b94bf777991 in ThePEG::Repository::makeRun (eg=..., name=...) at Repository.cc:316
#16 0x00002b94bf77ae79 in ThePEG::Repository::exec (command=..., os=...) at Repository.cc:786
#17 0x00002b94bf77be1f in ThePEG::Repository::execAndCheckReply (line=..., os=...) at Repository.cc:510
#18 0x00002b94bf77c0cb in ThePEG::Repository::read (is=..., os=..., prompt=...) at Repository.cc:566
#19 0x00002b94bf77c575 in ThePEG::Repository::read (filename=..., os=...) at Repository.cc:452

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc10/CMSSW_12_1_UBSAN_X_2021-08-20-2300/pyRelValMatrixLogs/run/537.0_DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+HARVESTGEN/step1_DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+HARVESTGEN.log#/

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc10/CMSSW_12_1_UBSAN_X_2021-08-20-2300/pyRelValMatrixLogs/run/537.0_DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+HARVESTGEN/step1_DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+HARVESTGEN.log#/

this follows the end of the discussion thread on #34531
where the 537.0 was failing for another reason.
I'm opening to separate it from the other issue which is now resolved

[cmsbuild]

A new Issue was created by @mrodozov Mircho Rodozov.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mrodozov]

assign generators
@Dominic-Stafford lets continue here

[cmsbuild]

New categories assigned: generators

@Saptaparna,@mkirsano,@SiewYan,@alberto-sanchez,@agrohsje,@GurpreetSinghChahal you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mrodozov]

Dears,

I requested an account here:
https://phab.hepforge.org/
to contact the devs of ThePEG and/or herwig7 but while waiting for that I want to ask if you know:

1. ThePEG is compiled with -g flag and that's why we see the failing line to be 115
   #0 0x00007fffbc54d305 in ThePEG::CFileLineReader::operator>> (this=0x7fffb79b5de0, d=<error reading variable>) at CFileLineReader.cc:115

• is there a reason that the default ThePEG config is with -g flag on by default and should we keep it this way ?

2. Is there a known config way to turn the debug flag off rather than patching the makefile, if you know any ?

I'm running a gdb run over this failure and it says it can't read a double, provided as a function argument, this is the whole report from it:

Thread 1 "cmsRun" received signal SIGSEGV, Segmentation fault.
0x00007fffbc54d305 in ThePEG::CFileLineReader::operator>> (this=0x7fffb79b5de0, d=<error reading variable>) at CFileLineReader.cc:115
115	  d = std::strtod(pos, &next);
(gdb) print d
$1 = (double &) <error reading variable>
(gdb) where
#0  0x00007fffbc54d305 in ThePEG::CFileLineReader::operator>> (this=0x7fffb79b5de0, d=<error reading variable>) at CFileLineReader.cc:115
#1  0x00007fffb9719b45 in ThePEG::FxFxFileReader::doReadEvent (this=0x7fffb79b5800)
    at /build/mrodozov/thepeg/build_her_thepeg/slc7_amd64_gcc10/external/gcc/10.3.0/include/c++/10.3.0/bits/stl_vector.h:1043
#2  0x00007fffb9748a8e in ThePEG::FxFxReader::readEvent (this=0x7fffb79b5800) at FxFxReader.cc:529
#3  0x00007fffb97418a1 in ThePEG::FxFxReader::scan (this=0x7fffb79b5800) at FxFxReader.cc:306
#4  0x00007fffb9746610 in ThePEG::FxFxReader::initialize (this=<optimized out>, eh=...) at FxFxReader.cc:273
#5  0x00007fffb971e6c9 in ThePEG::FxFxFileReader::initialize (this=0x7fffb79b5800, eh=...) at FxFxFileReader.cc:449
#6  0x00007fffb972c11a in ThePEG::FxFxEventHandler::initialize (this=0x7fffb79ae000) at FxFxEventHandler.cc:98
#7  0x00007fffbc5b755b in ThePEG::EventGenerator::doinit (this=0x7fffbf22bc00) at EventGenerator.cc:262
#8  0x00007fffbc5babe4 in ThePEG::InterfacedBase::init (this=0x7fffbf22bc00) at ../include/ThePEG/Interface/InterfacedBase.h:246
#9  ThePEG::EventGenerator::setup (this=this@entry=0x7fffbf22bc00, newRunName=..., newObjects=..., newParticles=..., newMatchers=...)
    at EventGenerator.cc:175
#10 0x00007fffbc5ff991 in ThePEG::Repository::makeRun (eg=..., name=...) at Repository.cc:316
#11 0x00007fffbc602e79 in ThePEG::Repository::exec (command=..., os=...) at Repository.cc:786
#12 0x00007fffbc603e1f in ThePEG::Repository::execAndCheckReply (line=..., os=...) at Repository.cc:510
#13 0x00007fffbc6040cb in ThePEG::Repository::read (is=..., os=..., prompt=...) at Repository.cc:566
#14 0x00007fffbc604575 in ThePEG::Repository::read (filename=..., os=...) at Repository.cc:452
#15 0x00007fffbdc1fe27 in (anonymous namespace)::HerwigGenericRead (ui=...) at HerwigAPI.cc:146
#16 0x00007fffbe1d1c69 in Herwig7Interface::callHerwigGenerator (this=0x7fffc84b7878)
    at /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc10/external/gcc/10.3.0/include/c++/10.3.0/bits/shared_ptr_base.h:1324
#17 0x00007fffbe1d4b33 in Herwig7Interface::initRepository (this=this@entry=0x7fffc84b7878, pset=...)
    at /build/mrodozov/thepeg/CMSSW_12_1_X_2021-08-29-2300/src/GeneratorInterface/Herwig7Interface/src/Herwig7Interface.cc:118
#18 0x00007fffbe83c0d4 in Herwig7Hadronizer::Herwig7Hadronizer (this=0x7fffc84b7798, pset=...)
    at /build/mrodozov/thepeg/CMSSW_12_1_X_2021-08-29-2300/src/GeneratorInterface/Herwig7Interface/plugins/Herwig7Hadronizer.cc:82
#19 0x00007fffbe84328b in edm::GeneratorFilter<Herwig7Hadronizer, gen::ExternalDecayDriver>::GeneratorFilter (this=this@entry=0x7fffc84b7700, 
    ps=..., __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/src/GeneratorInterface/Core/interface/GeneratorFilter.h:80
#20 0x00007fffbe84358f in std::make_unique<edm::GeneratorFilter<Herwig7Hadronizer, gen::ExternalDecayDriver>, edm::ParameterSet const&> ()
    at /cvmfs/cms-ib.cern.ch/nweek-02696/slc7_amd64_gcc10/external/gcc/10.3.0/include/c++/10.3.0/bits/unique_ptr.h:962
#21 edm::MakeModuleHelper<edm::one::EDFilterBase>::makeModule<edm::GeneratorFilter<Herwig7Hadronizer, gen::ExternalDecayDriver> > (pset=...)
    at /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/src/FWCore/Framework/interface/maker/MakeModuleHelper.h:39
#22 edm::WorkerMaker<edm::GeneratorFilter<Herwig7Hadronizer, gen::ExternalDecayDriver> >::makeModule (this=0x7fffc86f2ae8, p=...)
    at /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/src/FWCore/Framework/interface/maker/WorkerMaker.h:82
#23 0x00007ffff7cdb3a1 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#24 0x00007ffff7c1f316 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#25 0x00007ffff7c32395 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#26 0x00007ffff7cdda8b in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#27 0x00007ffff7cdb960 in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#28 0x00007ffff7cb2416 in edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#29 0x00007ffff7cb402b in edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#30 0x00007ffff7cb6e59 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::StreamID, edm::ProcessContext const*) ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#31 0x00007ffff7c9977e in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ProcessBlockHelperBase&, edm::ThinnedAssociationsHelper&, edm::SubProcessParentageHelper const*, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#32 0x00007ffff7ca60d3 in edm::ScheduleItems::initSchedule(edm::ParameterSet&, bool, edm::PreallocationConfiguration const&, edm::ProcessContext --Type <RET> for more, q to quit, c to continue without paging--
const*, edm::ProcessBlockHelperBase&) ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#33 0x00007ffff7bb9e3f in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#34 0x00007ffff7bbbd17 in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) ()
   from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-08-29-2300/lib/slc7_amd64_gcc10/libFWCoreFramework.so
#35 0x000000000040a190 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const
    ()
#36 0x00007ffff63a5f8f in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...)
    at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_12_1_0_pre1-slc7_amd64_gcc10/build/CMSSW_12_1_0_pre1-build/BUILD/slc7_amd64_gcc10/external/tbb/v2021.3.0/tbb-v2021.3.0/src/tbb/arena.cpp:696
#37 0x000000000040b077 in main::{lambda()#1}::operator()() const ()
#38 0x000000000040971c in main ()

Any help is welcome :)

[Dominic-Stafford]

Hi @mrodozov.

I'm not sure off the top of my head why thePEG is compiled with -g in CMSSW (it's not by default if one installs standalone Herwig). However since Herwig/thePEG problems are not uncommon, I think it would be good if we left the -g flag on to help debugging these, provided the cost of doing so isn't too high.

In terms of this issue, I emailed the Herwig authors, and one of them replied last night, saying he wasn't able to reproduce the issue with a standalone install of Herwig, so the problem is related to the CMSSW Herwig7Interface compared to the standard Heriwg UI. I'm currently having a look into this, and if I can't find a solution I have a mock-up of our Herwig Interface which I can send to the authors to get them to look into it further.

[Dr15Jones]

Maybe it would be useful to run the job using valgrind?

[mrodozov]

the valgrind report before it crash:

==13729== 
31-Aug-2021 16:46:55 CEST  Initiating request to open LHE file thread1/cmsgrid_final.lhe
31-Aug-2021 16:46:55 CEST  Successfully opened LHE file thread1/cmsgrid_final.lhe
31-Aug-2021 16:46:56 CEST  Initiating request to open LHE file thread2/cmsgrid_final.lhe
31-Aug-2021 16:46:56 CEST  Successfully opened LHE file thread2/cmsgrid_final.lhe
31-Aug-2021 16:46:57 CEST  Initiating request to open LHE file thread3/cmsgrid_final.lhe
31-Aug-2021 16:46:57 CEST  Successfully opened LHE file thread3/cmsgrid_final.lhe
31-Aug-2021 16:46:58 CEST  Initiating request to open LHE file thread0/cmsgrid_final.lhe
31-Aug-2021 16:46:58 CEST  Successfully opened LHE file thread0/cmsgrid_final.lhe
==13729== Invalid write of size 8
==13729==    at 0x41D51305: ThePEG::CFileLineReader::operator>>(double&) (CFileLineReader.cc:115)
==13729==    by 0x47986B44: ThePEG::FxFxFileReader::doReadEvent() (FxFxFileReader.cc:691)
==13729==    by 0x479B5A8D: ThePEG::FxFxReader::readEvent() (FxFxReader.cc:529)
==13729==    by 0x479AE8A0: ThePEG::FxFxReader::scan() (FxFxReader.cc:306)
==13729==    by 0x479B360F: ThePEG::FxFxReader::initialize(ThePEG::FxFxEventHandler&) (FxFxReader.cc:273)
==13729==    by 0x4798B6C8: ThePEG::FxFxFileReader::initialize(ThePEG::FxFxEventHandler&) (FxFxFileReader.cc:449)
==13729==    by 0x47999119: ThePEG::FxFxEventHandler::initialize() (FxFxEventHandler.cc:98)
==13729==    by 0x41DBB55A: ThePEG::EventGenerator::doinit() (EventGenerator.cc:262)
==13729==    by 0x41DBEBE3: init (InterfacedBase.h:246)
==13729==    by 0x41DBEBE3: ThePEG::EventGenerator::setup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<ThePEG::Pointer::RCPtr<ThePEG::InterfacedBase>, std::less<ThePEG::Pointer::RCPtr<ThePEG::InterfacedBase> >, std::allocator<ThePEG::Pointer::RCPtr<ThePEG::InterfacedBase> > >&, std::map<long, ThePEG::Pointer::RCPtr<ThePEG::ParticleData>, std::less<long>, std::allocator<std::pair<long const, ThePEG::Pointer::RCPtr<ThePEG::ParticleData> > > >&, std::set<ThePEG::Pointer::RCPtr<ThePEG::MatcherBase>, std::less<ThePEG::Pointer::RCPtr<ThePEG::MatcherBase> >, std::allocator<ThePEG::Pointer::RCPtr<ThePEG::MatcherBase> > >&) (EventGenerator.cc:175)
==13729==    by 0x41E03990: ThePEG::Repository::makeRun(ThePEG::Pointer::TransientRCPtr<ThePEG::EventGenerator>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (Repository.cc:316)
==13729==    by 0x41E06E78: ThePEG::Repository::exec(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::ostream&) (Repository.cc:786)
==13729==    by 0x41E07E1E: ThePEG::Repository::execAndCheckReply(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::ostream&) (Repository.cc:510)
==13729==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==13729== 
==13729== Warning: ignored attempt to set SIGRT32 handler in sigaction();
==13729==          the SIGRT32 signal is used internally by Valgrind

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

==13729== Unsupported clone() flags: 0x311
==13729== 
==13729== The only supported clone() uses are:
==13729==  - via a threads library (LinuxThreads or NPTL)
==13729==  - via the implementation of fork or vfork
==13729== 
==13729== Valgrind detected that your program requires
==13729== the following unimplemented functionality:
==13729==    Valgrind does not support general clone().
==13729== This may be because the functionality is hard to implement,
==13729== or because no reasonable program would behave this way,
==13729== or because nobody has yet needed it.  In any case, let us know at
==13729== www.valgrind.org and/or try to work around the problem, if you can.
==13729== 
==13729== Valgrind has to exit now.  Sorry.  Bye!
==13729== 

[Dr15Jones]

So the line looks like

https://github.com/hep-mirrors/herwig/blob/024ec3121cbf42deec7a662ddcd73d4c6c4d527b/MatrixElement/FxFx/FxFxFileReader.cc#L686-L693

specifically

   >> hepeup.PUP[i][0] >> hepeup.PUP[i][1] >> hepeup.PUP[i][2]

as a guess, maybe hepup.NUP is greater than the size of hep.PUP container?

[Dr15Jones]

So hepup is defined here
https://github.com/hep-mirrors/herwig/blob/fca0efc66543e1ae38c82d349a1b6d03797a1839/MatrixElement/FxFx/FxFxReader.h#L636

with the class

https://github.com/hep-mirrors/herwig/blob/fca0efc66543e1ae38c82d349a1b6d03797a1839/MatrixElement/FxFx/FxFx.h#L128

My supposition sees unlikely as things get resized
https://github.com/hep-mirrors/herwig/blob/fca0efc66543e1ae38c82d349a1b6d03797a1839/MatrixElement/FxFx/FxFx.h#L159-L167

So it looks like what we want to know is what is hepeup.NUP and hepeup.PUP.size()

[Dominic-Stafford]

OK, from what you're saying I think the problem is that the segfault is occurring once it reads past the end of the lhe file, right? This previously just raised an error (which we logged and ignored), but it seems to be now producing the segfault. I didn't get the herwig author to test this as I told him to only check the first 7 events, but I suspect this is why he wasn't able to reproduce the segfault. I'll ask him if he can reproduce it by producing more events

[Dr15Jones]

I don't know if it is passed the end or not. I'm just pointing out where valgrind says the line is and what data structures are involved.