Segmentation Fault While Running HYBRID_NEWKF_DISPLACED_MERGE

[A-A-Abdelhamid]

I'm encountering a seg fault from trklet::ProducerFakeDR when running the HYBRID_NEWKF_DISPLACED_MERGE configuration on a couple of SUSY PU200 sample files.
One of these files is /store/mc/Phase2Spring24DIGIRECOMiniAOD/DisplacedSUSY_stopToBottom_M-800_50mm_TuneCP5_14TeV-pythia8/GEN-SIM-DIGI-RAW-MINIAOD/PU200_AllTP_140X_mcRun4_realistic_v4-v1/2810000/fa1a17e4-27ef-4bce-a570-b1e03fc7a155.root

The job crashes at Event 6899, but only when processing the full file. If I run Event 6899 or 7000 in isolation, it processes successfully. This suggests the crash is due to a state or memory issue carrying over from a previous event.

This is the error message:

Begin processing the 901st record. Run 1, Event 6899, LumiSection 7 on stream 0 at 11-Dec-2025 21:16:38.219 CET

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.


Current Modules:

Module: trklet::ProducerFakeDR:ProducerFakeDR (crashed)

A fatal system signal has occurred: segmentation violation
Segmentation fault (core dumped)

Here is the gdb debug output:

Thread 1 "cmsRun" received signal SIGSEGV, Segmentation fault.
0x00007ffff7984fd3 in edm::RefCore::RefCore(edm::RefCore const&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libDataFormatsCommon.so
#0  0x00007ffff7984fd3 in edm::RefCore::RefCore(edm::RefCore const&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libDataFormatsCommon.so
#1  0x00007fffc7f4cdf6 in trklet::ProducerFakeDR::produce(edm::Event&, edm::EventSetup const&) () from /afs/cern.ch/user/a/alabdelh/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/pluginTrackFindingTrackletPlugins.so
#2  0x00007ffff7e5ef15 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#3  0x00007ffff7e4166c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#4  0x00007ffff7dc4f79 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#5  0x00007ffff7dd1344 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#6  0x00007ffff7f4f828 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreConcurrency.so
#7  0x00007ffff7a3c84b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=0x7ffff5bda100, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_amd64_gcc12/external/tbb/v2022.0.0-2c8b19d7f71a88d9ed1f550c4776837f/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#8  tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_amd64_gcc12/external/tbb/v2022.0.0-2c8b19d7f71a88d9ed1f550c4776837f/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#9  tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_amd64_gcc12/external/tbb/v2022.0.0-2c8b19d7f71a88d9ed1f550c4776837f/tbb-v2022.0.0/src/tbb/task_dispatcher.cpp:168
#10 0x00007ffff7d4728f in edm::FinalWaitingTask::wait() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#11 0x00007ffff7d56dde in edm::EventProcessor::processRuns() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#12 0x00007ffff7d50221 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#13 0x000000000040857b in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#14 0x00007ffff7a2af41 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_amd64_gcc12/external/tbb/v2022.0.0-2c8b19d7f71a88d9ed1f550c4776837f/tbb-v2022.0.0/src/tbb/arena.cpp:821
#15 0x000000000040a293 in main::{lambda()#1}::operator()() const ()
#16 0x00000000004051b8 in main ()

The other sample file /store/mc/Phase2Spring24DIGIRECOMiniAOD/DisplacedSUSY_stopToBottom_M-800_50mm_TuneCP5_14TeV-pythia8/GEN-SIM-DIGI-RAW-MINIAOD/PU200_AllTP_140X_mcRun4_realistic_v4-v1/2810000/a487a58e-d61e-4d1d-962c-f7229c07b32c.root crashes with error message:

Begin processing the 922nd record. Run 1, Event 5922, LumiSection 6 on stream 0 at 11-Dec-2025 21:17:04.192 CET

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Current Modules:
Module: trklet::ProducerFakeDR:ProducerFakeDR (crashed)

[A-A-Abdelhamid]

After "scram b distclean; export USER_CXXFLAGS "-g -O0", scram b -j" and then rerunning, it ran fine without errors. However, when I tried running on file /store/mc/Phase2Spring24DIGIRECOMiniAOD/DisplacedSUSY_stopToBottom_M-800_50mm_TuneCP5_14TeV-pythia8/GEN-SIM-DIGI-RAW-MINIAOD/PU200_AllTP_140X_mcRun4_realistic_v4-v1/2810000/a487a58e-d61e-4d1d-962c-f7229c07b32c.root I still get seg fault with this message:

A fatal system signal has occurred: segmentation violation                                                            
The following is the call stack containing the origin of the signal.                                                  
                                                                                                                      
Mon Dec 15 06:33:18 CET 2025                                                                                          
Thread 16 (Thread 0x7f9b84dff640 (LWP 3238403) "cmsRun"):                                                             
#0  0x00007f9babcd9fff in wait4 () from /lib64/libc.so.6                                                              
#1  0x00007f9ba7c4b217 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/C
MSSW_15_1_0_pre4/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so                                                   
#2  0x00007f9ba7c4e07a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms.cern.ch/el9_amd6
4_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so                                
#3  0x00007f9bac0d84d3 in std::execute_native_thread_routine (__p=0x7f9b879e6060) at ../../../../../libstdc++-v3/src/c
++11/thread.cc:82                                                                                                     
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6                                                       
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6                                                             
Thread 15 (Thread 0x7f9b79b20640 (LWP 3238417) "cmsRun"):                                                             
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6                                                         
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3                         
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3                                                    
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3                                                                             
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6                                                       
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6                                                             
Thread 14 (Thread 0x7f9b7931f640 (LWP 3238418) "cmsRun"):                                                             
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6                                                         
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3                         
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 13 (Thread 0x7f9b78b1e640 (LWP 3238419) "cmsRun"):
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 12 (Thread 0x7f9b7831d640 (LWP 3238420) "cmsRun"):
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 11 (Thread 0x7f9b77b1c640 (LWP 3238421) "cmsRun"):
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 10 (Thread 0x7f9b7731b640 (LWP 3238422) "cmsRun"):
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 9 (Thread 0x7f9b76b1a640 (LWP 3238423) "cmsRun"):
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 8 (Thread 0x7f9b76319640 (LWP 3238424) "cmsRun"):
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 7 (Thread 0x7f9b75b18640 (LWP 3238425) "cmsRun"):
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 6 (Thread 0x7f9b75317640 (LWP 3238426) "cmsRun"):
#0  0x00007f9babd0f9fe in epoll_wait () from /lib64/libc.so.6
#1  0x00007f9ba7aee442 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms.cern
.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x00007f9ba7aea3ad in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cms
sw/CMSSW_15_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba7af37a7 in XrdSysThread_Xeq () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exte
rnal/el9_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 5 (Thread 0x7f9b74b16640 (LWP 3238427) "cmsRun"):
#0  0x00007f9babcd5795 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1  0x00007f9babcda247 in nanosleep () from /lib64/libc.so.6
#2  0x00007f9ba7af4098 in XrdSysTimer::Wait(int) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre
4/external/el9_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x00007f9ba6b3f66c in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15
_1_0_pre4/external/el9_amd64_gcc12/lib/libXrdCl.so.3
#4  0x00007f9ba6b3f7a9 in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exter
nal/el9_amd64_gcc12/lib/libXrdCl.so.3
#5  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#6  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 4 (Thread 0x7f9b74315640 (LWP 3238428) "cmsRun"):
#0  0x00007f9babc8837a in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007f9babc93328 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x00007f9ba6bcdb16 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1
_0_pre4/external/el9_amd64_gcc12/lib/libXrdCl.so.3
#3  0x00007f9ba6bcdbc9 in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exter
nal/el9_amd64_gcc12/lib/libXrdCl.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 3 (Thread 0x7f9b73b14640 (LWP 3238429) "cmsRun"):
#0  0x00007f9babc8837a in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007f9babc93328 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x00007f9ba6bcdb16 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1
_0_pre4/external/el9_amd64_gcc12/lib/libXrdCl.so.3
#3  0x00007f9ba6bcdbc9 in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exter
nal/el9_amd64_gcc12/lib/libXrdCl.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 2 (Thread 0x7f9b73313640 (LWP 3238430) "cmsRun"):                                                     [77/1896]
#0  0x00007f9babc8837a in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007f9babc93328 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x00007f9ba6bcdb16 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1
_0_pre4/external/el9_amd64_gcc12/lib/libXrdCl.so.3
#3  0x00007f9ba6bcdbc9 in RunRunnerThread () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/exter
nal/el9_amd64_gcc12/lib/libXrdCl.so.3
#4  0x00007f9babc8b2ea in start_thread () from /lib64/libc.so.6
#5  0x00007f9babd103c0 in clone3 () from /lib64/libc.so.6
Thread 1 (Thread 0x7f9bac6cc540 (LWP 3238349) "cmsRun"):
#0  0x00007f9babd030bf in poll () from /lib64/libc.so.6
#1  0x00007f9ba7c4e297 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/el9_amd64_
gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00007f9ba7c4e494 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre
4/lib/el9_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f9bacddefd3 in edm::RefCore::RefCore(edm::RefCore const&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cm
ssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libDataFormatsCommon.so
#5  0x00007f9b7cb1d0eb in edm::Ref<edmNew::DetSetVector<TTCluster<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase
2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTCluster<edm::Ref<edm::DetSetVector<Phase
2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTC
luster<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2Tra
ckerDigi> > > >::FindForDetSetVector>::Ref (this=<optimized out>, this=<optimized out>) at /cvmfs/cms.cern.ch/el9_amd6
4_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/src/DataFormats/Common/interface/Ref.h:142
#6  0x00007f9b7cb212b9 in std::pair<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>
, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<P
hase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector
<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2Tr
ackerDigi> > > >::FindForDetSetVector>, std::bitset<64ul> >::pair (this=<optimized out>, this=<optimized out>) at /cvm
fs/cms.cern.ch/el9_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_pair.h:195
#7  0x00007f9b7cb5ab63 in std::construct_at<std::pair<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<
Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<e
dm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, ed
mNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDet
SetVector<Phase2TrackerDigi> > > >::FindForDetSetVector>, std::bitset<64ul> >, std::pair<edm::Ref<edmNew::DetSetVector
<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2Tr
ackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetS
etVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2Trac
kerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector>, std::bitset<64ul> > > (__
location=0x400764a6541efe3b) at /cvmfs/cms.cern.ch/el9_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca2
8/include/c++/12.3.1/bits/stl_construct.h:97
#8  0x00007f9b7cb4f705 in std::allocator_traits<std::allocator<std::pair<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref
<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >
, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2T
rackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::re
fhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector>, std::bitset<64ul> > > >::construct<std::p
air<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhel
per::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2Tracker
Digi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVect
or<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetV
ector>, std::bitset<64ul> >, std::pair<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDi
gi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVecto
r<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVec
tor<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase
2TrackerDigi> > > >::FindForDetSetVector>, std::bitset<64ul> > > (__a=..., __p=0x400764a6541efe3b) at /cvmfs/cms.cern.
ch/el9_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/alloc_traits.h:518
#9  0x00007f9b7cb44d80 in std::vector<std::pair<edm::Ref<edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2
TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::De
tSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > >, edmNew::
DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVec
tor<Phase2TrackerDigi> > > >::FindForDetSetVector>, std::bitset<64ul> >, std::allocator<std::pair<edm::Ref<edmNew::Det
SetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::FindForDetSetVector
<Phase2TrackerDigi> > > >, TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, Phase2TrackerDigi, edm::refhelper::Fi
ndForDetSetVector<Phase2TrackerDigi> > >, edmNew::DetSetVector<TTStub<edm::Ref<edm::DetSetVector<Phase2TrackerDigi>, P
hase2TrackerDigi, edm::refhelper::FindForDetSetVector<Phase2TrackerDigi> > > >::FindForDetSetVector>, std::bitset<64ul
> > > >::push_back (this=0x7f9b149485c0, __x=...) at /cvmfs/cms.cern.ch/el9_amd64_gcc12/external/gcc/12.3.1-40d504be63
70b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_vector.h:1281
#10 0x00007f9b7cba1790 in trklet::ProducerFakeDR::produce (this=0x7f9b29992e80, iEvent=..., iSetup=...) at src/L1Trigg
er/TrackFindingTracklet/plugins/ProducerFakeDR.cc:158
#11 0x00007f9bad25ef15 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRe
gistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/
el9_amd64_gcc12/libFWCoreFramework.so
#12 0x00007f9bad24166c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, ed
m::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_g
cc12/libFWCoreFramework.so
#13 0x00007f9bad1c4f79 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::Occurrence
Traits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm
::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm:
:OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el9_amd64
_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#14 0x00007f9bad1d1344 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType
)1> >::execute() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreF
ramework.so
#15 0x00007f9bad39e828 in tbb::detail::d2::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb:
:detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_1_0_pre4/lib/el9_amd64_gcc
12/libFWCoreConcurrency.so
#16 0x00007f9bace5d84b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter
> (waiter=..., t=0x7f9baafda800, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_amd6
4_gcc12/external/tbb/v2022.0.0-2c8b19d7f71a88d9ed1f550c4776837f/tbb-v2022.0.0/src/tbb/task_dispatcher.h:334
#17 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized o
ut>, this=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el9_amd64_gcc12/external/tbb/v2022.0
.0-2c8b19d7f71a88d9ed1f550c4776837f/tbb-v2022.0.0/src/tbb/task_dispatcher.h:470
#18 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/je
nkins/workspace/build-any-ib/w/BUILD/el9_amd64_gcc12/external/tbb/v2022.0.0-2c8b19d7f71a88d9ed1f550c4776837f/tbb-v2022
.0.0/src/tbb/task_dispatcher.cpp:168
#19 0x00007f9bad14728f in edm::FinalWaitingTask::wait() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_15_
1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#20 0x00007f9bad156dde in edm::EventProcessor::processRuns() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSS
W_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so
#21 0x00007f9bad150221 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/
CMSSW_15_1_0_pre4/lib/el9_amd64_gcc12/libFWCoreFramework.so 
#22 0x000000000040857b in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, v
oid>::operator()() const ()
#23 0x00007f9bace4bf41 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/
build-any-ib/w/BUILD/el9_amd64_gcc12/external/tbb/v2022.0.0-2c8b19d7f71a88d9ed1f550c4776837f/tbb-v2022.0.0/src/tbb/are
na.cpp:821
#24 0x000000000040a293 in main::{lambda()#1}::operator()() const ()
#25 0x00000000004051b8 in main ()

Current Modules:

Module: trklet::ProducerFakeDR:ProducerFakeDR (crashed)

A fatal system signal has occurred: segmentation violation
Segmentation fault (core dumped)