[ROOT] Unit Tests `testDataFormatsScoutingRunX` failing in ROOT IBs

[aandvalenzuela]

Hello,

There are two Unit Tests (testDataFormatsScoutingRun2 and testDataFormatsScoutingRun3 from module DataFormats/Scouting) failing in both ROOT6 and ROOT628 IBs.

I have reproduced the failure using CMSSW_13_1_ROOT628_X_2023-03-27-2300:

28-Mar-2023 10:02:46 CEST  Closed file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/mc/RunIISummer20UL18RECO/DoubleElectron_Pt-1To300-gun/AODSIM/FlatPU0to70EdalIdealGT_EdalIdealGT_106X_upgrade2018_realistic_v11_L1v1_EcalIdealIC-v2/270000/4CDD9457-E14C-D84A-9BD4-3140CB6AEEB6.root
/tmp/avalenzu/CMSSW_13_1_ROOT628_X_2023-03-27-2300/src/DataFormats/Scouting/test/testDataFormatsScoutingRun2.sh: line 25: 19155 Segmentation fault      "${SCRAM_TEST_PATH}"/scoutingCollectionsDumper.py -v 1 -n 1 --skip 81 -i testDataFormatsScoutingRun2_step1.root -k Scouting &> testDataFormatsScoutingRun2_step2.log

malloc_consolidate(): invalid chunk size
 *** Break *** abort
malloc(): unsorted double linked list corrupted
 *** Break *** abort

They fail when running scoutingCollectionsDumper.py (https://github.com/cms-sw/cmssw/blob/master/DataFormats/Scouting/test/testDataFormatsScoutingRun2.sh#L23) and I would say after analyzing the first event (https://github.com/cms-sw/cmssw/blob/master/DataFormats/Scouting/test/scoutingCollectionsDumper.py#L174).

Find the full log here.

Let's follow-up the discussion started on #41189 (comment).

FYI, @Dr15Jones, @smuzaffar, @iarspider

[cmsbuild]

A new Issue was created by @aandvalenzuela Andrea Valenzuela.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[aandvalenzuela]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[smuzaffar]

gdb shows this

Events processed = 1
[Thread 0x7fffd26c1700 (LWP 30936) exited]
malloc_consolidate(): invalid chunk size

Thread 1 "python3" received signal SIGABRT, Aborted.
0x00007ffff6ad2aff in raise () from /lib64/libc.so.6
(gdb) where
#0  0x00007ffff6ad2aff in raise () from /lib64/libc.so.6
#1  0x00007ffff6aa5ea5 in abort () from /lib64/libc.so.6
#2  0x00007ffff6b15097 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff6b1c4ec in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffff6b1ceb8 in malloc_consolidate () from /lib64/libc.so.6
#5  0x00007ffff6b1f268 in _int_malloc () from /lib64/libc.so.6
#6  0x00007ffff6b20b72 in malloc () from /lib64/libc.so.6
#7  0x00007ffff5ec19ac in operator new (sz=5560) at ../../../../libstdc++-v3/libsupc++/new_op.cc:50
#8  0x00007fffef7b72e4 in cling::IncrementalParser::getAllTransactions() ()
   from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ROOT628_X_2023-03-28-2300/external/el8_amd64_gcc11/lib/libCling.so
#9  0x00007fffef734d82 in cling::Interpreter::runAndRemoveStaticDestructors() ()
   from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_ROOT628_X_2023-03-28-2300/external/el8_amd64_gcc11/lib/libCling.so
#10 0x00007fffef507b8c in TCling::ResetGlobals (this=0x57e350)
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.01-24fc933274de02f0e1bb17b71bafaf30/root-6.28.01/core/metacling/src/TCling.cxx:3718
#11 0x00007fffef4ff9f7 in TCling::ShutDown (this=0x57e350)
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.01-24fc933274de02f0e1bb17b71bafaf30/root-6.28.01/core/metacling/src/TCling.cxx:1692
#12 0x00007ffff62de1a2 in TROOT::EndOfProcessCleanups (this=0x7ffff6655ba0 <ROOT::Internal::GetROOT1()::alloc>)
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.01-24fc933274de02f0e1bb17b71bafaf30/root-6.28.01/core/base/src/TROOT.cxx:1236
#13 0x00007fffc6835010 in ?? ()
#14 0x00007fffffff19e0 in ?? ()
#15 0x00007ffff66bd0fd in WrapperCall (method=0, nargs=140737188853984, args_=0x7ffff7fb1040, self=0x7fffffff1880, result=0x5f4fc0)
    at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.01-24fc933274de02f0e1bb17b71bafaf30/root-6.28.01/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:774

[smuzaffar]

@vgvassilev , these tests are failing for root 6.28 and root master based builds but work for root 6.26. tests fail at the end when python (https://github.com/cms-sw/cmssw/blob/master/DataFormats/Scouting/test/scoutingCollectionsDumper.py) process is finished. Any idea why this is failing for root 628/master?

[vgvassilev]

This area has been quite problematic due to the way that ROOT is being shut down. @pcanal, do you know how we can work this issue with EndOfProcessCleanups around?

[pcanal]

The destruction order is delicate and I remember we had some issue with the tear down triggering from python. Any chance to have a standalone reproducer (so that I can start by bisecting where/when it started failing)?

[pcanal]

Also could you run the failing example with valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp ... to get more information about this (seemingly odd because it seems the allocation is failing) problem.

[smuzaffar]

@pcanal , you can find the valgrid output here

[davidlange6]

A number of reports like this seem related.. ==19874== Invalid free() / delete / delete[] / realloc() ==19874== at 0x403C4CD: operator delete(void*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/external/valgrind/3.17.0-7ca83817e7379e83453f913e11e14834/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==19874== by 0x4ECCA891: ??? ==19874== by 0x9EDBF06: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/lcg/root/6.28.01-24fc933274de02f0e1bb17b71bafaf30/lib/libCling.so) ==19874== by 0x9E65DB8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02778/el8_amd64_gcc11/lcg/root/6.28.01-24fc933274de02f0e1bb17b71bafaf30/lib/libCling.so) ==19874== by 0x9C38B8B: TCling::ResetGlobals() (TCling.cxx:3718) ==19874== by 0x8026FE9: TROOT::EndOfProcessCleanups() (TROOT.cxx:1215) ==19874== by 0x551E600F: ??? ==19874== by 0x7D400FC: WrapperCall(long, unsigned long, void*, void*, void*) (clingwrapper.cxx:774) ==19874== by 0x7D403EA: Cppyy::CallV(long, void*, unsigned long, void*) (clingwrapper.cxx:825) ==19874== by 0x7C82290: GILCallV(long, void*, CPyCppyy::CallContext*) (Executors.cxx:68) ==19874== by 0x7C846EA: CPyCppyy::(anonymous namespace)::VoidExecutor::Execute(long, void*, CPyCppyy::CallContext*) (Executors.cxx:410) ==19874== by 0x7C6730B: CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) (CPPMethod.cxx:74) ==19874== Address 0x495f7cc0 is 0 bytes inside a block of size 20 free'd

…

On Mar 31, 2023, at 5:49 AM, Malik Shahzad Muzaffar ***@***.***> wrote: @pcanal , you can find the valgrid output here — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>