PyROOT in root 6.24 branch hangs while loading CMSSW library

[smuzaffar]

Hi,
We are trying to update root 6.24 branch (commit 7c0cfac) in CMSSW special integration builds (https://github.com/cms-sw/cmsdist/pull/6746/files ) but looks like pyROOT fails/hangs for some special dictionaries.

While building cmssw , we use https://github.com/cms-sw/cmssw/blob/master/FWCore/Utilities/scripts/edmCheckClassVersion to check for root dictionaries class versions. This works for most of our dictionaries e.g following two run fine ( https://github.com/cms-sw/cmssw/blob/master/DataFormats/TauReco/src/classes_def_hlt.xml, https://github.com/cms-sw/cmssw/blob/master/DataFormats/TauReco/src/classes_def_1.xml )

> src/FWCore/Utilities/scripts/edmCheckClassVersion -l libDataFormatsTauReco.so -x src/DataFormats/TauReco/src/classes_def_hlt.xml
> src/FWCore/Utilities/scripts/edmCheckClassVersion -l libDataFormatsTauReco.so -x src/DataFormats/TauReco/src/classes_def_1.xml

but it fails/hangs for https://github.com/cms-sw/cmssw/blob/master/DataFormats/TauReco/src/classes_def_2.xml

> src/FWCore/Utilities/scripts/edmCheckClassVersion -l libDataFormatsTauReco.so -x src/DataFormats/TauReco/src/classes_def_2.xml

Most of the times the above command just hangs with error https://muzaffar.web.cern.ch/root624/err1.log but once I was able to get this error https://muzaffar.web.cern.ch/root624/err.log . Can you please look in to it and see if this log helps?

In case you want to try it yourself then you go to cmsdev25 and do

>cd /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100
>cmsenv
>/build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/src/FWCore/Utilities/scripts/edmCheckClassVersion -l libDataFormatsTauReco.so -x /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/src/DataFormats/TauReco/src/classes_def_2.xml

FYI @mrodozov @makortel @Dr15Jones

[smuzaffar]

the root commit b802a6b was working fine. So the changes which we are testing are b802a6b...7c0cfac

[Axel-Naumann]

Would you be able to provide me with a valgrind report? --num-callers=50 --track-origins=yes --suppressions=$ROOTSYS/etc/valgrind-root.supp would help a lot.

[Axel-Naumann]

This might be a dupe of #7657 - but I'd like to make progress with both of them independently to not serialize progress towards v6.24/00! (I.e. I'd still very much appreciate the valgrind report.)

[smuzaffar]

@Axel-Naumann , valgrind also hangs without printing nay usefull information. Under gdb I see this [a]. If I build root in Debug mode then I do not get this segmentation fault.
[a]

(gdb) where
#0  0x00007ffff6f5d272 in _int_malloc () from /lib64/libc.so.6
#1  0x00007ffff6f6078c in malloc () from /lib64/libc.so.6
#2  0x00007ffff67ad7c5 in operator new (sz=127) at ../../../../libstdc++-v3/libsupc++/new_op.cc:50
#3  0x00007ffff683fc6d in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate (this=this@entry=0x7fffffff38c0, __pos=63, __len1=__len1@entry=0,
    __s=0x7ffff471c3cb "::", __len2=2)
    at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre6-slc7_amd64_gcc900/build/CMSSW_11_1_0_pre6-build/BUILD/slc7_amd64_gcc900/external/gcc/9.3.0/gcc-9.3.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:993
#4  0x00007ffff684127b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append (this=0x7fffffff38c0, __s=<optimized out>, __n=<optimized out>)
    at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_1_0_pre6-slc7_amd64_gcc900/build/CMSSW_11_1_0_pre6-build/BUILD/slc7_amd64_gcc900/external/gcc/9.3.0/gcc-9.3.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/char_traits.h:300
#5  0x00007ffff199d41f in cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCling.so
#6  0x00007ffff19237ef in TClingClassInfo::TClingClassInfo(cling::Interpreter*, char const*, bool) ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCling.so
#7  0x00007ffff189c644 in TCling::GetInterpreterTypeName(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCling.so
#8  0x00007ffff6b7745a in TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCore.so
#9  0x00007ffff6b9f27b in TProtoClass::FillTClass(TClass*) ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCore.so
#10 0x00007ffff6b80bea in TClass::Init(char const*, short, std::type_info const*, TVirtualIsAProxy*, char const*, char const*, int, int, ClassInfo_t*, bool) ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCore.so
#11 0x00007ffff6b821c2 in TClass::TClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int, bool) ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCore.so
#12 0x00007ffff6b826d9 in ROOT::CreateClass(char const*, short, std::type_info const&, TVirtualIsAProxy*, char const*, char const*, int, int) ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCore.so
#13 0x00007ffff6b918ea in ROOT::TGenericClassInfo::GetClass() ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCore.so
#14 0x00007ffff6b76fea in TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCore.so
#15 0x00007fffdedbc05b in ?? ()
#16 0x000000000000042e in ?? ()
#17 0x0000000006c6f350 in ?? ()
#18 0x0000000006d34680 in ?? ()
#19 0x0000000006c76630 in ?? ()
#20 0x0000000006c764c0 in ?? ()
#21 0x0000000006c6f350 in ?? ()
#22 0x0000000006c76630 in ?? ()
#23 0x00007fffffff42a8 in ?? ()
#24 0x00000000057251f0 in ?? ()
#25 0x00007ffff6b777d0 in ?? ()
   from /build/muz/r624/w/tmp/BUILDROOT/ea8220342d406ab7dbc2d210a1e9351b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_ROOT624_X_2021-03-25-1100/external/slc7_amd64_gcc900/lib/libCore.so
#26 0x00007fffffff42a8 in ?? ()
#27 0x00007fffffff4220 in ?? ()
#28 0x0000000100000000 in ?? ()
#29 0x0000000000000000 in ?? ()

[smuzaffar]

The call to ROOT.TClass.GetClass(name) https://github.com/cms-sw/cmssw/blob/master/FWCore/Utilities/scripts/edmCheckClassVersion#L129 failed for class reco::PFTau https://github.com/cms-sw/cmssw/blob/master/DataFormats/TauReco/interface/PFTau.h

[Axel-Naumann]

In case you want to try it yourself then you go to cmsdev25

Help! @pcanal maybe? What I'm after, given the backtrace of #7718 (comment) , is which frames cause the inf loop, i.e. which is the frame that doesn't return.

@smuzaffar what we can also try, because of the "only happens in release builds", is #7752

[smuzaffar]

@Axel-Naumann , do you have #7752 equivalent for v6.24 branch?

[Axel-Naumann]

@smuzaffar that's #7767

[smuzaffar]

thanks, I am testing these now

• v6.24 [v6.24] Cling: prevent double release of Transactions cms-sw/root#153
• master Cling: prevent double release of Transactions cms-sw/root#152

[smuzaffar]

@Axel-Naumann , looks like some latest development in v6.24 branch has fixed the hanging issue. I have tested 126c9c8 (without #7767) and this time cmssw build was successful. We get runtime errors now, see the details here cms-sw/cmsdist#6777 (comment) . You can find the crash log https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a7de73/13897/runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step3_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log.

With your changes in #7767 ( on top of 126c9c8 ) , things look in much better state. PR tests ( cms-sw#153 (comment) ) show no build or run time errors. But we do see some comparison differences for our reconstruction code.

[smuzaffar]

ROOT master based CMSSW releases also show the same resulsts:

• [ROOT master] Updated root to tip of branch master cms-sw/cmsdist#6778 (comment) without Cling: prevent double release of Transactions #7752 . There is no hanging at build time but we see runtime crashes (https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-46d8cb/13898/runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step3_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log)
• Cling: prevent double release of Transactions cms-sw/root#152 (comment) with Cling: prevent double release of Transactions #7752 . No build or runtime errors but there are comparison differences.