Crash in TMutex::UnLock when using XrootD 5.1.1

[mrodozov]

• [ checked] Checked for duplicates

Describe the bug

We are trying to update the XrootD version to 5.1.1 :
cms-sw/cmsdist#6831
but the test fails in the file open attempt
with this:

27-Apr-2021 13:36:43 CEST  Initiating request to open file root://eoscms.cern.ch//eos/cms/store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-12Nov2019_UL2018-v1/260000/0F90F012-3C97-344E-8E2F-F459A798FA74.root

Thread 1 "cmsRun" received signal SIGSEGV, Segmentation fault.
0x00007ffff74a42b1 in TMutex::UnLock (this=0x7fffb0c95750)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/lcg/root/6.22.08-1fec05f50351c1988c4bf604c7ec21c3/root-6.22.08/core/thread/src/TMutex.cxx:70

(gdb) where
#0  0x00007ffff74a42b1 in TMutex::UnLock (this=0x7fffb0c95750)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/lcg/root/6.22.08-1fec05f50351c1988c4bf604c7ec21c3/root-6.22.08/core/thread/src/TMutex.cxx:70
#1  0x00007ffff76c8b97 in TLockGuard::~TLockGuard (this=0x7fffffff0b90, __in_chrg=<optimized out>) at include/TVirtualMutex.h:86
#2  0x00007ffff67266df in TPluginHandler::CheckForExecPlugin (this=0x7fffd04c19c0, nargs=2)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/lcg/root/6.22.08-1fec05f50351c1988c4bf604c7ec21c3/root-6.22.08/core/base/src/TPluginManager.cxx:284
#3  0x00007ffff6765285 in TPluginHandler::ExecPluginImpl<char const*, bool> (this=0x7fffd04c19c0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/lcg/root/6.22.08-1fec05f50351c1988c4bf604c7ec21c3/root-6.22.08/core/base/inc/TPluginManager.h:151
#4  0x00007ffff6764dee in TPluginHandler::ExecPlugin<char const*, bool> (this=0x7fffd04c19c0, nargs=2)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/lcg/root/6.22.08-1fec05f50351c1988c4bf604c7ec21c3/root-6.22.08/core/base/inc/TPluginManager.h:173
#5  0x00007ffff6756a72 in TSystem::FindHelper (this=0x7ffff2c38000, 
    path=0x7fffb0d42060 "root://eoscms.cern.ch//eos/cms/store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-12Nov2019_UL2018-v1/260000/0F90F012-3C97-344E-8E2F-F459A798FA74.root", dirptr=0x0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/lcg/root/6.22.08-1fec05f50351c1988c4bf604c7ec21c3/root-6.22.08/core/base/src/TSystem.cxx:780
#6  0x00007ffff6864ec4 in TUnixSystem::IsPathLocal (this=0x7ffff2c38000, 
    path=0x7fffb0d42060 "root://eoscms.cern.ch//eos/cms/store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-12Nov2019_UL2018-v1/260000/0F90F012-3C97-344E-8E2F-F459A798FA74.root")
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/lcg/root/6.22.08-1fec05f50351c1988c4bf604c7ec21c3/root-6.22.08/core/unix/src/TUnixSystem.cxx:1605
#7  0x00007ffff6e1fd33 in TFile::GetType (
    name=0x7fffb0d42060 "root://eoscms.cern.ch//eos/cms/store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-12Nov2019_UL2018-v1/260000/0F90F012-3C97-344E-8E2F-F459A798FA74.root", option=0x7fffffff15b9 "", prefix=0x7fffffff1460)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/lcg/root/6.22.08-1fec05f50351c1988c4bf604c7ec21c3/root-6.22.08/io/io/src/TFile.cxx:4685
#8  0x00007ffff6e1db61 in TFile::Open (
    url=0x7fffb0d418e0 "root://eoscms.cern.ch//eos/cms/store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-12Nov2019_UL2018-v1/260000/0F90F012-3C97-344E-8E2F-F459A798FA74.root", options=0x7fffac8d86f9 "", ftitle=0x7fffac8d86f9 "", compress=101, netopt=0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/lcg/root/6.22.08-1fec05f50351c1988c4bf604c7ec21c3/root-6.22.08/io/io/src/TFile.cxx:4065
#9  0x00007fffac890f2d in edm::InputFile::InputFile(char const*, char const*, edm::InputType) ()
   from /build/mrodozov/xrootd/local_install/CMSSW_12_0_X_2021-04-26-1100/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#10 0x00007fffac8cd35d in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) ()
   from /build/mrodozov/xrootd/local_install/CMSSW_12_0_X_2021-04-26-1100/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#11 0x00007fffac8d037c in edm::RootPrimaryFileSequence::RootPrimaryFileSequence(edm::ParameterSet const&, edm::PoolSource&, edm::InputFileCatalog const&) ()
   from /build/mrodozov/xrootd/local_install/CMSSW_12_0_X_2021-04-26-1100/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
--Type <RET> for more, q to quit, c to continue without paging--
#12 0x00007fffac895320 in edm::PoolSource::PoolSource(edm::ParameterSet const&, edm::InputSourceDescription const&) ()
   from /build/mrodozov/xrootd/local_install/CMSSW_12_0_X_2021-04-26-1100/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#13 0x00007fffac892ebf in edmplugin::PluginFactory<edm::InputSource* (edm::ParameterSet const&, edm::InputSourceDescription const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&, edm::InputSourceDescription const&) const ()
   from /build/mrodozov/xrootd/local_install/CMSSW_12_0_X_2021-04-26-1100/lib/slc7_amd64_gcc900/pluginIOPoolInput.so
#14 0x00007ffff7c33fd3 in edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&, edm::InputSourceDescription const&) const ()
   from /build/mrodozov/xrootd/local_install/CMSSW_12_0_X_2021-04-26-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#15 0x00007ffff7bc36cf in edm::makeInput(edm::ParameterSet&, edm::CommonParams const&, std::shared_ptr<edm::ProductRegistry>, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&) () from /build/mrodozov/xrootd/local_install/CMSSW_12_0_X_2021-04-26-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#16 0x00007ffff7bc6f5e in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) ()
   from /build/mrodozov/xrootd/local_install/CMSSW_12_0_X_2021-04-26-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#17 0x00007ffff7bc8eda in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) ()
   from /build/mrodozov/xrootd/local_install/CMSSW_12_0_X_2021-04-26-1100/lib/slc7_amd64_gcc900/libFWCoreFramework.so
#18 0x000000000040ba11 in tbb::interface7::internal::delegated_function<main::{lambda()#1}::operator()() const::{lambda()#1} const, void>::operator()() const ()
#19 0x00007ffff5ef5552 in tbb::interface7::internal::task_arena_base::internal_execute (this=0x7fffffff3100, d=...) at ../../src/tbb/arena.cpp:1105
#20 0x000000000040ca13 in main::{lambda()#1}::operator()() const ()
#21 0x000000000040b62c in main ()

After using XrootD 5.1.1 to open a file ROOT crashes in TMutex::UnLock

Expected behavior

cmsRun src/CalibTracker/SiStripChannelGain/test/testSSTGain_PCL_FromRECO_cfg.py era=A
Using Dataset: /ZeroBias/Run2018A-SiStripCalMinBias-12Nov2019_UL2018-v1/ALCARECO
27-Apr-2021 14:00:10 CEST  Initiating request to open file root://eoscms.cern.ch//eos/cms/store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-12Nov2019_UL2018-v1/260000/0F90F012-3C97-344E-8E2F-F459A798FA74.root
27-Apr-2021 14:00:12 CEST  Successfully opened file root://eoscms.cern.ch//eos/cms/store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-12Nov2019_UL2018-v1/260000/0F90F012-3C97-344E-8E2F-F459A798FA74.root
%MSG-i SiStripGainsPCLWorker:   SiStripGainsPCLWorker:ALCARECOSiStripCalib@beginRun 27-Apr-2021 14:00:20 CEST  Run: 315420
 Resetting APV struct
%MSG

this -> Successfully opened file root://
and then working on the file

To Reproduce

Use this install script:

/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/6831/14586/install.sh
cd CMSSW_12_0_X_2021-04-26-1100
cmsenv
cmsRun src/CalibTracker/SiStripChannelGain/test/testSSTGain_PCL_FromRECO_cfg.py era=A

Setup

1. 6-22 -
   https://github.com/cms-sw/root/commits/cms/v6-22-00-patches/d1d36a5f17904f7d3661c97d28da8f3954e02340
2. SLC7
3. Installed from source

Additional context

The pull request from above
cms-sw/cmsdist#6831
is setuping cmake release type to Debug for both ROOT and XrootD
as requested in #7335

[mrodozov]

this seems to be due to something in xrootd which is now solved in cms, closing

[github-actions]

Hi @mrodozov, @pcanal, @jalopezg-r00t,

It appears this issue is closed, but wasn't yet added to a project. Please add upcoming versions that will include the fix, or 'not applicable' otherwise.

Sincerely,
🤖

[github-actions]

Hi @mrodozov, @pcanal, @jalopezg-r00t,

It appears this issue is closed, but wasn't yet added to a project. Please add upcoming versions that will include the fix, or 'not applicable' otherwise.

Sincerely,
🤖