Update CUDA to 12.6.3 and cuDNN to 9.6.0

[fwyzard]

Drop support for Power, which is no longer supported in CUDA, and package nvidia-smi along with the compatibility drivers.

[cmsbuild]

A new Pull Request was created by @fwyzard for branch IB/CMSSW_15_0_X/master.

@cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[fwyzard]

test parameters:

• full_cmssw = true
• enable_tests = gpu

[fwyzard]

please test

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43803/summary.html
COMMIT: ff7d71c
CMSSW: CMSSW_15_0_X_2025-01-16-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9620/43803/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

[fwyzard]

please test

[smuzaffar]

onnxruntime failed [a]. looks like we need to update onnxruntime microsoft/onnxruntime#20953

[a] https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43803/externals/onnxruntime/1.17.1-39c9283bf1157284e74f206fa39f9945/log

onnxruntime-1.17.1/onnxruntime/contrib_ops/cuda/moe/ft_moe/moe_kernel.cu(70): error: identifier "FLT_MAX" is undefined

[fwyzard]

Ah, thanks.
I saw many 404 errors in the logs, I so thought it was a (transient) network problem.

[smuzaffar]

I am updating https://github.com/cms-externals/onnxruntime with latest onnxruntime tag v1.20.1.

[smuzaffar]

please test with #9625

[fwyzard]

We need to update also cuDNN, I'll push it here and restart the tests.

[cmsbuild]

Pull request #9620 was updated.

[fwyzard]

please test with #9625

[cmsbuild]

-1

Failed Tests: GpuUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43824/summary.html
COMMIT: 880e2dd
CMSSW: CMSSW_15_0_X_2025-01-16-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9620/43824/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @CMS-HGCAL HGCal raw data handling: Unpacker, Geometry, Local Reco cmssw#45665
• @bsunanda Phase2-hgx361A Update some of the test scripts in Validation/HGCalValidation/test/python using a new tool that finds the right ERA and Global Tag cmssw#47097
• @fwyzard Remove deprecated cudaLimitDevRuntimeSyncDepth functionality cmssw#47121
• @fabiocos MTD geometry: temporarily revert the BTL numbering scheme update cmssw#47115
• @mmusich add fillDescriptions to several plugins used at HLT (4/N) cmssw#47107
• @bsunanda Phase2-hgx361B Update some of the test scripts in Validation/HGCalValidation/test/python using a new tool which finds the right ERA and Global Tag cmssw#47098
• @bsunanda Phase2-hgx361D Simplify the scripts in SimG4Core/PrintGeomInfo/test/python by using a new tool that finds th right ERA and Global Tag cmssw#47112
• @bsunanda Phase2-hgx361C Update the remaining test scripts i Validation/HGCalValidation.test/python with the new tool that finds the right ERA and Global tag for a Phase2 scenario cmssw#47103
• @trackreco Speedup of mkFit hit converters, and introduction of mkFit HLT customization for 2025 cmssw#47106
• @Pruthvi-ch Update to Cell areas and valid cells for v18/v19 version of HGCAL cmssw#47077
• @cms-sw Update tag for Geometry-TestReference to V00-18-00 #9623
• @cms-sw Buildrule: fixes when extra micro-arch is subset of default microarch #9617
• @cms-sw Revert "Update tag for Geometry-TestReference to V00-17-00" #9621
• @mseidel42 Rivet 4.0.2 and YODA 2.0.2 #9615
• @cms-sw Update tag for RecoTracker-MkFit to V00-15-00 #9622

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43824/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43824/git-merge-result

GPU Unit Tests

I found 3 errors in the following unit tests:

---> test testCudaDeviceAdditionKernel had ERRORS
---> test testCudaDeviceAdditionWrapper had ERRORS
---> test testTorchSimpleDnnCUDA had ERRORS

Comparison Summary

Summary:

• You potentially removed 120 lines from the logs
• Reco comparison results: 38319 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3819085
• DQMHistoTests: Total failures: 95801
• DQMHistoTests: Total nulls: 35
• DQMHistoTests: Total successes: 3723229
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -0.10200000000000001 KiB( 48 files compared)
• DQMHistoSizes: changed ( 140.045,... ): -0.008 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 141.042 ): 0.023 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 145.301 ): 0.004 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 145.408 ): 0.008 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 145.5 ): -0.023 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 145.604 ): -0.098 KiB JetMET/SUSYDQM
• Checked 214 log files, 184 edm output root files, 49 DQM output files
• TriggerResults: found differences in 4 / 47 workflows

GPU Comparison Summary

Summary:

• You potentially removed 24 lines from the logs
• Reco comparison results: 24 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 53071
• DQMHistoTests: Total failures: 904
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 52167
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 24 log files, 30 edm output root files, 7 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test for el9_amd64_gcc13

[cmsbuild]

-1

Failed Tests: GpuUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43851/summary.html
COMMIT: 880e2dd
CMSSW: CMSSW_15_0_X_2025-01-19-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9620/43851/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @mmusich fix miscellaneous python configuration files cmssw#47135

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43851/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43851/git-merge-result

GPU Unit Tests

I found 2 errors in the following unit tests:

---> test testCudaDeviceAdditionKernel had ERRORS
---> test testCudaDeviceAdditionWrapper had ERRORS

Comparison Summary

Summary:

• You potentially removed 172 lines from the logs
• Reco comparison results: 12 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3819085
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3819059
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 214 log files, 184 edm output root files, 49 DQM output files
• TriggerResults: no differences found

GPU Comparison Summary

Summary:

• You potentially removed 24 lines from the logs
• Reco comparison results: 24 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 53071
• DQMHistoTests: Total failures: 868
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 52203
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 24 log files, 30 edm output root files, 7 DQM output files
• TriggerResults: no differences found

[cmsbuild]

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43852/summary.html
COMMIT: 880e2dd
CMSSW: CMSSW_15_0_X_2025-01-17-2300/el9_amd64_gcc13
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9620/43852/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @mmusich fix miscellaneous python configuration files cmssw#47135

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43852/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de0284/43852/git-merge-result

Build

I found compilation error when building:

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/gcc/13.2.0-b4f157aad5ba3fefd6a4401833585549/bin/c++ -c -DGNU_GCC -D_GNU_SOURCE -DTBB_USE_GLIBCXX_VERSION=130200 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DTBB_PREVIEW_TASK_GROUP_EXTENSIONS=1 -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -DBOOST_MATH_DISABLE_STD_FPCLASSIFY -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -DDD4HEP_USE_GEANT4_UNITS=1 -DCMSSW_GIT_HASH='CMSSW_15_0_X_2025-01-17-2300' -DPROJECT_NAME='CMSSW' -DPROJECT_VERSION='CMSSW_15_0_X_2025-01-17-2300' -Isrc -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/dd4hep/v01-29-00-cd7bdac48c4a3f308cf8a3422ae17389/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/pcre/8.43-322dba8751ceabf66cf89a6077221d46/include -isystem/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/boost/1.80.0-ae354c86c9828efe7c6e37f80473137a/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/bz2lib/1.0.6-452db86d4d93dcf1690037af99319ec9/include -isystem/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/clhep/2.4.7.1-dd3c7a054bd6da4e71a71d531b36e267/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/gsl/2.6-e1d9acaffa4d45817db870679c9cda91/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/libuuid/2.34-543db95ed1650444408e40e641f34da9/include -isystem/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/lcg/root/6.32.09-ebbf3cd6d7ee0939a46ec35a9d287e9e/include -isystem/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/tbb/v2021.9.0-c1066da3248e24c5fe7a0100f9ac160a/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/xerces-c/3.1.3-e24b915ca5dee2ac502754af591a659e/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/xz/5.2.5-b6caa493ffc2cdd51db397ec2ac3b210/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/zlib/1.2.13-33497fa85fdb17c9984295219fa199e1/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-777da7040cb89c68e8201affdcbf2aca/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/eigen/3bb6a48d8c171cf20b5f8e48bfb4e424fbd4f79e-777da7040cb89c68e8201affdcbf2aca/include/eigen3 -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/fmt/10.2.1-d2066ca2935f75b53bf88bf3b87de839/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/md5/1.0.0-2f660b2cdbedb9b324f19c722f640619/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/OpenBLAS/0.3.27-066b1de6ce4eab16e12dc0e36f9bef59/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/tinyxml2/6.2.0-a65222deac81e9aef99925452e933be2/include -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -fPIC -MMD -MF tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalWaferType.cc.d src/Geometry/HGCalCommonData/src/HGCalWaferType.cc -o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalWaferType.cc.o
>> Building shared library tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/libGeometryHGCalCommonData.so
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/gcc/13.2.0-b4f157aad5ba3fefd6a4401833585549/bin/c++ -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -shared -Wl,-E    -Wl,-z,defs     tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/AHCalParameters.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalCalibrationCell.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalCassette.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalCell.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalCellOffset.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalCellUV.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalDDDConstants.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalGeomParameters.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalGeomRotation.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalGeomTools.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalGeomUtils.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalGeometryMode.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalParameters.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalParametersFromDD.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalProperty.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalTileIndex.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalTypes.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalWaferIndex.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalWaferMask.cc.o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/HGCalWaferType.cc.o -o tmp/el9_amd64_gcc13/src/Geometry/HGCalCommonData/src/GeometryHGCalCommonData/libGeometryHGCalCommonData.so -Wl,-E -Wl,--hash-style=gnu -Wl,--as-needed -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2025-01-17-2300/biglib/el9_amd64_gcc13 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2025-01-17-2300/lib/el9_amd64_gcc13 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2025-01-17-2300/external/el9_amd64_gcc13/lib -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_0_X_2025-01-17-2300/static/el9_amd64_gcc13 -lDetectorDescriptionCore -lDetectorDescriptionDDCMS -lDataFormatsForwardDetId -lDataFormatsGeometryVector -lFWCoreFramework -lDataFormatsDetId -lDataFormatsMath -lFWCoreCommon -lFWCoreServiceRegistry -lDataFormatsCommon -lFWCoreParameterSet -lFWCoreMessageLogger -lDataFormatsProvenance -lFWCoreConcurrency -lFWCorePluginManager -lFWCoreReflection -lFWCoreUtilities -lFWCoreVersion -lDDAlign -lDDCond -lDDCore -lDDParsers -lPhysics -lHist -lMatrix -lGenVector -lMathMore -lTree -lNet -lGeom -lThread -lMathCore -lRIO -lboost_program_options -lCore -lboost_thread -lboost_date_time -lCLHEP -lpcre -lbz2 -lgsl -luuid -ltbb -lxerces-c -llzma -lz -lfmt -lcms-md5 -lopenblas -lcrypt -ldl -lrt -lstdc++fs -ltinyxml2
In member function 'operator[]',
    inlined from '__ct_base ' at src/Geometry/HGCalCommonData/src/HGCalCellOffset.cc:601:23:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/gcc/13.2.0-b4f157aad5ba3fefd6a4401833585549/include/c++/13.2.0/array:203:24: error: array subscript 20 is above array bounds of 'struct array[6]' [-Werror=array-bounds=]
  203 |         return _M_elems[__n];
      |                        ^
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/gcc/13.2.0-b4f157aad5ba3fefd6a4401833585549/include/c++/13.2.0/array: In member function '__ct_base ':
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/gcc/13.2.0-b4f157aad5ba3fefd6a4401833585549/include/c++/13.2.0/array:109:55: note: while referencing '_M_elems'
  109 |       typename __array_traits<_Tp, _Nm>::_Type        _M_elems;


[fwyzard]

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/gcc/13.2.0-b4f157aad5ba3fefd6a4401833585549/include/c++/13.2.0/array:203:24: error: array subscript 20 is above array bounds of 'struct array[6]' [-Werror=array-bounds=]

Is this a pre-existing error ?

[smuzaffar]

Is this a pre-existing error ?

yes @fwyzard ( https://cmssdt.cern.ch/SDT/cgi-bin/showBuildLogs.py/el9_amd64_gcc13/www/fri/15.0-fri-23/CMSSW_15_0_X_2025-01-17-2300 ).

[smuzaffar]

+externals

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_15_0_X/master IBs (test failures were overridden). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @rappoccio, @mandrenguyen, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1

[smuzaffar]

@fwyzard , @iarspider noticed that this broke slc7 as new CUDNN requires newer glibc

error: Failed dependencies:
    libm.so.6(GLIBC_2.27)(64bit) is needed by external+cudnn+9.6.0.74-467ff84bcca63100ddb7d8055fd61b0c-1-1.x86_64

So looks like more and more projects have stop supporting CentOS7. I think it is time for us to drop slc7 support for 15.0.X and above. What do you think @makortel , @antoniovilela, @mandrenguyen, @sextonkennedy ?

[makortel]

So looks like more and more projects have stop supporting CentOS7. I think it is time for us to drop slc7 support for 15.0.X and above. What do you think?

I agree it is time to drop slc7. It just means the work to improve memory monitoring/profiling tooling becomes even more important (since we'll loose the remaining capability to run IgProf memory profiling)

[smuzaffar]

thanks @makortel , I will open a cmssw github issue so that every one knows about slc7 support

[smuzaffar]

@fwyzard , do we need to update https://github.com/cms-patatrack/cuda-compatible-runtime/blob/master/Makefile#L15 to have 12.6.3 and deploy it 12.6.3 under /cvmfs/patatrack.cern.ch/external ?

[fwyzard]

Mhm, not sure, but I'll check tomorrow.