[CUDART] Improved cuda-runtime package

[smuzaffar]

Improved cuda-runtime package. Include all public libs.headers

[smuzaffar]

test parameters:

• full_cmssw = true
• enable = gpu

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_15_1_X/cudart.

@cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

please test for CMSSW_15_1_CUDART_X

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45845/summary.html
COMMIT: f75447e
CMSSW: CMSSW_15_1_CUDART_X_2025-05-04-2300/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9828/45845/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

Trying to install the rpm package external+hwloc+2.12.0-af9daca5de88b067b0fe494e77bfe58b just built.
Checking package dependencies: external+hwloc+2.12.0-af9daca5de88b067b0fe494e77bfe58b
Done checking package dependencies: external+hwloc+2.12.0-af9daca5de88b067b0fe494e77bfe58b
Checking local path dependency for rpm package external+hwloc+2.12.0-af9daca5de88b067b0fe494e77bfe58b just build.
RPM installation stderr hwloc:
error: Failed dependencies:
	libnvidia-ml.so.1()(64bit) is needed by external+hwloc+2.12.0-af9daca5de88b067b0fe494e77bfe58b-1-1.x86_64

Failed to install RPM for hwloc
Build logs cleanup cudnn
Build successful cudnn.


[smuzaffar]

please test for CMSSW_15_1_CUDART_X

[cmsbuild]

Pull request #9828 was updated.

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45850/summary.html
COMMIT: 8c2b514
CMSSW: CMSSW_15_1_CUDART_X_2025-05-04-2300/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9828/45850/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

FATAL: malformed spec found while quering it. Command: 
source /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/rpm-env.sh ;  rpm -q --specfile /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/tmpspec-cuda-runtime --info --define "cmsdist_directory /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cmsdist" --define "compilerv 1231" --define "cmscompilerv 12" --define "cmsos el8_amd64" --define "almalinux_ver 8" --define "almalinux 8" --define "centos_ver 8" --define "centos 8" --define "rhel 8" --define "dist .el8" --define "el8 1" --define "package_vectorization %{nil}" --define "cmsswdata_version_link 1" --define "archfirst yes" --define "cmsBuild_bootstrap 1"  --define 'buildroot /foo'
Resulted in:

warning: line 30: It's not recommended to have unversioned Obsoletes: Obsoletes: external+cuda-runtime+1.0
error: line 439: Unknown tag: Privides: libnvidia-ml.so.1()(64bit)
error: query of specfile /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/tmpspec-cuda-runtime failed, can't parse
Traceback (most recent call last):
  File "./pkgtools/cmsBuild", line 5085, in 
    build(opts, args[1:], PKGFactory)
  File "./pkgtools/cmsBuild", line 4259, in build


[smuzaffar]

please test for CMSSW_15_1_CUDART_X

[cmsbuild]

Pull request #9828 was updated.

[cmsbuild]

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45851/summary.html
COMMIT: b617162
CMSSW: CMSSW_15_1_CUDART_X_2025-05-04-2300/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9828/45851/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @bsunanda Phase2-hgx363W Make an example to show the wafer characteristic of all valid DetIds of the silicon type cmssw#48003
• @bsunanda Phase2-hgx363U Update the list of parameters to be saved on DB for HGCal cmssw#47999
• @fwyzard Update CUDA to 12.8.1 #9819

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45851/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45851/git-merge-result

Build

I found compilation error when building:

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.8.1-c6bbcf8cc4b37a17804bfdc25b1286a3/bin/nvcc -dlink -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_CUDART_X_2025-05-04-2300/static/el8_amd64_gcc12 -lHeterogeneousTestCUDAKernel_nv -lHeterogeneousTestCUDADevice_nv -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_CUDART_X_2025-05-04-2300/biglib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_CUDART_X_2025-05-04-2300/lib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_CUDART_X_2025-05-04-2300/external/el8_amd64_gcc12/lib -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.8.1-c6bbcf8cc4b37a17804bfdc25b1286a3/lib64 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.8.1-c6bbcf8cc4b37a17804bfdc25b1286a3/lib64/stubs -lcudadevrt --diag-suppress 20014 -std=c++20 -O3 --generate-line-info --source-in-ptx --display-error-number --expt-relaxed-constexpr --extended-lambda -gencode arch=compute_60,code=[sm_60,compute_60] -gencode arch=compute_70,code=[sm_70,compute_70] -gencode arch=compute_75,code=[sm_75,compute_75] -gencode arch=compute_80,code=[sm_80,compute_80] -gencode arch=compute_89,code=[sm_89,compute_89] -gencode arch=compute_90,code=[sm_90,compute_90] -Wno-deprecated-gpu-targets -diag-suppress=3012 -diag-suppress=3189 -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -Xcudafe --gnu_version=120300 --cudart shared --compiler-options '-O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS  -std=c++20 -fPIC ' tmp/el8_amd64_gcc12/src/HeterogeneousTest/CUDAKernel/plugins/HeterogeneousTestCUDAKernelPlugins/CUDATestKernelAdditionAlgo.cu.o -o tmp/el8_amd64_gcc12/src/HeterogeneousTest/CUDAKernel/plugins/HeterogeneousTestCUDAKernelPlugins/HeterogeneousTestCUDAKernelPlugins_cudadlink.o
>> Building edm plugin tmp/el8_amd64_gcc12/src/HeterogeneousTest/CUDAKernel/plugins/HeterogeneousTestCUDAKernelPlugins/libHeterogeneousTestCUDAKernelPlugins.so
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DEIGEN_DONT_PARALLELIZE -DEIGEN_MAX_ALIGN_BYTES=64 -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -shared -Wl,-E    -Wl,-z,defs     tmp/el8_amd64_gcc12/src/HeterogeneousTest/CUDAKernel/plugins/HeterogeneousTestCUDAKernelPlugins/CUDATestKernelAdditionAlgo.cu.o tmp/el8_amd64_gcc12/src/HeterogeneousTest/CUDAKernel/plugins/HeterogeneousTestCUDAKernelPlugins/CUDATestKernelAdditionModule.cc.o tmp/el8_amd64_gcc12/src/HeterogeneousTest/CUDAKernel/plugins/HeterogeneousTestCUDAKernelPlugins/HeterogeneousTestCUDAKernelPlugins_cudadlink.o -o tmp/el8_amd64_gcc12/src/HeterogeneousTest/CUDAKernel/plugins/HeterogeneousTestCUDAKernelPlugins/libHeterogeneousTestCUDAKernelPlugins.so -Wl,-E -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,noexecstack -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_CUDART_X_2025-05-04-2300/biglib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_CUDART_X_2025-05-04-2300/lib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_CUDART_X_2025-05-04-2300/external/el8_amd64_gcc12/lib -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.8.1-c6bbcf8cc4b37a17804bfdc25b1286a3/lib64 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.8.1-c6bbcf8cc4b37a17804bfdc25b1286a3/lib64/stubs -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_CUDART_X_2025-05-04-2300/static/el8_amd64_gcc12 -lFWCoreFramework -lHeterogeneousCoreCUDAServices -lFWCoreCommon -lFWCoreServiceRegistry -lDataFormatsCommon -lFWCoreParameterSet -lHeterogeneousCoreCUDAUtilities -lFWCoreAbstractServices -lFWCoreMessageLogger -lDataFormatsProvenance -lFWCoreConcurrency -lFWCorePluginManager -lFWCoreReflection -Wl,--push-state -Wl,--no-as-needed -lHeterogeneousTestCUDAKernel -Wl,--pop-state -lFWCoreUtilities -lFWCoreVersion -Wl,--push-state -Wl,--no-as-needed -lHeterogeneousTestCUDADevice -Wl,--pop-state -lTree -lNet -lThread -lMathCore -lRIO -lboost_program_options -lCore -lboost_thread -lboost_date_time -lpcre -lbz2 -lcudart -lcudadevrt -lnvToolsExt -lnvidia-ml -luuid -ltbb -llzma -lz -lcuda -lfmt -lcms-md5 -lcrypt -ldl -lrt -lstdc++fs -ltinyxml2
Leaving library rule at src/HeterogeneousTest/CUDAKernel/plugins
@@@@ Running EDM write config for HeterogeneousTestCUDAKernelPlugins
error: edmWriteConfigs caught an exception while loading a plugin library.
The executable will return success (0) so scram will continue,
but no cfi files will be written.
An exception of category 'PluginLibraryLoadError' occurred.
Exception Message:
unable to load /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_CUDART_X_2025-05-04-2300/tmp/el8_amd64_gcc12/src/HeterogeneousTest/CUDAKernel/plugins/HeterogeneousTestCUDAKernelPlugins/libHeterogeneousTestCUDAKernelPlugins.so because libcudart.so.12: cannot open shared object file: No such file or directory


[smuzaffar]

please test for CMSSW_15_1_CUDART_X

[cmsbuild]

Pull request #9828 was updated.

[cmsbuild]

-1

Failed Tests: UnitTests rocmUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45892/summary.html
COMMIT: d31ae0d
CMSSW: CMSSW_15_1_CUDART_X_2025-05-05-1100/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9828/45892/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @wddgit Remove SubProcess related code from the DependencyGraph service cmssw#47996
• @Tizianop6 Vertex time reconstruction: added a flag whether to use MVA selection. MTD Primary Vertex Validation: added plots to check PID perfomance and track time assingment.  cmssw#47997
• @ftorrresd Nano comaprison report will show the WF name and id cmssw#47943
• @mmusich test_mille.py: dump output to stdout instead of file cmssw#48024
• @yiyangzha Add '22+'23 Scouting Glo-ParT Inference Facility cmssw#47988
• @bsunanda Phase2-hgx363Z Update the wafer information testing code to include placement index cmssw#48018
• @bsunanda Phase2-hgxc63AA Take care of the extra cells in the LDThree wafers in the V17 version of HGCal cmssw#48019
• @perrotta Update 2025 L1T menu in autoCond to L1Menu_Collisions2025_v1_1_1_xml cmssw#48013
• @SegmentLinking Refactor LST standalone to remove script-generated code cmssw#48014
• @bsunanda Phase2-hgx363X Make a setup like HGCal TB with 47 all-silicon layers of detectors (26 for EE and 21 for HE) with some lateral structure cmssw#48006
• @belforte Update crab-dev.spec #9831
• @cms-sw Update tag for RecoBTag-Combined to V01-30-00 #9829

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45892/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45892/git-merge-result

Unit Tests

I found 2 errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS
---> test TestDQMGUIUpload had ERRORS

ROCm Unit Tests

I found 3 errors in the following unit tests:

---> test testRocmSoALayoutAndView_t had ERRORS
---> test alpakaTestBufferROCmAsync had ERRORS
---> test alpakaTestRadixSortROCmAsync had ERRORS

Comparison Summary

Summary:

• You potentially added 32 lines to the logs
• Reco comparison results: 10978 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4037687
• DQMHistoTests: Total failures: 61566
• DQMHistoTests: Total nulls: 26
• DQMHistoTests: Total successes: 3976075
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 215 log files, 184 edm output root files, 50 DQM output files
• TriggerResults: found differences in 5 / 48 workflows

[cmsbuild]

Pull request #9828 was updated.

[smuzaffar]

please test for CMSSW_15_1_CUDART_X

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45936/summary.html
COMMIT: 064413c
CMSSW: CMSSW_15_1_CUDART_X_2025-05-05-1100/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9828/45936/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ '[' -e /cvmfs/projects.cern.ch/cms-restricted/x86_64/rhel8/external/cuda/12.8.1/nvvm/lib/libnvvm.so ']'
+ '[' -e /cvmfs/projects.cern.ch/cms-restricted/x86_64/rhel8/external/cuda/12.8.1/nvvm/lib/libnvvm.a ']'
+ echo 'ERROR: Unable to find nvvm/lib/libnvvm'
ERROR: Unable to find nvvm/lib/libnvvm
+ exit 1
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.SXsCeR (%install)


RPM build errors:
Macro expanded in comment on line 467: %{pkginstroot}/lib64



[smuzaffar]

please test for CMSSW_15_1_CUDART_X

[cmsbuild]

Pull request #9828 was updated.

[smuzaffar]

please test for CMSSW_15_1_CUDART_X

[cmsbuild]

Pull request #9828 was updated.

[cmsbuild]

-1

Failed Tests: rocmUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45996/summary.html
COMMIT: 1e6b3ec
CMSSW: CMSSW_15_1_CUDART_X_2025-05-07-2300/el8_amd64_gcc12
Additional Tests: CUDA,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9828/45996/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @kyungminparkdrums [15_1_X] Fix ECAL DQM crash from ME formatter when there is no luminosity block cmssw#48027
• @mmusich reduce memory allocations per event in string manipulation in SiStripMonitorTrack::analyze() cmssw#48026
• @mmusich add HLT trimmed pixel vertices monitoring in online DQM cmssw#48025
• @SegmentLinking Add pT3 DNN to LST for Improved Fake Rejection cmssw#47995
• @Parsifal-2045 Fix HLT MTV to work with singleIterPatatrack and LST procModifiers cmssw#48031
• @lrobertshaw [L1T] PR containing simulation and emulation related to jet mass reconstruction (Phase-2) cmssw#47792
• @SegmentLinking Migrate LST inputs to SoA collections cmssw#47793
• @JHiltbrand Add Config for Running HcalCalIsoTrkProducerFilter ALCARECO on new AlCaHcalIsoTrk AlCaRaw Stream cmssw#47970
• @iarspider Add ability to output JetTagClass to std::ostream cmssw#48020
• @bsunanda Phase2-hgx363AB Correct the dd4hep version for V19 geometry definition of HGCal cmssw#48029
• @makortel Use separate cache identifiers for different Principals, and add an option to use the Event cache identifier in SleepProducer cmssw#48038
• @drkovalskyi BPHNano size reduction cmssw#48033
• @cms-tsg-storm HLT menu development for 15_0_X (6/N) [15_1_X]  cmssw#48042
• @bsunanda Phase2-hgx363Y Add an example to list cell IDs of a given wafer type cmssw#48011
• @SegmentLinking LST pixelSeeds Eta Name Correction cmssw#48048
• @makortel Extend validated plugin testing cmssw#48015

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45996/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2e2d6e/45996/git-merge-result

ROCm Unit Tests

I found 2 errors in the following unit tests:

---> test testRocmSoALayoutAndView_t had ERRORS
---> test alpakaTestBufferROCmAsync had ERRORS

Comparison Summary

Summary:

• You potentially added 17 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 69 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4015624
• DQMHistoTests: Total failures: 9065
• DQMHistoTests: Total nulls: 50
• DQMHistoTests: Total successes: 4006489
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 166.443 KiB( 49 files compared)
• DQMHistoSizes: changed ( 139.001,... ): 0.001 KiB HLT/Filters
• DQMHistoSizes: changed ( 145.014,... ): 6.954 KiB HLT/Filters
• DQMHistoSizes: changed ( 16834.0,... ): 48.450 KiB HLT/HLTEgammaValidation
• Checked 215 log files, 184 edm output root files, 50 DQM output files
• TriggerResults: found differences in 10 / 48 workflows