runTheMatrix.py option --maxmem_profile not being set in baseline tests

[gartung]

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @gartung.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[gartung]

Could be related to changes made in this cms-bot PR 2458
or this change

[gartung]

https://cmssdt.cern.ch/jenkins/job/ib-run-baseline/15890/consoleText

++ runTheMatrix.py -j 16 --job-reports -l 2500.201,250202.181,29834.999,29700.0,29696.0,29634.911,29634.75,29634.0,24834.911,17034.0,16834.0,14234.0,14034.0,13234.0,13034.0,12846.0,12834.7,12834.0,12434.0,11634.0,10224.0,1001.0,1000.0,25202.0,312.0,101.0,1330.0,1306.0,145.713,145.604,145.5,145.408,145.301,145.202,145.104,145.014,141.042,140.56,140.045,139.001,136.874,136.793,136.731,135.4,25.0,9.0,8.0,7.3,5.1,4.53,4.22 --command ' --maxmem_profile --customise Validation/Performance/TimeMemoryJobReport.customiseWithTimeMemoryJobReport '

Appears that last part of command wasn't propagated.

[gartung]

The command was customized without --maxmem_profile
https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-baseline-tests/CMSSW_15_1_X_2025-05-06-2300/el8_amd64_gcc12/AuthenticAMD/matrix-results/matrixTests.15879.log

+ runTheMatrix.py -j 16 --job-reports -l 2500.201,250202.181,29834.999,29700.0,29696.0,29634.911,29634.75,29634.0,24834.911,17034.0,16834.0,14234.0,14034.0,13234.0,13034.0,12846.0,12834.0,12434.0,11634.0,10224.0,1001.0,1000.0,25202.0,312.0,101.0,1330.0,1306.0,145.713,145.604,145.5,145.408,145.301,145.202,145.104,145.014,141.042,140.56,140.045,139.001,136.874,136.793,136.731,135.4,25.0,9.0,8.0,7.3,5.1,4.53,4.22 --command ' --customise Validation/Performance/TimeMemoryJobReport.customiseWithTimeMemoryJobReport '

[gartung]

I guess this is where it was removed

cms-bot/pr_testing/test_multiple_prs.sh

Line 1496 in 641472a

if [ "${ENABLE_MEMORY_PROFILE}" = "true" -a "${ex_type}" = "ROCM" ] ; then

[iarspider]

I will look into it. One thing to notice that this line was added to avoid running memory profiling for ROCm tests (but only for ROCm tests) since it is known to crash - cms-sw/cmssw#47450

[smuzaffar]

@gartung , I just looked in to it and looks like baseline jobs are using --maxmem_profile e.g. baseline for last night IB has properly added this option to runTheMatrix

• https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-baseline-tests/CMSSW_15_1_X_2025-05-15-2300/el8_amd64_gcc12/AuthenticAMD/matrix-results/matrixTests.16066.log
• https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-baseline-tests/CMSSW_15_1_X_2025-05-15-2300/el8_amd64_gcc12/AuthenticAMD/matrix-results/matrixTests.16072.log
• https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-baseline-tests/CMSSW_15_1_X_2025-05-15-2300/el8_amd64_gcc12/AuthenticAMD/matrix-results/matrixTests.16073.log

Do you have an example where it did not work ( your last ref to logs are no more available)

[makortel]

Here cms-sw/cmssw#48041 (comment) is a case where the --maxmem_profile seems to be visible in baseline, but not in the PR tests

• maxmem report is empty https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_15_1_X_2025-05-12-1100+497a37/68458/maxmem-comparison/
• PR test log https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-497a37/46092/runTheMatrix-results/matrixTests.log
• baseline log https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-baseline-tests/CMSSW_15_1_X_2025-05-12-1100/el8_amd64_gcc12/AuthenticAMD/matrix-results/matrixTests.15970.log

[smuzaffar]

ah ok , thanks @makortel . In that case it PR was run for el8_amd64_gcc14 which is not treated as production arch that is why bot did not add the option for PR relvals while baseline is generated for el8_amd64_gcc12 (i.e. the productuionarch) so baseline has used it.

Anyway, I am fixing/cleaning usage of --maxmem_profile here https://github.com/cms-sw/cms-bot/pull/2506/files . I can update it to enable this for all archs for development release cycle

[makortel]

Thanks @smuzaffar, I wasn't paying attention to the archs (when looking for recent PR tests that would show the problem). I think it's still ok to not run the maxmemory for non-production archs.

I'll monitor PR tests in the coming days more carefully to see if I see more of these symptoms.

[smuzaffar]

ok, so I will not change the current behavior . maxmem profile will only run for production arch.

I guess this issue can be closed then