Extend CMSSW to a distributed application over MPI [16.0.x] #49930

fwyzard · 2026-01-24T11:39:52Z

PR description:

Let multiple CMSSW processes on the same or different machines coordinate event processing and transfer data products over MPI.

The implementation is based on four CMSSW modules. Two are responsible for setting up the communication channels and coordinate the event processing:

a "remote controller" called MPIController;
a "remote source" called MPISource; and two are responsible for the transfer of data products:
a "sender" called MPISender;
a "receiver" called MPIReceiver.

Data products can be serialised and transferred using the trivial serialisation from HeterogeneousCore/TrivialSerialisation - if available - or the ROOT-based serialisation.

Various tests are used to validate the implementation: matching the local and remote event id, transferring SoA products with trivial serialisation, and transferring legacy product with ROOT serialisation.

PR validation:

See #32632.

If this PR is a backport please specify the original PR and why you need to backport that PR.

Backport of #32632 to 16.0.x for testing online in parallel to the 2026 data taking.

Let multiple CMSSW processes on the same or different machines coordinate event processing and transfer data products over MPI. The implementation is based on four CMSSW modules. Two are responsible for setting up the communication channels and coordinate the event processing: - a "remote controller" called MPIController; - a "remote source" called MPISource; and two are responsible for the transfer of data products: - a "sender" called MPISender; - a "receiver" called MPIReceiver. Data products can be serialised and transferred using the trivial serialisation from HeterogeneousCore/TrivialSerialisation - if available - or the ROOT-based serialisation. Various tests are used to validate the implementation: matching the local and remote event id, transferring SoA products with trivial serialisation, and transferring legacy product with ROOT serialisation. Co-authored-by: Andrea Bocci <[email protected]> Co-authored-by: Anna Polova <[email protected]> Co-authored-by: Mario Gonzalez <[email protected]>

cmsbuild · 2026-01-24T11:40:17Z

A new Pull Request was created by @fwyzard for CMSSW_16_0_X.

It involves the following packages:

HeterogeneousCore/MPICore (heterogeneous)

@cmsbuild, @fwyzard, @makortel can you please review it and eventually sign? Thanks.
@makortel, @rovere this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

Backported from Multiple improvements to HeterogeneousCore/MPICore #49935

cmsbuild · 2026-01-24T11:40:17Z

cms-bot internal usage

fwyzard · 2026-01-24T11:40:33Z

please test with #49882

fwyzard · 2026-01-24T11:40:38Z

+heterogeneous

cmsbuild · 2026-01-24T11:40:56Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_16_0_X IBs after it passes the integration tests and once validation in the development release cycle CMSSW_16_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @ftenchini, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: #49882

fwyzard · 2026-01-24T11:42:04Z

type ngt

cmsbuild · 2026-01-24T14:16:15Z

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca6da2/50873/summary.html
COMMIT: c4ae7ce
CMSSW: CMSSW_16_0_X_2026-01-23-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49930/50873/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 13 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4149103
DQMHistoTests: Total failures: 73
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4149010
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 227 log files, 198 edm output root files, 53 DQM output files
TriggerResults: no differences found

fwyzard · 2026-01-26T17:22:29Z

Ops, looks like I deleted the branch by mistake :-/

fwyzard · 2026-01-26T17:43:35Z

please test

cmsbuild · 2026-01-26T19:57:51Z

-1

Failed Tests: UnitTests
Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca6da2/50908/summary.html
COMMIT: 8ba3e94
CMSSW: CMSSW_16_0_X_2026-01-26-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49930/50908/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testMPISoATransfer had ERRORS

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4149103
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4149077
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 227 log files, 198 edm output root files, 53 DQM output files
TriggerResults: no differences found

fwyzard · 2026-01-27T16:10:18Z

+heterogeneous

cmsbuild · 2026-01-27T16:10:22Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_16_0_X IBs after it passes the integration tests and once validation in the development release cycle CMSSW_16_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

fwyzard · 2026-01-27T16:11:05Z

please test with #49882

cmsbuild · 2026-01-27T18:01:00Z

-1

Failed Tests: UnitTests
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ca6da2/50938/summary.html
COMMIT: c4ae7ce
CMSSW: CMSSW_16_0_X_2026-01-27-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49930/50938/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test testMPISoATransfer had ERRORS

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 12 differences found in the comparisons
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4149103
DQMHistoTests: Total failures: 101
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4148982
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 227 log files, 198 edm output root files, 53 DQM output files
TriggerResults: no differences found

fwyzard · 2026-01-28T10:24:06Z

urgent

aimed at CMSSW_16_0_0

fwyzard · 2026-01-28T11:50:13Z

Unit tests worked locally with CMSSW_16_0_X_2026-01-27-2300, on top of #49882 and #49953.
Let's get those merged and re-run here.

fwyzard · 2026-01-28T12:15:31Z

hold

Need to merge #49882 first.

cmsbuild · 2026-01-28T12:15:51Z

Pull request has been put on hold by @fwyzard
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

cmsbuild added this to the CMSSW_16_0_X milestone Jan 24, 2026

cmsbuild added pending-signatures tests-pending orp-pending heterogeneous-pending changes-dataformats labels Jan 24, 2026

fwyzard changed the title ~~Extend CMSSW to a distributed application over MPI~~ Extend CMSSW to a distributed application over MPI [16.0.x] Jan 24, 2026

cmsbuild added tests-started requires-external and removed tests-pending labels Jan 24, 2026

cmsbuild added fully-signed heterogeneous-approved and removed pending-signatures heterogeneous-pending labels Jan 24, 2026

cmsbuild mentioned this pull request Jan 24, 2026

[Backport to 16_0_X (#49734)] Remove PortableHostMultiCollection and PortableDeviceMultiCollection #49882

Merged

cmsbuild added the ngt label Jan 24, 2026

cmsbuild added tests-approved and removed tests-started labels Jan 24, 2026

fwyzard closed this Jan 26, 2026

fwyzard deleted the MPI_updates_160x branch January 26, 2026 08:14

fwyzard restored the MPI_updates_160x branch January 26, 2026 17:22

cmsbuild added tests-started and removed tests-pending labels Jan 26, 2026

cmsbuild added tests-rejected and removed tests-started labels Jan 26, 2026

fwyzard force-pushed the MPI_updates_160x branch from 8ba3e94 to c4ae7ce Compare January 27, 2026 16:09

cmsbuild added tests-approved tests-started and removed tests-rejected tests-approved labels Jan 27, 2026

cmsbuild added fully-signed heterogeneous-approved and removed pending-signatures heterogeneous-pending labels Jan 27, 2026

cmsbuild added the requires-external label Jan 27, 2026

cmsbuild added tests-rejected and removed tests-started labels Jan 27, 2026

cmsbuild added the urgent label Jan 28, 2026

cmsbuild added pending-signatures hold and removed fully-signed labels Jan 28, 2026

Extend CMSSW to a distributed application over MPI [16.0.x] #49930

Are you sure you want to change the base?

Extend CMSSW to a distributed application over MPI [16.0.x] #49930

Conversation

fwyzard commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR.

Uh oh!

cmsbuild commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fwyzard commented Jan 24, 2026

Uh oh!

fwyzard commented Jan 24, 2026

Uh oh!

cmsbuild commented Jan 24, 2026

Uh oh!

fwyzard commented Jan 24, 2026

Uh oh!

cmsbuild commented Jan 24, 2026

Comparison Summary

Uh oh!

fwyzard commented Jan 26, 2026

Uh oh!

fwyzard commented Jan 26, 2026

Uh oh!

cmsbuild commented Jan 26, 2026

Failed Unit Tests

Comparison Summary

Uh oh!

fwyzard commented Jan 27, 2026

Uh oh!

cmsbuild commented Jan 27, 2026

Uh oh!

fwyzard commented Jan 27, 2026

Uh oh!

cmsbuild commented Jan 27, 2026

Failed Unit Tests

Comparison Summary

Uh oh!

fwyzard commented Jan 28, 2026

Uh oh!

fwyzard commented Jan 28, 2026

Uh oh!

fwyzard commented Jan 28, 2026

Uh oh!

cmsbuild commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fwyzard commented Jan 24, 2026 •

edited

Loading

cmsbuild commented Jan 24, 2026 •

edited

Loading

cmsbuild commented Jan 24, 2026 •

edited

Loading