Trivial serialisation plugin mechanism

[fwyzard]

PR description:

Introduce ngt::SerialiserFactory, a plugin-based trivial serialiser, and the serialisation plugins for all PortableHostCollection- and PortableHostObject-derived data formats.

This relies on some new components:

• ngt::AnyBuffer: ngt::AnyBuffer behaves like std::any, with two differences: it can only be used with trivially copyable types, and provides access to the underlying memory buffer to allow memcpy'ing its content.
• the ngt::MemoryCopyTraits<T> type traits. The struct ngt::MemoryCopyTraits<T> should be specialised for the types that can be safely memcpy'ed.

And is accompanied by various tests:

• Catch2 tests for the ngt::MemoryCopyTraits.
• Catch2 tests for the ngt::SerialiserFactory.
• integration tasts based on the ngt::GenericCloner.

The ngt::GenericCloner is included to demonstrate how the new functionalities can be used, and to implement various integration tests. This EDProducer will clone all the event products declared in its configuration, using either the plugin-based NGT trivial serialisation, or the products' ROOT dictionaries.

PR tests:

The new unit tests pass.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49475/46953

• There are other open Pull requests which might conflict with changes you have proposed:
  • File DataFormats/Portable/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Enabling usage of filters in MPI offloading #49233
  • File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Enabling usage of filters in MPI offloading #49233
  • File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): Added AoSWrapper and transposition function #49072
  • File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): Added AoSWrapper and transposition function #49072
  • File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072, Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152, Enabling usage of filters in MPI offloading #49233
  • File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152, Enabling usage of filters in MPI offloading #49233, [Do not merge] LLVM21 clang-format suggestions #49293
  • File DataFormats/PortableTestObjects/plugins/BuildFile.xml modified in PR(s): Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152
  • File DataFormats/SiPixelClusterSoA/BuildFile.xml modified in PR(s): Add missing dependencies to SiPixelClusterSoA #49471
  • File FWCore/Framework/interface/WrapperBaseHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152, Enabling usage of filters in MPI offloading #49233, Specialise Handle and OrphanHandle for WrapperBase #49449
  • File FWCore/Framework/interface/WrapperBaseOrphanHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152, Enabling usage of filters in MPI offloading #49233, Specialise Handle and OrphanHandle for WrapperBase #49449
  • File FWCore/Framework/src/WrapperBaseHandle.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152, Enabling usage of filters in MPI offloading #49233, Specialise Handle and OrphanHandle for WrapperBase #49449
  • File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Enabling usage of filters in MPI offloading #49233, Specialise Handle and OrphanHandle for WrapperBase #49449
  • File FWCore/Integration/plugins/WrapperBaseProducer.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Enabling usage of filters in MPI offloading #49233, Specialise Handle and OrphanHandle for WrapperBase #49449
  • File FWCore/Integration/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Enabling usage of filters in MPI offloading #49233, Specialise Handle and OrphanHandle for WrapperBase #49449
  • File FWCore/Integration/test/testWrapperBaseProducer_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Enabling usage of filters in MPI offloading #49233, Specialise Handle and OrphanHandle for WrapperBase #49449

[cmsbuild]

A new Pull Request was created by @fwyzard for master.

It involves the following packages:

• DataFormats/BeamSpot (reconstruction)
• DataFormats/EcalDigi (simulation)
• DataFormats/EcalRecHit (reconstruction)
• DataFormats/HGCalDigi (simulation)
• DataFormats/HGCalReco (reconstruction)
• DataFormats/HcalDigi (simulation)
• DataFormats/HcalRecHit (reconstruction)
• DataFormats/ParticleFlowReco (reconstruction)
• DataFormats/Portable (heterogeneous)
• DataFormats/PortableTestObjects (heterogeneous)
• DataFormats/SiPixelClusterSoA (heterogeneous, reconstruction)
• DataFormats/SiPixelDigiSoA (heterogeneous, reconstruction)
• DataFormats/TrackSoA (heterogeneous, reconstruction)
• DataFormats/TrackingRecHitSoA (heterogeneous, reconstruction)
• DataFormats/VertexSoA (heterogeneous, reconstruction)
• FWCore/Framework (core)
• FWCore/Integration (core)
• HeterogeneousCore/SerialisationCore (****)

The following packages do not have a category, yet:

HeterogeneousCore/SerialisationCore
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@Dr15Jones, @civanch, @cmsbuild, @fwyzard, @jfernan2, @kpedro88, @makortel, @mandrenguyen, @mdhildreth, @smuzaffar, @srimanob can you please review it and eventually sign? Thanks.
@IzaakWN, @ReyerBand, @VinInn, @VourMa, @abdoulline, @argiro, @bsunanda, @dkotlins, @elusian, @ferencek, @gpetruc, @hatakeyamak, @lgray, @makortel, @mariadalfonso, @missirol, @mmasciov, @mmusich, @mroguljic, @mtosi, @pfs, @rchatter, @rovere, @thomreis, @tsusa, @wang0jin, @wddgit this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[fwyzard]

Should this package be simply HeterogeneousCore/Serialisation ?
Or something more explicit ?

[fwyzard]

please test

[fwyzard]

type ngt

[fwyzard]

@ghyls FYI

[cmsbuild]

+1

Size: This PR adds an extra 100KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7ef6e7/49681/summary.html
COMMIT: 63bc511
CMSSW: CMSSW_16_0_X_2025-11-25-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49475/49681/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 2 lines to the logs
• Reco comparison results: 3 differences found in the comparisons
• Reco comparison had 4 failed jobs
• DQMHistoTests: Total files compared: 55
• DQMHistoTests: Total histograms compared: 4503038
• DQMHistoTests: Total failures: 122
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4502896
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 54 files compared)
• Checked 235 log files, 208 edm output root files, 55 DQM output files
• TriggerResults: no differences found

[cmsbuild]

+1

Size: This PR adds an extra 84KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7ef6e7/50312/summary.html
COMMIT: 314b7b7
CMSSW: CMSSW_16_0_X_2025-12-16-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49475/50312/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 1 lines from the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 5 differences found in the comparisons
• Reco comparison had 4 failed jobs
• DQMHistoTests: Total files compared: 55
• DQMHistoTests: Total histograms compared: 4513634
• DQMHistoTests: Total failures: 14
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4513600
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 54 files compared)
• Checked 235 log files, 208 edm output root files, 55 DQM output files
• TriggerResults: no differences found

[jfernan2]

+1

[makortel]

Comparison differences are related to #47071

[makortel]

+heterogeneous

[mandrenguyen]

@cms-sw/simulation-l2 this is the last pending item for the next pre-release. Any chance you can take a look before the overnight IB?

[kpedro88]

+simulation
for minor additions in Digi data format packages

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

+1

[smuzaffar]

@fwyzard @makortel , fwlite builds are broken as some of the fwlite dataformat packages now see DataFormats/TrivialSerialisation/interface/MemoryCopyTraits.h [a]. I can add only DataFormats/TrivialSerialisation/interface/MemoryCopyTraits.h in to fwlite_build_set to fix the fwlite builds . Let me know if this is acceptable

[a] https://cmssdt.cern.ch/SDT/cgi-bin/showBuildLogs.py/fwlite/el8_amd64_gcc13/www/thu/16.0-thu-11/CMSSW_16_0_X_2025-12-18-1100

In file included from src/DataFormats/BeamSpot/interface/BeamSpotHost.h:5,
                 from src/DataFormats/BeamSpot/src/classes.cc:1:
src/DataFormats/Portable/interface/PortableHostObject.h:11:10: fatal error: DataFormats/TrivialSerialisation/interface/MemoryCopyTraits.h: No such file or directory
   11 | #include "DataFormats/TrivialSerialisation/interface/MemoryCopyTraits.h"

[fwyzard]

The DataFormats/TrivialSerialisation packages does not have any dependencies, so it should be fine to add it.

[makortel]

I can add only DataFormats/TrivialSerialisation/interface/MemoryCopyTraits.h in to fwlite_build_set to fix the fwlite builds . Let me know if this is acceptable

I'd add the entire DataFormats/TrivialSerialisation package. As @fwyzard wrote, that package does not have any dependencies.

[smuzaffar]

the package plugin directory uses HeterogeneousCore/TrivialSerialisation , sO I have to add it too then

[makortel]

In the core meeting on Tuesday we talked about ignoring the plugin directories of *DataFormats/* in FWLite. If we do that, we can avoid adding HeterogeneousCore/TrivialSerialisation (and I'd like to avoid that).