Implement a `GenericCloner` test module based on an `edmplugin::PluginFactory` #49152

ghyls · 2025-10-13T21:45:29Z

PR description:

Implements an alternative to #47504 based on a plugin factory. This PR and #47504 provide the same functionality:

Specialise Handle and OrphanHandle for WrapperBase: this lets users produce and consume collections via their wrapper, using their run time type information (e.g. type name, type id, ...) instead of the compile time types. Implement integration test for these specialisations.

Introduce a TrivialCopyTraits structure, that should be specialised for types that can be safely memcpyed, without requiring a full ROOT streamer de/serialisation. Specialise the TrivialCopyTraits for arithmetic types and vectors of arithmetic types, and all PortableObject and PortableCollection types.

Implement edmtest::GenericCloner: this EDProducer will consume and clone all the event products declared by its configuration; a direct memcpy will be used for types that support the TrivialCopyTraits interface; the ROOT dictionaries will be used to serialise and deserialise the other types.

The plugin factory returns TrivialSerialiserSource objects for a given type. TrivialSerialisers can then be initialized by calling TrivialSerialiserSource::initialize with a WrapperBase object. The initialized TrivialSerialiser is const or non const depending on the constness of the WrapperBase object used to initialize it. A pointer to this WrapperBase object is stored in TrivialSerialiserBase as const edm::WrapperBase*.

PR validation:

Various unit tests have been included:

FWCore/TestModules/test/testGenericCloner_cfg.py: produce, clone and validate products of type int, std::string, and edm::EventID. does not clone a transient cms.int32().
DataFormats/Portable/test/testGenericCloner_cfg.py: Produce, clone and validate a portable object, a portable collection, and some portable multicollections.
DataFormats/Common/test/test_catch2_TrivialCopyTraits.cpp: Implements a TrivialCopyTraits specialization for a simple struct. Tests the methods of TrivialCopyTraits with the types int, double, std::vector<float>, and the struct.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild · 2025-10-13T21:45:58Z

cms-bot internal usage

cmsbuild · 2025-10-13T21:48:00Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49152/46438

Found files with invalid states:
- DataFormats/Common/interface/TrivialCopyTraits.h:
  - Added: fea94c6
  - Deleted: 323ff5e
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/AnyBuffer.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File DataFormats/Common/interface/WrapperBase.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Added SoABlocks feature #48629, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/Framework/interface/WrapperBaseHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/Framework/interface/WrapperBaseOrphanHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/Framework/src/WrapperBaseHandle.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49152/46438/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49152/46438/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2025-10-13T21:55:44Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49152/46439

Found files with invalid states:
- DataFormats/Common/interface/TrivialCopyTraits.h:
  - Added: fea94c6
  - Deleted: 323ff5e
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/AnyBuffer.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File DataFormats/Common/interface/WrapperBase.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Added SoABlocks feature #48629, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/Framework/interface/WrapperBaseHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/Framework/interface/WrapperBaseOrphanHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/Framework/src/WrapperBaseHandle.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504

cmsbuild · 2025-10-13T21:56:06Z

A new Pull Request was created by @ghyls for master.

It involves the following packages:

DataFormats/Common (core)
DataFormats/Portable (heterogeneous)
FWCore/Framework (core)
FWCore/TestModules (core)
TrivialSerialisation/Common (****)

The following packages do not have a category, yet:

TrivialSerialisation/Common
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@Dr15Jones, @cmsbuild, @fwyzard, @makortel, @smuzaffar can you please review it and eventually sign? Thanks.
@makortel, @missirol, @mmusich, @rovere, @wddgit this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

fwyzard

First round of review.

fwyzard · 2025-10-14T10:08:36Z

TrivialSerialisation/Common/BuildFile.xml

+<!-- <use name="CondFormats/CSCObjects"/> -->
+<!-- <use name="CondFormats/DataRecord"/> -->
+<!-- <use name="DataFormats/MuonDetId"/> -->


please remove the commented-out lines

fwyzard · 2025-10-14T10:15:50Z

TrivialSerialisation/Common/interface/TrivialSerialiser.h

+#include "DataFormats/Common/interface/Wrapper.h"
+#include "TrivialSerialisation/Common/interface/TrivialSerialiserBase.h"
+#include "TrivialSerialisation/Common/interface/TrivialCopyTraits.h"
+


Can you add explicit includes for all the types being used, like edm::AnyBuffer, edm::Exception, etc ?

fwyzard · 2025-10-14T10:17:24Z

TrivialSerialisation/Common/interface/TrivialSerialiserSource.h

+
+}  // namespace ngt
+
+#endif  // TrivialSerialisation_Common_TrivialSerialiserSource_h


Missing end-of-line at the end of the file.

fwyzard · 2025-10-14T10:17:42Z

TrivialSerialisation/Common/interface/TrivialSerialiserSourceBase.h

+namespace ngt {
+  class TrivialSerialiserSourceBase {
+  public:
+    TrivialSerialiserSourceBase() {};


Suggested change

TrivialSerialiserSourceBase() {};

TrivialSerialiserSourceBase() = default;

?

Yes, thank you, it should be = default

fwyzard · 2025-10-14T10:28:22Z

DataFormats/Common/interface/WrapperBase.h

 #include "DataFormats/Common/interface/FillViewHelperVector.h"
 #include "DataFormats/Provenance/interface/ViewTypeChecker.h"

+#include <span>


Not needed ?

fwyzard · 2025-10-14T10:28:30Z

DataFormats/Common/interface/WrapperBase.h


 ----------------------------------------------------------------------*/

+#include "DataFormats/Common/interface/AnyBuffer.h"


Not needed ?

fwyzard · 2025-10-14T10:28:55Z

DataFormats/Common/interface/Wrapper.h

 #include <algorithm>
 #include <cassert>
 #include <memory>
+#include <span>


Not needed ?

fwyzard · 2025-10-14T10:29:22Z

DataFormats/Common/interface/Wrapper.h

 #include "DataFormats/Common/interface/WrapperDetail.h"
 #include "DataFormats/Provenance/interface/ProductID.h"
+#include "FWCore/Utilities/interface/EDMException.h"
+#include "FWCore/Utilities/interface/TypeDemangler.h"


Not needed ?

fwyzard · 2025-10-14T17:12:54Z

DataFormats/Common/interface/TrivialCopyTraits.h

I think this file should go under DataFormats/Common ?
The interface described by the trait can also be used independently from the serialisation.

Can you add some unit tests for the traits ?
For example, check that some types (e.g. int or std::vector<float>) are properly supported, that some other types (e.g. std::map) are not supported, and that you can write a simple type and specialise the traits for it.

fwyzard · 2025-10-14T17:18:53Z

TrivialSerialisation/Common/interface/TrivialSerialiserSource.h

+
+namespace ngt {
+  template <typename T>
+  class TrivialSerialiserSource : public TrivialSerialiserSourceBase {


I think we need to find a name that does not contain Source, because Source has a specific meaning in CMSSW.

Right, didn't think about that... TrivialSerialiserFactory and TrivialSerialiserBase are already taken, so maybe TrivialSerialiserGenerator could be a better option... Anyway, as you noticed the naming scheme is a bit complicated now:

I would suggest renaming:
TrivialSerialiserSourceFactory -> SerialiserFactory
TrivialSerialiserSourceBase -> SerialiserBase
TrivialSerialiserSource -> Serialiser

and keep TrivialSerialiser and TrivialSerialiserBase as they are.

So we would get Serialiser plugins from the factory, and use them to initialise const / mutable (readers / writers) TrivialSerialiser objects.

fwyzard · 2025-10-14T17:20:58Z

Can you add an example of how one will use this with SoA types ?
For example, how do we support the types under DataFormats/PortableTestObjects ?

cmsbuild · 2025-10-15T09:44:23Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49152/46462

Found files with invalid states:
- DataFormats/Common/interface/TrivialCopyTraits.h:
  - Added: fea94c6
  - Deleted: 323ff5e
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/AnyBuffer.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File DataFormats/Common/interface/WrapperBase.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Added SoABlocks feature #48629, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/Framework/interface/WrapperBaseHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/Framework/interface/WrapperBaseOrphanHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/Framework/src/WrapperBaseHandle.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49152/46462/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49152/46462/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2025-10-30T14:32:17Z

Pull request #49152 was updated.

Co-authored-by: Andrea Bocci <andrea.bocci@cern.ch>

…tMulticollection, and PortableHostObjects Co-authored-by: Andrea Bocci <andrea.bocci@cern.ch>

…td types

Co-authored-by: Andrea Bocci <andrea.bocci@cern.ch>

cmsbuild · 2025-10-30T15:28:19Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49152/46623

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/AnyBuffer.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233, Introduce ngt::AnyBuffer #49235
- File DataFormats/Common/interface/TrivialCopyTraits.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233, Add a const overload of bareProduct() and a markAsPresent() method to Wrapper.h #49236
- File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Added SoABlocks feature #48629, Added AoSWrapper and transposition function #49072, Enabling usage of filters in MPI offloading #49233
- File DataFormats/Portable/interface/PortableHostObject.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233
- File FWCore/Framework/interface/WrapperBaseHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233
- File FWCore/Framework/interface/WrapperBaseOrphanHandle.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233
- File FWCore/Framework/src/WrapperBaseHandle.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Enabling usage of filters in MPI offloading #49233

cmsbuild · 2025-10-30T15:28:43Z

Pull request #49152 was updated.

cmsbuild · 2025-12-18T12:59:45Z

Milestone for this pull request has been moved to CMSSW_16_1_X. Please open a backport if it should also go in to CMSSW_16_0_X.

makortel · 2025-12-18T14:56:55Z

Did #49475 supersede this PR?

ghyls · 2025-12-18T15:04:56Z

Did #49475 supersede this PR?

Yes it did. I'm closing this one, since #49475 is already merged.
Thank you!

cmsbuild added this to the CMSSW_16_0_X milestone Oct 13, 2025

cmsbuild added core-pending pending-signatures tests-pending orp-pending new-package-pending code-checks-pending heterogeneous-pending labels Oct 13, 2025

cmsbuild added code-checks-rejected code-checks-pending and removed code-checks-pending code-checks-rejected labels Oct 13, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 13, 2025

cmsbuild mentioned this pull request Oct 14, 2025

Added SoABlocks feature #48629

Merged

fwyzard suggested changes Oct 14, 2025

View reviewed changes

ghyls force-pushed the devel_implement_generic_product_tools-v3 branch from 923af7a to ff045af Compare October 15, 2025 09:42

cmsbuild added code-checks-pending and removed code-checks-approved labels Oct 15, 2025

cmsbuild added code-checks-rejected and removed code-checks-pending labels Oct 15, 2025

ghyls force-pushed the devel_implement_generic_product_tools-v3 branch from ff045af to a54d2bb Compare October 15, 2025 10:24

cmsbuild added code-checks-pending and removed code-checks-rejected labels Oct 15, 2025

ghyls and others added 6 commits October 30, 2025 16:06

Introduce TrivialCopyTraits and TrivialSerialiser

8b9e447

Co-authored-by: Andrea Bocci <andrea.bocci@cern.ch>

Specialize Handle and OrphanHandle for WrapperBase

787a7c7

Introduced the Serialiser edmplugin factory

2c22926

Co-authored-by: Andrea Bocci <andrea.bocci@cern.ch>

Specialized TrivialCopyTraits for PortableHostCollection, PortableHos…

d232b74

…tMulticollection, and PortableHostObjects Co-authored-by: Andrea Bocci <andrea.bocci@cern.ch>

Added Serialiser plugins for test PortableHostCollections and basic s…

79955cd

…td types

Added unit tests for TrivialCopyTraits and the Serialiser plugin factory

94d5ad8

Co-authored-by: Andrea Bocci <andrea.bocci@cern.ch>

ghyls force-pushed the devel_implement_generic_product_tools-v3 branch from b0ef6bd to 94d5ad8 Compare October 30, 2025 15:26

cmsbuild added code-checks-pending and removed code-checks-approved labels Oct 30, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 30, 2025

This was referenced Nov 18, 2025

Add bareProduct() const method #49415

Merged

Specialise Handle and OrphanHandle for WrapperBase #49449

Merged

Trivial serialisation plugin mechanism #49475

Merged

cmsbuild mentioned this pull request Nov 29, 2025

Extend CMSSW to a distributed application over MPI #32632

Merged

cmsbuild modified the milestones: CMSSW_16_0_X, CMSSW_16_1_X Dec 18, 2025

ghyls closed this Dec 18, 2025


		} // namespace ngt

		#endif // TrivialSerialisation_Common_TrivialSerialiserSource_h No newline at end of file

	TrivialSerialiserSourceBase() {};
	TrivialSerialiserSourceBase() = default;


		----------------------------------------------------------------------*/

		#include "DataFormats/Common/interface/AnyBuffer.h"

Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152

Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152

Uh oh!

Conversation

ghyls commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

cmsbuild commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Oct 13, 2025

Uh oh!

cmsbuild commented Oct 13, 2025

Uh oh!

cmsbuild commented Oct 13, 2025

Uh oh!

fwyzard left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fwyzard commented Oct 14, 2025

Uh oh!

cmsbuild commented Oct 15, 2025

Uh oh!

cmsbuild commented Oct 30, 2025

Uh oh!

cmsbuild commented Oct 30, 2025

Uh oh!

cmsbuild commented Oct 30, 2025

Uh oh!

cmsbuild commented Dec 18, 2025

Uh oh!

makortel commented Dec 18, 2025

Uh oh!

ghyls commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Implement a `GenericCloner` test module based on an `edmplugin::PluginFactory` #49152

Implement a `GenericCloner` test module based on an `edmplugin::PluginFactory` #49152

ghyls commented Oct 13, 2025 •

edited

Loading

cmsbuild commented Oct 13, 2025 •

edited

Loading

ghyls commented Dec 18, 2025 •

edited

Loading