Implement a `GenericCloner` test module #47504

fwyzard · 2025-03-04T20:52:59Z

PR description:

Specialise Handle and OrphanHandle for WrapperBase: this lets users produce and consume collections via their wrapper, using their run time type information (e.g. type name, type id, ...) instead of the compile time types. Implement integration test for this specialisations.

Introduce a MemcpyTraits structure, that should be specialised for types that can be safely memcpyed, without requiring a full ROOT streamer de/serialisation.
Specialise the MemcpyTraits for arithmetic types and vectors of arithmetic types, and all PortableObject and PortableCollection types.

Implement edmtest::GenericCloner: this EDProducer will consume and clone all the event products declared by its configuration; a direct memcpy will be used for types that support the MemcpyTraits interface; the ROOT dictionaries will be use to serialise and deserialise the other tyepes.

PR validation:

Unit tests pass.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

May be backported to 15.0.x as part of the MPI work.

This EDProducer will clone all the event products declared by its configuration, using their ROOT dictionaries.

cmsbuild · 2025-03-04T20:53:25Z

cms-bot internal usage

fwyzard · 2025-03-04T20:53:56Z

@makortel what do you think of this approach ?

I don't mind moving stuff around, of course: consider this more of a prototype.

fwyzard · 2025-03-04T20:54:04Z

please test

cmsbuild · 2025-03-04T20:55:03Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/43956

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File DataFormats/Common/interface/WrapperBase.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/43956/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/43956/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2025-03-04T20:58:53Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47504/43957

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/Wrapper.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File DataFormats/Common/interface/WrapperBase.h modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericProduct and GenericCloner test module #46821

cmsbuild · 2025-03-04T20:59:13Z

A new Pull Request was created by @fwyzard for master.

It involves the following packages:

DataFormats/Common (core)
DataFormats/Portable (heterogeneous)
FWCore/Framework (core)
FWCore/Integration (core)
FWCore/TestModules (core)

@Dr15Jones, @cmsbuild, @fwyzard, @makortel, @smuzaffar can you please review it and eventually sign? Thanks.
@makortel, @missirol, @mmusich, @rovere, @wddgit this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

fwyzard · 2025-03-04T20:59:59Z

FWCore/Framework/interface/WrapperBaseHandle.h

Most of this could actually go directly under DataFormats/Common, only the specialisation of the Event methods need to go in FWCore/Framework.
Let me know if I should split and rearrange this.

fwyzard · 2025-03-04T21:00:30Z

FWCore/Framework/interface/WrapperBaseOrphanHandle.h

Most of this could actually go directly under DataFormats/Common, only the specialisation of the Event methods need to go in FWCore/Framework.
Let me know if I should split and rearrange this.

fwyzard · 2025-03-04T21:02:24Z

DataFormats/Portable/interface/PortableCollection.h

+      return {object->metadata().size()};
+    }
+
+    static std::vector<std::pair<void*, std::size_t>> regions(PortableDeviceCollection<T, TDev>& object) {


To make the specialisation for PortableDeviceCollection work, some kind of pass-through specialisation for edm::DeviceProduct is still needed.

makortel · 2025-09-02T23:01:21Z

About the directory structure, we should create a new one parallel to DataFormats, e.g. DataFormatsSerialisation ?

Right. Following the structure of the CondDB, i.e. CondCore package, maybe package called TrivialSerialization that would contain a subpackage for the infrastructure, and subpackages that mirror the DataFormats subpackages that need the new plugins (as in your example).

fwyzard · 2025-09-03T06:17:05Z

Hi Matti, thanks for the suggestions. I suspect the API will need to remain a bit more complicated to support efficient transfers across processes, but we'll try to simplify it as much as possible. .A

cmsbuild · 2025-09-10T06:55:36Z

Milestone for this pull request has been moved to CMSSW_16_0_X. Please open a backport if it should also go in to CMSSW_15_1_X.

…gin::PluginFactory

fwyzard · 2025-10-28T17:21:02Z

type ngt

fwyzard · 2025-11-23T19:04:15Z

The framework part of this PR (the first two commits) has been rebased and moved to #49449.

The second part has been reimplemented with the plugin-based serialisation approach, and will be moved to a different PR.

fwyzard · 2025-11-25T22:18:43Z

The plugin-based serialisation mechanism is implemented in #49475.

fwyzard added 3 commits March 4, 2025 13:49

Specialize Handle and OrphanHandle for WrapperBase

acf9d6a

Implement integration tests for WrapperBase specialisations

4ba7794

Implement edmtest::GenericCloner

31d3ae7

This EDProducer will clone all the event products declared by its configuration, using their ROOT dictionaries.

cmsbuild added this to the CMSSW_15_1_X milestone Mar 4, 2025

cmsbuild added core-pending pending-signatures tests-pending orp-pending code-checks-pending heterogeneous-pending labels Mar 4, 2025

cmsbuild added tests-started and removed tests-pending labels Mar 4, 2025

cmsbuild added code-checks-rejected and removed code-checks-pending labels Mar 4, 2025

fwyzard force-pushed the implement_generic_product_tools_v2 branch from c559d9f to e5278a6 Compare March 4, 2025 20:56

cmsbuild added tests-pending code-checks-pending and removed tests-started code-checks-rejected labels Mar 4, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 4, 2025

fwyzard commented Mar 4, 2025

View reviewed changes

cmsbuild modified the milestones: CMSSW_15_1_X, CMSSW_16_0_X Sep 10, 2025

cmsbuild mentioned this pull request Sep 12, 2025

Moved Pool based tests from FWCore to IOPool #48908

Merged

This was referenced Oct 6, 2025

Added AoSWrapper and transposition function #49072

Draft

Switch from fmt external to std standard for formatting library #49086

Merged

ghyls added a commit to ghyls/cmssw that referenced this pull request Oct 13, 2025

Implemented a version of cms-sw#47504's GenericCloner based on edmplu…

c7c5bc5

…gin::PluginFactory

ghyls added a commit to ghyls/cmssw that referenced this pull request Oct 13, 2025

Implemented a version of cms-sw#47504's GenericCloner based on edmplu…

cd9fb0c

…gin::PluginFactory

ghyls added a commit to ghyls/cmssw that referenced this pull request Oct 13, 2025

Implemented a version of cms-sw#47504's GenericCloner based on edmplu…

054ccff

…gin::PluginFactory

ghyls added a commit to ghyls/cmssw that referenced this pull request Oct 13, 2025

Implemented a version of cms-sw#47504's GenericCloner based on edmplu…

57cdca9

…gin::PluginFactory

ghyls added a commit to ghyls/cmssw that referenced this pull request Oct 13, 2025

Implemented a version of cms-sw#47504's GenericCloner based on edmplu…

c9dc52f

…gin::PluginFactory

ghyls mentioned this pull request Oct 13, 2025

Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152

Closed

cmsbuild mentioned this pull request Oct 16, 2025

Add PortableHostCollection constructor for code that does not use alpaka #49177

Merged

This was referenced Oct 28, 2025

Enabling usage of filters in MPI offloading #49233

Closed

Introduce ngt::AnyBuffer #49235

Closed

cmsbuild added the ngt label Oct 28, 2025

cmsbuild mentioned this pull request Oct 28, 2025

Add a const overload of bareProduct() and a markAsPresent() method to Wrapper.h #49236

Closed

cmsbuild mentioned this pull request Nov 18, 2025

Add bareProduct() const method #49415

Merged

fwyzard mentioned this pull request Nov 23, 2025

Specialise Handle and OrphanHandle for WrapperBase #49449

Merged

fwyzard closed this Nov 23, 2025

fwyzard deleted the implement_generic_product_tools_v2 branch November 25, 2025 22:18

Implement a GenericCloner test module #47504

Implement a GenericCloner test module #47504

Uh oh!

Conversation

fwyzard commented Mar 4, 2025

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

cmsbuild commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fwyzard commented Mar 4, 2025

Uh oh!

fwyzard commented Mar 4, 2025

Uh oh!

cmsbuild commented Mar 4, 2025

Uh oh!

cmsbuild commented Mar 4, 2025

Uh oh!

cmsbuild commented Mar 4, 2025

Uh oh!

fwyzard Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fwyzard Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

fwyzard Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

makortel commented Sep 2, 2025

Uh oh!

fwyzard commented Sep 3, 2025 via email

Uh oh!

cmsbuild commented Sep 10, 2025

Uh oh!

fwyzard commented Oct 28, 2025

Uh oh!

fwyzard commented Nov 23, 2025

Uh oh!

fwyzard commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Implement a `GenericCloner` test module #47504

Implement a `GenericCloner` test module #47504

cmsbuild commented Mar 4, 2025 •

edited

Loading

fwyzard Mar 4, 2025 •

edited

Loading