Introduce `ngt::AnyBuffer` by fwyzard · Pull Request #49235 · cms-sw/cmssw

fwyzard · 2025-10-28T17:14:42Z

PR description:

ngt::AnyBuffer behaves like std::any, with two differences: it can only be used with trivially copyable types, and provides access to the underlying memory buffer to allow memcpy'ing its content.

fwyzard · 2025-10-28T17:14:57Z

please test

cmsbuild · 2025-10-28T17:15:06Z

cms-bot internal usage

cmsbuild · 2025-10-28T17:16:37Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49235/46582

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/AnyBuffer.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement a GenericCloner test module #47504, Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152, Enabling usage of filters in MPI offloading #49233

cmsbuild · 2025-10-28T17:17:06Z

A new Pull Request was created by @fwyzard for master.

It involves the following packages:

DataFormats/Common (core)

@Dr15Jones, @makortel, @smuzaffar can you please review it and eventually sign? Thanks.
@makortel, @missirol, @mmusich, @rovere, @wddgit this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

fwyzard · 2025-10-28T17:19:38Z

@makortel @Dr15Jones the use case for edm::AnyBuffer is for sending small, trivially copyable objects over a serialisation/deserialisation chain (e.g. like MPI_Send/MPI_Recv) whenthe sender and receiver know the concrete type, but the code doing the send and receive does not.

Examples are the various variations of adding MPI support to CMSSW in #32632 or #49233, and the stand-alone tests in #47504 or #49152.

fwyzard · 2025-10-28T17:20:33Z

type ngt

cmsbuild · 2025-10-28T19:00:28Z

+1

Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-60ef34/48869/summary.html
COMMIT: 846c960
CMSSW: CMSSW_16_0_X_2025-10-28-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49235/48869/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3939953
DQMHistoTests: Total failures: 32
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3939901
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

makortel · 2025-10-28T21:57:18Z

Examples are the various variations of adding MPI support to CMSSW in #32632 or #49233, and the stand-alone tests in #47504 or #49152.

While this PR looks seemingly straightforward, I want to understand the wider context (again) first before commenting (especially #49152). It will probably take me something along a week or two to have a good moment to start digesting them.

fwyzard · 2025-10-28T22:16:47Z

OK, understood, though we would appreciate not having to rebuild about 1000 packages every time 🤷🏻‍♂️

mandrenguyen · 2025-10-29T06:02:10Z

+1

fwyzard · 2025-11-23T18:24:32Z

Will be moved to ngt::AnyBuffer under HeterogeneousCore/SerialisationCore.

ngt::AnyBuffer behaves like std::any, with two differences: it can only be used with trivially copyable types, and provides access to the underlying memory buffer to allow memcpy'ing its content.

cmsbuild · 2025-11-27T07:16:00Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49235/46993

There are other open Pull requests which might conflict with changes you have proposed:
- File HeterogeneousCore/SerialisationCore/README.md modified in PR(s): Trivial serialisation plugin mechanism #49475
- File HeterogeneousCore/SerialisationCore/interface/AnyBuffer.h modified in PR(s): Trivial serialisation plugin mechanism #49475

cmsbuild · 2025-11-27T07:16:25Z

Pull request #49235 was updated. @cmsbuild can you please check and sign again.

fwyzard · 2025-11-27T07:24:56Z

@cms-sw/core-l2 we are still thinking about the new package name, but let me know if you would prefer to have this under FWCore and in the edm namespace.

fwyzard · 2025-11-27T07:25:03Z

please test

cmsbuild · 2025-11-27T09:23:52Z

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-60ef34/49710/summary.html
COMMIT: b0f1776
CMSSW: CMSSW_16_0_X_2025-11-26-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49235/49710/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

Comparison Summary

Summary:

You potentially added 2 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 4 differences found in the comparisons
Reco comparison had 4 failed jobs
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4269233
DQMHistoTests: Total failures: 67
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4269146
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 227 log files, 198 edm output root files, 53 DQM output files
TriggerResults: no differences found

fwyzard · 2025-12-16T22:52:29Z

Included in #49475.

cmsbuild added this to the CMSSW_16_0_X milestone Oct 28, 2025

cmsbuild added core-pending pending-signatures orp-pending tests-started code-checks-pending labels Oct 28, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 28, 2025

cmsbuild added the ngt label Oct 28, 2025

cmsbuild added tests-approved and removed tests-started labels Oct 28, 2025

cmsbuild mentioned this pull request Oct 28, 2025

Implement a GenericCloner test module based on an edmplugin::PluginFactory #49152

Closed

cmsbuild added orp-approved and removed orp-pending labels Oct 29, 2025

cmsbuild mentioned this pull request Nov 5, 2025

Enabling usage of filters in MPI offloading #49233

Closed

fwyzard closed this Nov 23, 2025

fwyzard deleted the AnyBuffer branch November 23, 2025 18:24

Introduce ngt::AnyBuffer

b0f1776

ngt::AnyBuffer behaves like std::any, with two differences: it can only be used with trivially copyable types, and provides access to the underlying memory buffer to allow memcpy'ing its content.

fwyzard restored the AnyBuffer branch November 27, 2025 07:14

fwyzard reopened this Nov 27, 2025

fwyzard force-pushed the AnyBuffer branch from 846c960 to b0f1776 Compare November 27, 2025 07:14

fwyzard changed the title ~~Introduce edm::AnyBuffer~~ Introduce ngt::AnyBuffer Nov 27, 2025

cmsbuild added tests-pending new-package-pending code-checks-pending and removed core-pending tests-approved code-checks-approved labels Nov 27, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 27, 2025

cmsbuild added tests-started and removed tests-pending labels Nov 27, 2025

cmsbuild added tests-approved and removed tests-started labels Nov 27, 2025

This was referenced Nov 28, 2025

Trivial serialisation plugin mechanism #49475

Merged

Extend CMSSW to a distributed application over MPI #32632

Merged

fwyzard closed this Dec 16, 2025

fwyzard deleted the AnyBuffer branch December 16, 2025 22:52

fwyzard changed the title ~~Introduce ngt::AnyBuffer~~ Introduce ngt::AnyBuffer Jan 30, 2026

Conversation

fwyzard commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

Uh oh!

fwyzard commented Oct 28, 2025

Uh oh!

cmsbuild commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Oct 28, 2025

Uh oh!

cmsbuild commented Oct 28, 2025

Uh oh!

fwyzard commented Oct 28, 2025

Uh oh!

fwyzard commented Oct 28, 2025

Uh oh!

cmsbuild commented Oct 28, 2025

Comparison Summary

Uh oh!

makortel commented Oct 28, 2025

Uh oh!

fwyzard commented Oct 28, 2025

Uh oh!

mandrenguyen commented Oct 29, 2025

Uh oh!

fwyzard commented Nov 23, 2025

Uh oh!

cmsbuild commented Nov 27, 2025

Uh oh!

cmsbuild commented Nov 27, 2025

Uh oh!

fwyzard commented Nov 27, 2025

Uh oh!

fwyzard commented Nov 27, 2025

Uh oh!

cmsbuild commented Nov 27, 2025

Comparison Summary

Uh oh!

fwyzard commented Dec 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fwyzard commented Oct 28, 2025 •

edited

Loading

cmsbuild commented Oct 28, 2025 •

edited

Loading