Remove PortableHostMultiCollection and PortableDeviceMultiCollection #49734

Electricks94 · 2026-01-07T09:12:33Z

This PR removes PortableHostMultiCollection and PortableDeviceMultiCollection and replaces them with SoABlocks (see #48629).

This has two major advantages:

Simplifications of dictionaries because variadic templates are not necessary anymore
Code simplifications because BlocksView holds views to all sublayouts

In a future PRs the code can be simplified even more because with kernels that use a BlocksView as input can work on the entire data. Hence, it is not necessarry anymore to have every sub-view as input. In this PR, however, I tried to make as few changes as possible to not blow up the PR too much.

@felicepantaleo @leobeltra fyi

cmsbuild · 2026-01-07T09:13:08Z

cms-bot internal usage

Electricks94 · 2026-01-07T09:14:30Z

type ngt

cmsbuild · 2026-01-07T09:15:34Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49734/47318

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Portable/README.md modified in PR(s): Improve PortableHost{Collection,Object} dictionary declarations #48824, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Enabling usage of filters in MPI offloading #49233
- File DataFormats/Portable/interface/PortableCollectionCommon.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableDeviceCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072
- File DataFormats/Portable/interface/PortableHostCollection.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Added AoSWrapper and transposition function #49072, Enabling usage of filters in MPI offloading #49233
- File DataFormats/Portable/scripts/portableHostCollectionHints modified in PR(s): Improve PortableHost{Collection,Object} dictionary declarations #48824
- File DataFormats/PortableTestObjects/src/classes_def.xml modified in PR(s): Improve PortableHost{Collection,Object} dictionary declarations #48824
- File DataFormats/SoATemplate/interface/SoACommon.h modified in PR(s): Various fixes for heterogeneous utilities #47605, Added AoSWrapper and transposition function #49072
- File DataFormats/TrackingRecHitSoA/interface/TrackingRecHitsHost.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File RecoLocalTracker/Phase2TrackerRecHits/plugins/alpaka/Phase2OTRecHitsSoAConverter.cc modified in PR(s): [NGT] Extension of CA Pixel Tracking to Phase-2 Outer Tracker Disks (TID) #49346
- File RecoTracker/LSTCore/src/classes_def.xml modified in PR(s): Improve PortableHost{Collection,Object} dictionary declarations #48824
- File RecoTracker/PixelSeeding/plugins/alpaka/CAHitNtuplet.cc modified in PR(s): Use TkPixelCPERecord for PixelCPEFastParams* #46852, [NGT] Extension of CA Pixel Tracking to Phase-2 Outer Tracker Disks (TID) #49346
- File RecoTracker/PixelTrackFitting/plugins/PixelTrackProducerFromSoAAlpaka.cc modified in PR(s): [NGT] Extension of CA Pixel Tracking to Phase-2 Outer Tracker Disks (TID) #49346

cmsbuild · 2026-01-07T09:15:57Z

A new Pull Request was created by @Electricks94 for master.

It involves the following packages:

DQM/SiPixelHeterogeneous (dqm)
DataFormats/HGCalDigi (simulation)
DataFormats/Portable (heterogeneous)
DataFormats/PortableTestObjects (heterogeneous)
DataFormats/SoATemplate (heterogeneous)
DataFormats/TrackSoA (heterogeneous, reconstruction)
DataFormats/TrackingRecHitSoA (heterogeneous, reconstruction)
DataFormats/VertexSoA (heterogeneous, reconstruction)
HeterogeneousCore/AlpakaTest (heterogeneous)
RecoLocalTracker/Phase2TrackerRecHits (reconstruction)
RecoLocalTracker/SiPixelRecHits (reconstruction)
RecoTauTag/HLTProducers (hlt)
RecoTracker/LST (reconstruction)
RecoTracker/LSTCore (reconstruction)
RecoTracker/PixelSeeding (reconstruction)
RecoTracker/PixelTrackFitting (reconstruction)
RecoVertex/PixelVertexFinding (reconstruction)

@Martin-Grunewald, @Moanwar, @civanch, @cmsbuild, @ctarricone, @fwyzard, @gabrielmscampos, @jfernan2, @kpedro88, @makortel, @mandrenguyen, @mdhildreth, @mmusich, @nothingface0, @rseidita, @srimanob can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @IzaakWN, @VinInn, @VourMa, @azotz, @dgulhan, @dkotlins, @elusian, @fabiocos, @felicepantaleo, @ferencek, @fioriNTU, @gpetruc, @idebruyn, @jandrea, @makortel, @martinamalberti, @mbluj, @missirol, @mmasciov, @mmusich, @mroguljic, @mtosi, @pfs, @rovere, @threus, @tsusa this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2026-01-07T09:50:23Z

enable gpu

mmusich · 2026-01-07T09:50:37Z

test parameters:

workflows = ph2_hlt

fwyzard · 2026-01-07T09:52:28Z

DataFormats/Portable/README.md

-```
-The layouts should not be added as parameters for the device collection. Those script can be use equally with the
-single layout collections or multi layout collections.
+Multi layout collections have been removed and replaced by `SoABlocks`. See the `DataFormats/SoATemplate`


Seems broken off ?

fwyzard · 2026-01-07T09:55:12Z

DataFormats/Portable/scripts/portableDeviceCollectionHints


 collectionName = sys.argv[1]

+# TODO: do we need to change something here becuase of the removal of PortableMultiCollection?


I don't think this ever supported PortableMultiCollections.
Do we need a special syntax for the SoA blocks ?

For the SoA blocks no special syntax is needed. It is equivalent to a normal PortableCollection. I adapted the file to be consistent with #48824. Which means first Collection declaration. Then Collection::Layout followed by the SoALayout that is aliased by Collection::Layout. Lastly the definition with edm::Wrapper

Then, can you remove the TODO ?

fwyzard · 2026-01-07T09:56:55Z

DataFormats/Portable/scripts/portableHostCollectionHints


 import sys
-
+# TODO: do we need to change something here becuase of the removal of PortableMultiCollection?


I think you should remove the part if len(layouts) > 1:

fwyzard · 2026-01-07T09:59:54Z

DataFormats/PortableTestObjects/interface/TestSoA.h


  using TestSoA3 = TestSoALayout3<>;

+  GENERATE_SOA_BLOCKS(SoABlocks2, SOA_BLOCK(firstLayout, TestSoALayout), SOA_BLOCK(secondLayout, TestSoALayout2))


maybe just

Suggested change

GENERATE_SOA_BLOCKS(SoABlocks2, SOA_BLOCK(firstLayout, TestSoALayout), SOA_BLOCK(secondLayout, TestSoALayout2))

GENERATE_SOA_BLOCKS(SoABlocks2, SOA_BLOCK(first, TestSoALayout), SOA_BLOCK(second, TestSoALayout2))

?

Same below.

fwyzard · 2026-01-07T10:02:42Z

DataFormats/Portable/interface/PortableDeviceCollection.h

+  // Either int32_t for normal layouts or std::array<int32_t, N> for SoABlocks layouts
+  auto size() const {
+    return layout_.metadata().size();
+    ;


Suggested change

;

fwyzard · 2026-01-07T10:08:15Z

DataFormats/Portable/interface/PortableHostCollection.h


 // generic SoA-based product in host memory
+// TODO: Should we define a size alias for the number of elements? Also should be switch to unsigned and/or 64 bit?
+// Exeeding the size of int32_t fails without clear error messages.


This needs to be an int32_t (actually a plain int) because of how ROOT stores the columns 🤷🏻‍♂️ .

I understand. How about allowing any integral type for the constructor of the PortableCollections and then implementing a check if the value can be savely casted to int? If it is save to cast we would do so and pass it on to the Layout, otherwise we throw an error.

I just realised that the PortableCollections have already a constructor that takes std::integral as input type for the elements. So the idea would be just to align all other constructors as well and add a check for the cast.

fwyzard · 2026-01-07T10:08:35Z

DataFormats/Portable/interface/PortableHostCollection.h

    alpaka::memset(std::forward<TQueue>(queue), *buffer_, 0x00);
  }

-  // part of the ROOT read streamer


Why remove the comment ?

fwyzard · 2026-01-07T10:09:38Z

DataFormats/Portable/interface/PortableHostCollection.h

+  // Either int32_t for normal layouts or std::array<int32_t, N> for SoABlocks layouts
+  auto size() const {
+    return layout_.metadata().size();
+    ;


Suggested change

;

jfernan2 · 2026-01-20T16:20:02Z

+1

kpedro88 · 2026-01-20T16:48:55Z

+simulation

gabrielmscampos · 2026-01-21T09:04:43Z

+dqm

cmsbuild · 2026-01-21T09:05:11Z

This pull request is fully signed and it will be integrated in one of the next master IBs (test failures were overridden). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @ftenchini, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2026-01-21T09:24:11Z

+1

makortel · 2026-01-21T21:35:12Z

This PR caused 4 DQM integration test to fail, see #49892

fwyzard · 2026-01-21T21:59:19Z

I guess it's to be expected if the DQM tests reuse old streamer files as input ?

As far as I know there is no guarantee that streamer files are readable with any release different from what was used to write them.

makortel · 2026-01-21T22:35:52Z

As far as I know there is no guarantee that streamer files are readable with any release different from what was used to write them.

Correct.

…tions_160X [Backport to 16_0_X (#49734)] Remove PortableHostMultiCollection and PortableDeviceMultiCollection

cmsbuild added this to the CMSSW_16_1_X milestone Jan 7, 2026

cmsbuild added reconstruction-pending simulation-pending dqm-pending hlt-pending pending-signatures tests-pending orp-pending code-checks-pending heterogeneous-pending tracking trk changes-dataformats labels Jan 7, 2026

cmsbuild added the ngt label Jan 7, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 7, 2026

fwyzard reviewed Jan 7, 2026

View reviewed changes

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Jan 20, 2026

cmsbuild added simulation-approved and removed simulation-pending labels Jan 20, 2026

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Jan 21, 2026

cmsbuild added orp-approved and removed orp-pending labels Jan 21, 2026

cmsbuild merged commit c158e79 into cms-sw:master Jan 21, 2026
15 of 21 checks passed

Electricks94 mentioned this pull request Jan 21, 2026

[Backport to 16_0_X (#49734)] Remove PortableHostMultiCollection and PortableDeviceMultiCollection #49882

Merged

cmsbuild mentioned this pull request Jan 21, 2026

Run the ONNX test on GPU only if CUDA is available #49883

Merged

makortel mentioned this pull request Jan 21, 2026

DictionaryNotFound exception in DQM/Integration unit tests #49892

Open

smuzaffar mentioned this pull request Jan 21, 2026

RecoTrackerLSTCore-standalone-compilation test failed #49893

Open

slava77 mentioned this pull request Jan 21, 2026

LST standalone fix after interface change in Portable*MultiCollection #49895

Merged

This was referenced Jan 22, 2026

update the streamer files for the DQMGPUvsCPU stream cms-data/DQM-Integration#17

Merged

update run number used for streamDQMGPUvsCPU based client tests #49898

Merged

cmsbuild mentioned this pull request Jan 23, 2026

Merging CMSSW_16_1 into CMSSW_16_1_EVOLUTION_X #49910

Merged

Electricks94 mentioned this pull request Jan 28, 2026

[Backport to 16_0_X (#49860)] Refactor of PortableCollections #49953

Merged

cmsbuild added a commit that referenced this pull request Jan 28, 2026

Merge pull request #49882 from Electricks94/RemovePortableMultiCollec…

81e8d53

…tions_160X [Backport to 16_0_X (#49734)] Remove PortableHostMultiCollection and PortableDeviceMultiCollection

ariostas mentioned this pull request Feb 2, 2026

Fix typo in LST causing segfault #50001

Open


		collectionName = sys.argv[1]

		# TODO: do we need to change something here becuase of the removal of PortableMultiCollection?


		import sys

		# TODO: do we need to change something here becuase of the removal of PortableMultiCollection?


		using TestSoA3 = TestSoALayout3<>;

		GENERATE_SOA_BLOCKS(SoABlocks2, SOA_BLOCK(firstLayout, TestSoALayout), SOA_BLOCK(secondLayout, TestSoALayout2))

Remove PortableHostMultiCollection and PortableDeviceMultiCollection #49734

Remove PortableHostMultiCollection and PortableDeviceMultiCollection #49734

Uh oh!

Conversation

Electricks94 commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Electricks94 commented Jan 7, 2026

Uh oh!

cmsbuild commented Jan 7, 2026

Uh oh!

cmsbuild commented Jan 7, 2026

Uh oh!

mmusich commented Jan 7, 2026

Uh oh!

mmusich commented Jan 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfernan2 commented Jan 20, 2026

Uh oh!

kpedro88 commented Jan 20, 2026

Uh oh!

gabrielmscampos commented Jan 21, 2026

Uh oh!

cmsbuild commented Jan 21, 2026

Uh oh!

mandrenguyen commented Jan 21, 2026

Uh oh!

Uh oh!

makortel commented Jan 21, 2026

Uh oh!

fwyzard commented Jan 21, 2026

Uh oh!

makortel commented Jan 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Electricks94 commented Jan 7, 2026 •

edited

Loading

cmsbuild commented Jan 7, 2026 •

edited

Loading