Skip to content

Conversation

@iguinn
Copy link
Collaborator

@iguinn iguinn commented May 29, 2025

Depends on legend-exp/legend-pydataobj#154

Added multi-processing to build_dsp. Build_dsp was re-organized to take advantage of LH5Iterator.map. This has enabled the use of multiprocessing. The parent process:

  1. Handles input arguments and figures out which files/groups/Tables to use, and reads in any config files
  2. Uses LH5Iterator.map to create a task that will execute a processing chain for a single channel and return the full Table.
  3. Loop through the results of map and write the results to disk. The results are returned as an asyncronous iterator, so this will be concurrent with the running of the processing chains

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant