Parallel post-processing in the Python wrapper #571
Description
At the moment --threads/--procs --merge-output-files is not really usable with many threads/processes because the reshaping is done sequentially and therefore takes a lot of time (in my tests, it takes as long as the full simulation).
The solution (not so much work I believe) would be to run the reshaping in parallel. I would proceed as follows:
- swap the
reboost.build_hit()call with a simple explicit iterator (this is also needed to resolve Remove pyg4ometry dependency #539) - check if we can use as-is the existing multi-processing capabilities of
LH5Iterator, otherwise we just implement some simple parallelism over files - in general, it's not obvious if the python threads will be performant. We need to check