Add wd1 processing #41

jwaiton · 2025-04-22T09:19:27Z

This PR introduces the functionality of processing into MULE, with single channel decoding from waveDump 1 .dat files now being possible. Addresses #11 which will be resolved when this is complete.

The data is stored in h5 files with a storage path and name provided by the user. The h5 format is similar but does not match the wavedump 2 formatting, hence the need for malleable reader and writers (as introduced in #40). This will be resolved in future PRs to be equivalent across both.

This PR rests on top of #40, so should be merged after it.

bpalmeiro

First round of comments, have fun!

bpalmeiro · 2026-01-22T17:05:19Z

packs/core/core_utils.py

+# THIS SHOULD BE MOVED ELSEWHERE
+class MalformedHeaderError(Exception):
+    '''
+    Header created for when two headers don't match up consecutively.


Guess you mean exception?

bpalmeiro · 2026-01-22T17:05:31Z

packs/core/io.py

 import pandas as pd
+import numpy as np


bpalmeiro · 2026-01-22T17:15:16Z

packs/proc/processing_utils.py

+    header = np.fromfile(file_object, dtype = 'i', count = 6)
+


I guess these are fixed for WD2 right?

Its fixed for WD1, WD2 uses an adaptively sized header, but since each file in Wavedump1 is a channel, this issue doesn't occur.

bpalmeiro · 2026-01-22T17:17:23Z

packs/proc/processing_utils.py

+    sanity_header = header.copy()
+
+    # continue only if data exists


Why copy it before knowing if it has anything?

because we rewrite the header variable in the next steps to compare to this 'initial sanity check' header. If it is malformed, an error is returned.

This code could be restructured to check if it has none before copying, but copying these headers once isn't particularly expensive.

bpalmeiro · 2026-01-22T17:30:32Z

packs/proc/processing_utils.py

+        header = np.fromfile(file_object, dtype = 'i', count = 6)
+
+        # check if header has correct number of elements and correct information ONCE.
+        if sanity_header is not None:


This comparison should be made at the beginning and not compared all the time; this object is unchanged, right?

Also, given you already did the while with this in the 1st iteration technically you've checked this already

It is only made at the beginning in the first iteration, if it passes the checks sanity_header is set to None after it is checked once and as such this if statement is never checked again.

bpalmeiro · 2026-01-23T09:52:49Z

packs/proc/processing_utils.py

+        save_path    (str)   :  Path to saved file
+        sample_size  (int)   :  Size of each sample in an event (2 ns in the case of V1730B digitiser)
+        overwrite    (bool)  :  Boolean for overwriting pre-existing files
+        counts       (int)   :  The number of events per chunks. -1 implies no chunking of data.


counts not used, print mod used and not reported

This should be altered to print_mod, will change

bpalmeiro · 2026-01-23T09:53:07Z

packs/tests/data/configs/process_WD1.conf

+[optional]
+
+overwrite        = True
+counts           = -1  


counts not used

any reason for it?

whoopsies, it should have been replaced with print_mod as the lazy processing no longer requires chunking. I'll fix this

bpalmeiro · 2026-01-23T09:54:49Z

packs/tests/data/configs/process_WD1_1channel.conf

+file_path = '/home/casper/Documents/MULE/packs/tests/data/one_channel_WD1.dat'
+save_path = '/home/casper/Documents/MULE/packs/tests/data/one_channel_WD1_tmp.h5'


shall these paths be more generic? :)

I could try and tie them into the provided environment variables for the MULE directory, but these are just sample configs. They're not meant to work out of the box, but provide a template to work upon.

bpalmeiro · 2026-01-23T09:57:43Z

packs/tests/processing_test.py

+    assert [x for x in reader(save_path, 'RAW', 'rwf')] == [x for x in reader(comparison_path, 'RAW', 'rwf')]
+
+
+def test_lazy_loading_malformed_data(MULE_dir):


You may can also add the sanity_header being None if you keep that part of the code

But the None case and the reverse are tested in the process of WD1 processing, but I can create explicit tests.

this comment comes from the aligment confusion, disregard

But i can transform it to check not only the specific values but also the "len(header) == 6", right?

bpalmeiro · 2026-01-23T09:59:56Z

packs/proc/proc.py

            if conf_dict['wavedump_edition'] == 2:
                process_bin_WD2(**arg_dict)
+            elif conf_dict['wavedump_edition'] == 1:
+                process_bin_WD1(**arg_dict)


Suggested change

if conf_dict['wavedump_edition'] == 2:

process_bin_WD2(**arg_dict)

elif conf_dict['wavedump_edition'] == 1:

process_bin_WD1(**arg_dict)

if conf_dict['wavedump_edition'] == 2:

process_bin_WD2(**arg_dict)

elif conf_dict['wavedump_edition'] == 1:

process_bin_WD1(**arg_dict)

also test the new case? :)

jwaiton mentioned this pull request Apr 22, 2025

Develop WD1 decoding #16

Closed

3 tasks

bpalmeiro self-assigned this Jan 16, 2026

jwaiton added 6 commits January 22, 2026 17:07

add WD1 rwf type

e28d086

add MalformedHeaderError

726691e

add lazy WD1 processing

a19f3dd

include test for process_event_lazy_WD1

0f5b647

add WD1 processing

80f813d

add tests

fee38cc

jwaiton force-pushed the add-WD1-processing branch from 27d632d to fee38cc Compare January 22, 2026 17:07

bpalmeiro reviewed Jan 23, 2026

View reviewed changes

		file_path = '/home/casper/Documents/MULE/packs/tests/data/one_channel_WD1.dat'
		save_path = '/home/casper/Documents/MULE/packs/tests/data/one_channel_WD1_tmp.h5'

		assert [x for x in reader(save_path, 'RAW', 'rwf')] == [x for x in reader(comparison_path, 'RAW', 'rwf')]


		def test_lazy_loading_malformed_data(MULE_dir):

Add wd1 processing #41

Are you sure you want to change the base?

Add wd1 processing #41

Uh oh!

Conversation

jwaiton commented Apr 22, 2025

Uh oh!

bpalmeiro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jwaiton Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jwaiton Jan 23, 2026 •

edited

Loading