OptoDAS IO

[dylanmikesell]

Hi, I am wondering if anyone is working to add support for OptoDAS HDF5 file format? I tried dascore.read() today on my file and discovered that the file format was not found. I have not dug into the read() function yet to see where it checks file types, but I have used the simpleDAS package in the past to read this type of data. Example of headers/metadata look like

-- Header info file: test.hdf5 --
	Experiment:            unknown
	File timestamp:        2020-09-16 06:59:59
	Type of data:          Phase rate per distance, unit: rad/m/s
	Sampling frequency:    625.00 Hz
	Data shape:            6250 samples x 20000 channels
	Gauge length:          8.2 m
	Sensitivities:         9.36e+06 rad/m/strain
	Regions of interest:   10000:89996:4
	63 channels requested:[10000 10016 10032 ... 10960 10976 10992]
	63 channels found:    [10000 10016 10032 ... 10960 10976 10992]

If no one is working to add this to dascore.io, I will give it a try. Or have I simply missed something and this file format can be read using dascore?

[d-chambers]

Hi @dylanmikesell,

We don't currently support this format but, provided we can get a small sample data file, it should be trivial to add. Do you have one you can share? In the docs we have a section on contributing a new file format. You can take a look and see if it is something you want to tackle, otherwise I should have some time in a week or two I can work on it.

[d-chambers]

Hey @dylanmikesell, any updates on a test file?

[dylanmikesell]

Yes @d-chambers, I have a 22 MB file. How would you like me to get it to you?

[d-chambers]

Thanks, got it. I will work on it when I find a few minutes, probably next week

[d-chambers]

Hey @dylanmikesell, #347 implements support for the OptoDAS format. Can you take a few minutes to run some simple tests on your data? You should be able to install it with pip like this:

pip install git+https://github.com/dasdae/dascore.git@opto_das

Then just create a spool from your data directory (or a smaller subset) and run some simple sanity checks, such as these:

import dascore as dc

spool = dc.spool("path/to/my/data").update()

assert len(spool)
df = spool.get_contents()
assert len(df) == len(spool)

patch1 = spool[0]
assert patch1.shape

patch2 = spool[-1]
assert patch2.shape

Also let me know if you need any additional meta data from the file by checking the patch attrs (eg print(patch.attrs)). There is a lot there, but I think I got the essentials.

Once you are happy with it I will merge it into master.

Thanks!

[d-chambers]

closed by #347