Support for Treble DAS

[chauvetige]

Hi

I am getting my first data from Treble DAS (https://terra15.com.au/products/treble-das-interrogator/) and I wrote a function to load the data (format version 6 for terra15 files). If you like you can include it to your loader if it can benefit to other.

import h5py
import numpy as np
import xdas
from xdas import DataArray
from xdas.virtual import VirtualSource
from datetime import datetime

def read_Terra15(fname):
  with h5py.File(fname, "r") as file:
    ti=np.datetime64(datetime.utcfromtimestamp(file['data_product']['gps_time'][0])).astype('datetime64[ms]')
    tf=np.datetime64(datetime.utcfromtimestamp(file['data_product']['gps_time'][-1])).astype('datetime64[ms]')
    d0 = file.attrs['sensing_range_start']
    dx = file.attrs['dx']
    data = VirtualSource(file['data_product']['data'])
  nt, nd = data.shape
  t = {"tie_indices": [0, nt - 1], "tie_values": [ti, tf]}
  d = {"tie_indices": [0, nd - 1], "tie_values": [d0, d0+ (nd - 1) * dx]}
  return DataArray(data, {"time": t, "distance": d})

It works as expected with

fname="20240820_Test_velocity_UTC-YMD20240820-HMS200944.686_seq_00000000000.hdf5"
da = xdas.open_dataarray(fname, engine=read_Terra15)
da

<xdas.DataArray (time: 1288485, distance: 388)>
VirtualSource: 1.9GB (float32)
Coordinates:
  * time (time): 2024-08-20T20:09:44.627 to 2024-08-20T20:12:31.864
  * distance (distance): 600.320 to 2496.841

or with multiple files

folder="/mnt/data2/Planpincieux2024/7-OpticFiber/202408/UTC-YMD20240820-HMS200944.686/*"
da = xdas.open_mfdataarray(folder, engine=read_Terra15)
da

<xdas.DataArray (time: 279363276, distance: 388)>
VirtualStack: 403.8GB (float32)
Coordinates:
  * time (time): 2024-08-20T20:09:44.627 to 2024-08-21T06:14:04.116
  * distance (distance): 600.320 to 2496.841

I can provide you some file if you want to test it.

Best
Thomas

[atrabattoni]

Hello,

That’s great, nice contribution! Would you dare doing a PR? That would make you an official contributor :). I’ll let you know if I need some data samples thanks.

[atrabattoni]

I created a branch feature/terra15 that include your code. Could you try to use engine="terra15" and see if it works correctly with your dataset?
To install that branch you can either clone xdas and do a editable installation or :
pip install git+https://github.com/xdas-dev/xdas.git@feature/terra15

[chauvetige]

Hello

It seems that datetime.fromtimestamp(file[\"data_product\"][\"gps_time\"][0]), timezone.utc) is not working but datetime.fromtimestamp(file[\"data_product\"][\"gps_time\"][0]),tz=timezone.utc) is fine.

I push a request with the change. I put tz in option like this you can choose your time zone.

{
	"name": "TypeError",
	"message": "Invalid object for specifying NumPy datetime metadata",
	"stack": "---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[2], line 2
      1 fname=\"/mnt/data2/Planpincieux2024/7-OpticFiber/202408/UTC-YMD20240822-HMS112759.092/20240821_TT_RGH_TAP1_980_velocity_UTC-YMD20240822-HMS112759.092_seq_00000000000.hdf5\"
----> 2 da = xdas.open_dataarray(fname, engine='terra15')

File ~/Documents/Projets/Gricad/xdas/xdas/core/routines.py:351, in open_dataarray(fname, group, engine, **kwargs)
    348     from .. import io
    350     module = getattr(io, engine)
--> 351     return module.read(fname)
    352 else:
    353     raise ValueError(\"engine not recognized\")

File ~/Documents/Projets/Gricad/xdas/xdas/io/terra15.py:12, in read(fname)
     10 def read(fname):
     11     with h5py.File(fname, \"r\") as file:
---> 12         ti = np.datetime64(
     13             datetime.fromtimestamp(file[\"data_product\"][\"gps_time\"][0]), timezone.utc
     14         ).astype(\"datetime64[ms]\")
     15         tf = np.datetime64(
     16             datetime.fromtimestamp(file[\"data_product\"][\"gps_time\"][-1]), timezone.utc
     17         ).astype(\"datetime64[ms]\")
     18         d0 = file.attrs[\"sensing_range_start\"]

TypeError: Invalid object for specifying NumPy datetime metadata"
}

Best
Thomas

[atrabattoni]

Thanks, I still haven't found a simple way to include automatic tests for file formats reading.

I can't found your request. I gave you right permissions so you can commit the fix.

[atrabattoni]

Looks good enough for me.