Merge pull request #20 from xdas-dev/dev

Version 0.1.2

.github/workflows/code-coverage.yaml

@@ -0,0 +1,32 @@
+name: Code Coverage
+
+on: [push]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install '.[tests]'
+
+      - name: Run tests with coverage
+        run: coverage run --source=xdas -m pytest
+
+      - name: Generate coverage report
+        run: coverage xml
+
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v4
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}

README.md

@@ -10,6 +10,7 @@
 
 [![Documentation Status](https://readthedocs.org/projects/xdas/badge/?version=latest)](https://xdas.readthedocs.io/en/latest/?badge=latest)
 [![Tests Status](https://github.com/xdas-dev/xdas/actions/workflows/tests.yaml/badge.svg)](https://github.com/xdas-dev/xdas/actions/workflows/tests.yaml)
+[![codecov](https://codecov.io/gh/xdas-dev/xdas/graph/badge.svg?token=00MD52JRA3)](https://codecov.io/gh/xdas-dev/xdas)
 [![PyPI](https://img.shields.io/pypi/v/xdas)](https://pypi.org/project/xdas/)
 [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
 [![DOI](https://zenodo.org/badge/560867006.svg)](https://zenodo.org/badge/latestdoi/560867006)

docs/api/index.md

@@ -4,11 +4,12 @@
 :maxdepth: 1
 
 xdas
+atoms
+io
 fft
-signal
-processing
 parallel
+processing
+signal
 synthetics
 virtual
-atoms
 ```

docs/api/io.md

@@ -0,0 +1,27 @@
+```{eval-rst}
+.. currentmodule:: xdas.io
+```
+
+# xdas.io
+
+```{eval-rst}
+.. autosummary::
+   :toctree: ../_autosummary
+
+    get_free_port
+```
+
+```{eval-rst}
+.. currentmodule:: xdas.io.asn
+```
+
+
+## ASN
+
+```{eval-rst}
+.. autosummary::
+   :toctree: ../_autosummary
+
+   ZMQPublisher
+   ZMQSubscriber
+```

docs/api/processing.md

@@ -12,4 +12,6 @@
    DataArrayLoader
    RealTimeLoader
    DataArrayWriter
+   ZMQPublisher
+   ZMQSubscriber
 ```

docs/conf.py

@@ -11,7 +11,7 @@
 author = "Alister Trabattoni"
 
 # The full version, including alpha/beta/rc tags
-release = "0.1"
+release = "0.1.2"
 
 
 # -- General configuration ---------------------------------------------------

docs/release-notes.md

@@ -1,5 +1,11 @@
 # Release notes
 
+## 0.1.2
+
+- Add ZeroMQ streaming capabilities (@atrabattoni)
+- Add support of Terra15 format (@chauvetige).
+- Fix Febus engine (@ClaudioStrumia)
+
 ## 0.1.1
 
 - Add support for `hdf5plugin` compression schemes.

docs/user-guide/convert-displacement.md

@@ -32,7 +32,7 @@ import xdas.signal as xp
 
 strain = xp.integrate(strain_rate, dim="time")
 deformation = xp.integrate(strain, dim="distance")
-displacement = xp.sliding_mean_removal(deformation, wlen=2000.0)
+displacement = xp.sliding_mean_removal(deformation, wlen=2000.0, dim="distance")
 displacement.plot(yincrease=False, vmin=-0.5, vmax=0.5);
 ```
 

docs/user-guide/data-formats.md

@@ -20,14 +20,20 @@ os.chdir("../_data")
 
 ## Implemented file formats
 
-The formats that are currently implemented are: ASN, FEBUS, OPTASENSE and SINTELA. To read them you have to specifiy which one you want in the `engine` argument in {py:func}`xdas.open_dataarray` for a single file or {py:func}`xdas.open_mfdataarray` for multiple files:
+The formats that are currently implemented are: ASN, FEBUS, OPTASENSE, SINTELA and TERRA15. To read them you have to specifiy which one you want in the `engine` argument in {py:func}`xdas.open_dataarray` for a single file or {py:func}`xdas.open_mfdataarray` for multiple files:
 
 | DAS constructor   | `engine` argument |
 |:-----------------:|:-----------------:|
 | ASN               | `"asn"`           |
 | FEBUS             | `"febus"`         |
 | OPTASENSE         | `"optasense"`     |
 | SINTELA           | `"sintela"`       |
+| TERRA15           | `"terra15"`       |
+
+```{warning}
+Due to poor documentation of the various version of the Febus format, it is recommended to manually provide the required trimming and the position of the timestamps within each block. For example to trim 100 samples on both side of each block and to set the timestamp location at the center of the block for a block of 2000 samples:
+`xdas.open_dataarray("path.h5", engine="febus", overlaps=(100, 100), offset=1000)`
+```
 
 ## Extending *xdas* with your file format
 

docs/user-guide/index.md

@@ -10,4 +10,5 @@ interpolated-coordinates
 convert-displacement
 atoms
 processing
+streaming
 ```

docs/user-guide/streaming.md

@@ -0,0 +1,91 @@
+---
+file_format: mystnb
+kernelspec:
+  name: python3
+---
+
+# Streaming data 
+
+Xdas allows to stream data over any network using [ZeroMQ](https://zeromq.org). Xdas use the Publisher and Subscriber patterns meaning that on one node the data is published and that any number of subscribers can receive the data stream. 
+
+Streaming data with Xdas is done by simply dumping each chunk to NetCDF binaries and to send those as packets. This ensure that each packet is self described and that feature such as compression are available (which can be very helpful to minimize the used bandwidth). 
+
+Xdas implements the {py:class}`~xdas.processing.ZMQPublisher` and {py:class}`~xdas.processing.ZMQSubscriber`.Those object can respectively be used as a Writer and a Loader as described in the [](processing) section. Both are initialized by giving an network address. The publisher use the `submit` method to send packets while the subscriber is an infinite iterator that yields packets.
+
+In this section, we will mimic the use of several machine by using multithreading, where each thread is supposed to be a different machine. In real-life application, the publisher and subscriber are generally called in different machine or software.
+
+## Simple use case
+
+```{code-cell}
+import threading
+import time
+
+import xdas as xd
+from xdas.processing import ZMQPublisher, ZMQSubscriber
+```
+
+First we generate some data and split it into packets
+
+```{code-cell}
+da = xd.synthetics.dummy()
+packets = xd.split(da, 5)
+```
+
+We then publish the packets on machine 1.
+
+```{code-cell}
+address = f"tcp://localhost:{xd.io.get_free_port()}"
+publisher = ZMQPublisher(address)
+
+def publish():
+    for packet in packets:
+        publisher.submit(packet)
+        # give a chance to the subscriber to connect in time and to get the last packet
+        time.sleep(0.1)  
+
+machine1 = threading.Thread(target=publish)
+machine1.start()
+```
+
+Let's receive the packets on machine 2.
+
+```{code-cell}
+subscriber = ZMQSubscriber(address)
+
+packets = []
+
+def subscribe():
+    for packet in subscriber:
+        packets.append(packet)
+
+machine2 = threading.Thread(target=subscribe)
+machine2.start()
+```
+
+Now we wait for machine 1 to finish sending its packet and see if everything went well.
+
+```{code-cell}
+machine1.join()
+print(f"We received {len(packets)} packets!")
+assert xd.concatenate(packets).equals(da)
+```
+
+## Using encoding
+
+To reduce the volume of the transmitted data, compression is often useful. Xdas enable the use of the ZFP algorithm when storing data but also when streaming it. Encoding is declared the same way.
+
+```{code-cell}
+:tags: [remove-output]
+
+import hdf5plugin
+
+address = f"tcp://localhost:{xd.io.get_free_port()}"
+encoding = {"chunks": (10, 10), **hdf5plugin.Zfp(accuracy=1e-6)}
+publisher = ZMQPublisher(address, encoding)  # Add encoding here, the rest is the same
+```
+
+{py:class}`~xdas.io.asn.ZMQSubscriber`
+
+```{note}
+Xdas also implements the ZeroMQ protocol used by the OptoDAS interrogators by ASN. Equivalent {py:class}`~xdas.io.asn.ZMQPublisher` and {py:class}`~xdas.io.asn.ZMQSubscriber` can be found in {py:mod}`xdas.io.asn`. This can be useful get data in real-time from one instrument of that kind. Note that compression is not available with that protocol yet.
+```

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "xdas"
-version = "0.1.1"
+version = "0.1.2"
 requires-python = ">= 3.10"
 authors = [
     { name = "Alister Trabattoni", email = "alister.trabattoni@gmail.com" },
@@ -23,6 +23,7 @@ dependencies = [
     "watchdog",
     "xarray",
     "xinterp",
+    "pyzmq",
 ]
 
 [project.optional-dependencies]