add GTS-headers using CSV file to map filepath with headers and updated downloader functionality

.github/workflows/tests-docker.yml

@@ -26,9 +26,11 @@ jobs:
         docker version
         docker compose version
         python3 -V
-    - name: setup wis2box configuration
+    - name: setup wis2box configuration, replace localhost with IP on host 📦
       run: |
+        export IP=$(hostname -I | awk '{print $1}')
         cp tests/test.env wis2box.env
+        sed -i "s/localhost/$IP/g" wis2box.env
         cat wis2box.env
         python3 wis2box-ctl.py config
     - name: build wis2box
@@ -39,11 +41,18 @@ jobs:
       run: |
         python3 wis2box-ctl.py start
         python3 wis2box-ctl.py status -a
-        docker logs wis2box-management
-    - name: setup wis2box-management ⚙️
+    - name: show environment and check collections exist ⚙️
       run: |
         sleep 30
         python3 wis2box-ctl.py execute wis2box environment show
+        curl http://localhost/oapi/collections/stations
+        curl http://localhost/oapi/collections/discovery-metadata
+        curl http://localhost/oapi/collections/messages
+    - name: downloader subscribe to topics
+      run: |
+        python3 wis2box-ctl.py execute wis2box auth add-token -p wis2downloader github123 -y
+        echo '{"topic": "origin/a/wis2/+/data/core/#"}' > test.json
+        curl -X POST http://localhost/wis2downloader/subscriptions -H "Content-Type: application/json" -H "Authorization: Bearer github123" -d @test.json
     - name: populate stations from CSV 📡
       run: |
         python3 wis2box-ctl.py execute wis2box metadata station publish-collection
@@ -176,6 +185,7 @@ jobs:
     - name: sleep 30 seconds then run integration tests ⚙️
       run: |
         sleep 30
+        docker logs wis2box-management
         pytest -s tests/integration
     - name: run flake8 ⚙️
       run: |

docker-compose.monitoring.yml

@@ -114,7 +114,7 @@ services:
     <<: *logging
   web-proxy:
     <<: *logging
-  wis2-downloader:
+  wis2downloader:
     <<: *logging
 
 volumes:

docker-compose.yml

@@ -145,9 +145,9 @@ services:
     depends_on:
       - wis2box-management
 
-  wis2-downloader:
-    container_name: wis2-downloader
-    build: ./wis2-downloader
+  wis2downloader:
+    container_name: wis2downloader
+    build: ./wis2downloader
     env_file:
       - wis2box.env
     volumes:

docs/source/index.rst

@@ -33,6 +33,7 @@ The user guide helps you setup your own wis2box instance.
    user/getting-started
    user/setup
    user/data-ingest
+   user/gts-headers-in-wis2
    user/public-services-setup
    user/downloading-data
 

docs/source/user/downloading-data.rst

@@ -11,86 +11,93 @@
 WIS2 Global Services include a Global Broker that provides users the ability to subscribe to data (via topics) and download to their
 local environment / workstation / decision support system from the WIS2 Global Cache.
 
-wis2-downloader
----------------
+wis2downloader
+--------------
 
-wis2box enables subscribe and data download workflow the WIS2 network, by using the ``wis2-downloader`` container, inside of which runs the `wis2-downloader tool <https://github.com/wmo-im/wis2-downloader`_
+wis2box enables subscribe and data download workflow the WIS2 network, by using the ``wis2downloader`` container, inside of which runs the `wis2downloader tool <https://github.com/wmo-im/wis2downloader`_
 
-``wis2-downloader`` is a Python package that provides subscription and download capability, by connecting to pre-defined MQTT-broker.
+``wis2downloader`` is a Python package that provides subscription and download capability, by connecting to pre-defined MQTT-broker.
 
-The following environment variables are used by the ``wis2-downloader``:
+The following environment variables are used by the ``wis2downloader``:
 
 - ``DOWNLOAD_BROKER_HOST``: The hostname of the MQTT-broker to connect to. Defaults to ``globalbroker.meteo.fr``
 - ``DOWNLOAD_BROKER_PORT``: The port of the MQTT-broker to connect to. Defaults to ``443`` (HTTPS for websockets)
 - ``DOWNLOAD_BROKER_USERNAME``: The username to use to connect to the MQTT-broker. Defaults to ``everyone``
 - ``DOWNLOAD_BROKER_PASSWORD``: The password to use to connect to the MQTT-broker. Defaults to ``everyone``
-- ``DOWNLOAD_BROKER_PROTOCOL``: The protocol to use to connect to the MQTT-broker. Defaults to ``websockets``
+- ``DOWNLOAD_BROKER_TRANSPORT``: ``websockets`` or ``tcp``, the transport-mechanism to use to connect to the MQTT-broker. Defaults to ``websockets``,
 - ``DOWNLOAD_RETENTION_PERIOD_HOURS``: The retention period in hours for the downloaded data. Defaults to ``24``
 - ``DOWNLOAD_WORKERS``: The number of download workers to use. Defaults to ``8``. Determines the number of parallel downloads.
+- ``DOWNLOAD_MIN_FREE_SPACE_GB``: The minimum free space in GB to keep on the volume hosting the downloads. Defaults to ``1``.
 
 To override the default configuration, you can set the environment variables in the wis2box.env file.
 
-By default the wis2-downloader is not subscribed to any topics. You can add subscriptions using the API endpoint, as described below.
+By default the wis2downloader is not subscribed to any topics. You can add subscriptions using the `wis2box downloader` commands, which will call the API over the internal docker network.
 
-The files downloaded by the wis2-downloader will be saved in `${WIS2BOX_HOST_DATADIR}/downloads`, where `${WIS2BOX_HOST_DATADIR}` is the directory on your host you defined in the `wis2box.env` file.
+The files downloaded by the wis2downloader will be saved in `${WIS2BOX_HOST_DATADIR}/downloads`, where `${WIS2BOX_HOST_DATADIR}` is the directory on your host you defined in the `wis2box.env` file.
 
 Maintaining and Monitoring Subscriptions
 ----------------------------------------
 
-The wis2-downloader has an API-endpoint that is proxied on the path `/wis2-downloader` on the wis2box host-url. This endpoint can be used to add, delete and list subscriptions.
+Inside the 'wis2downloader' container, you can use the CLI to list, add and delete subscriptions.
 
-The endpoint is secured using an API token, that can be created using the 'wis2box auth' command inside the wis2box-management container as follows:
+You can also interact with the `wis2downloader` API-endpoint from outside the wis2box-host using curl or other HTTP clients and providing an authentication token in the request headers.
 
-```bash
-python3 wis2box.ctl.py execute wis2box auth add-token --path wis2-downloader -y
-```
+Logging into the wis2downloader container
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Record the generated token, so you can use it to authenticate requests to the API.
+To log into the wis2downloader container, you can use the following command:
+
+.. code-block:: console
+
+  python3 wis2box-ctl.py login wis2downloader
+
+This will log you into the container and you can use the CLI to interact with the subscriptions.
 
 Listing subscriptions
 ~~~~~~~~~~~~~~~~~~~~~
 
-To list the active subscriptions, a GET request can be made to the `wis2-downloader/list` endpoint, making sure to pass the API token as a header:
+To list the current subscriptions, you can use the following command:
+
+.. code-block:: console
+
+  wis2downloader list-subscriptions
+
+This will return a JSON object with the current subscriptions.
+
+Adding a subscription
+~~~~~~~~~~~~~~~~~~~~~
+
+To add a subscription, you can use the following command:
+
+.. code-block:: console
+
+  wis2downloader add-subscription --topic <topic>
+
+This will add a subscription to the topic you specify and return the JSON object with the current subscriptions.
 
-```bash
-curl http://localhost/wis2-downloader/list -H "Authorization: Bearer <API-token>"
-```
+Deleting a subscription
+~~~~~~~~~~~~~~~~~~~~~~~
 
-The list of the currently active subscriptions should be returned as a JSON object.
+To delete a subscription, you can use the following command:
 
-Adding subscriptions
-~~~~~~~~~~~~~~~~~~~~
+.. code-block:: console
 
-Subscriptions can be added via a GET request to the `./add` endpoint that is proxied on /wis2-downloader on the wis2box host, with the following form:
+  wis2downloader delete-subscription --topic <topic>
 
-```bash
-curl http://localhost/wis2-downloader/add?topic=<topic-name>&target=<download-directory> -H "Authorization: Bearer <API-token>"
-```
+This will delete the subscription to the topic you specify and return the JSON object with the current subscriptions.
 
-- `topic` specifies the topic to subscribe to. *Special characters (+, #) must be URL encoded, i.e. `+` = `%2B`, `#` = `%23`.*
-- `target` specifies the directory to save the downloads to, relative to `download_dir` from `config.json`. *If this is not provided, the directory will default to that of the topic hierarchy.*
 
-For example:
-```bash
-curl http://localhost/wis2-downloader/add?topic=cache/a/wis2/%2B/data/core/weather/%23&target=example_data -H "Authorization: Bearer <API-token>"
-```
+Managing subscriptions from outside the wis2box
+-----------------------------------------------
 
-The list of active subscriptions after addition should be returned as a JSON object.
+The wis2downloader API-endpoint is proxied on the path `/wis2downloader` on the wis2box host-url, allowing you to interact with it using curl or other HTTP clients from any machine that can reach the wis2box host.
 
-Deleting subscriptions
-~~~~~~~~~~~~~~~~~~~~~~
+The wis2box-proxy by default secures the path `/wis2downloader` with a bearer token, which can be generated using the `wis2box auth` command as follows:
 
-Subscriptions are deleted similarly via a GET request to the `./delete` endpoint, with the following form:
-```bash
-curl http://<flask-host>:<flask-port>/delete?topic=<topic-name> -H "Authorization: Bearer <API-token>"
-```
+.. code-block:: console
 
-For example:
-```bash
-curl http://localhost:8080/delete?topic=cache/a/wis2/%2B/data/core/weather/%23 -H "Authorization: Bearer <API-token>"
-```
+  python3 wis2box.ctl.py execute wis2box auth add-token --path wis2downloader -y
 
-The list of active subscriptions after deletion should be returned as a JSON object.
 
 
 

docs/source/user/gts-headers-in-wis2.rst

@@ -0,0 +1,57 @@
+.. _gts-headers-in-wis2:
+
+Optional: adding GTS headers to WIS2 notifications during the transition period
+===============================================================================
+
+Overview
+--------
+
+This section provides guidance how to add GTS headers to your WIS2 notifications.
+
+By adding GTS headers to your WIS2 notifications, you can stop your MSS and still have your data available on the GTS during the transition period.
+
+To enable the WIS2 to GTS Gateway to correctly identify the data to be republished on the GTS, you need to include the GTS property in the WIS2 Notification Message as follows:
+
+.. code-block:: json
+
+   "properties": {
+     "gts": {
+       "ttaaii": "FTAE31",
+       "cccc": "VTBB"
+     }
+   }
+
+wis2box can add these to WIS2 Notifications automatically, provided you specify the additional file `gts_headers_mappings.json` that contains the required information to map the GTS headers to the incoming filenames.
+
+Note that this is optional and only required if you want to turn off the existing system responsible for sending data to the GTS during the transition period.
+
+gts_headers_mapping.csv
+-----------------------
+
+If you want to add GTS headers to your WIS2 notifications, a CSV file is required that maps GTS headers to incoming filenames. 
+
+The CSV file should be named (exactly) `gts_headers_mapping.csv` and should be placed in the directory defined using the `WIS2BOX_HOST_DATADIR` environment variable.
+
+The CSV should contain the following columns: `string_in_filepath`, `TTAAii`, and `CCCC`.
+
+Example content for `gts_headers_mapping.csv`:
+
+.. code-block:: csv
+
+   string_in_filepath,TTAAii,CCCC
+   ISMD01LIIB,ISMD01,LIBB
+   ISMD02LIIB,ISMD02,LIBB
+
+In this example, whenever `ISMD01LIIB` or `ISMD02LIIB` is contained in the filepath of the incoming file,
+the corresponding GTS headers will be added to the WIS2 Notification Message as a dictionary in the `properties` field:
+
+.. code-block:: json
+
+   "properties": {
+     "gts": {
+       "ttaaii": "ISMD01",
+       "cccc": "LIBB"
+     }
+   }
+
+If the `gts_headers_mapping.csv` file is not present in the directory you defined using the `WIS2BOX_HOST_DATADIR` environment variable, wis2box will not add any GTS headers to the WIS2 Notification Message.