Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WCMP2 and internal message bus fixes #628

Merged
merged 5 commits into from
Feb 27, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/tests-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
env:
TOPIC_HIERARCHY: mw-mw_met_centre.data.core.weather.surface-based-observations.synop
DISCOVERY_METADATA: /data/wis2box/metadata/discovery/mw-surface-weather-observations.yml
DISCOVERY_METADATA_ID: urn:x-wmo:md:mw-mw_met_centre:surface-weather-observations
DISCOVERY_METADATA_ID: urn:wmo:md:mw-mw_met_centre:surface-weather-observations
TEST_DATA: /data/wis2box/observations/malawi
TEST_DATA_UPDATE: /data/wis2box/observations/malawi_update
run: |
Expand All @@ -64,7 +64,7 @@ jobs:
env:
TOPIC_HIERARCHY: it-roma_met_centre.data.core.weather.surface-based-observations.synop
DISCOVERY_METADATA: /data/wis2box/metadata/discovery/it-surface-weather-observations.yml
DISCOVERY_METADATA_ID: urn:x-wmo:md:it-roma_met_centre:surface-weather-observations
DISCOVERY_METADATA_ID: urn:wmo:md:it-roma_met_centre:surface-weather-observations
TEST_DATA: /data/wis2box/observations/italy
run: |
python3 wis2box-ctl.py execute wis2box dataset publish $DISCOVERY_METADATA
Expand All @@ -75,7 +75,7 @@ jobs:
env:
TOPIC_HIERARCHY: dz-alger_met_centre.data.core.weather.surface-based-observations.synop
DISCOVERY_METADATA: /data/wis2box/metadata/discovery/dz-surface-weather-observations.yml
DISCOVERY_METADATA_ID: urn:x-wmo:md:dz-alger_met_centre:surface-weather-observations
DISCOVERY_METADATA_ID: urn:wmo:md:dz-alger_met_centre:surface-weather-observations
TEST_DATA: /data/wis2box/observations/algeria
run: |
python3 wis2box-ctl.py execute wis2box dataset publish $DISCOVERY_METADATA
Expand All @@ -86,7 +86,7 @@ jobs:
env:
TOPIC_HIERARCHY: ro-rnimh.data.core.weather.surface-based-observations.synop
DISCOVERY_METADATA: /data/wis2box/metadata/discovery/ro-synoptic-weather-observations.yml
DISCOVERY_METADATA_ID: urn:x-wmo:md:ro-rnimh:synoptic-weather-observations
DISCOVERY_METADATA_ID: urn:wmo:md:ro-rnimh:synoptic-weather-observations
TEST_DATA: /data/wis2box/observations/romania
run: |
python3 wis2box-ctl.py execute wis2box dataset publish $DISCOVERY_METADATA
Expand All @@ -97,7 +97,7 @@ jobs:
env:
TOPIC_HIERARCHY: cd-brazza_met_centre.data.core.weather.surface-based-observations.synop
DISCOVERY_METADATA: /data/wis2box/metadata/discovery/cd-surface-weather-observations.yml
DISCOVERY_METADATA_ID: urn:x-wmo:md:cd-brazza_met_centre:surface-weather-observations
DISCOVERY_METADATA_ID: urn:wmo:md:cd-brazza_met_centre:surface-weather-observations
TEST_DATA: /data/wis2box/observations/congo
run: |
python3 wis2box-ctl.py execute wis2box dataset publish $DISCOVERY_METADATA
Expand All @@ -108,7 +108,7 @@ jobs:
env:
TOPIC_HIERARCHY: cn-cma.data.core.weather.prediction.forecast.medium-range.probabilistic.global
DISCOVERY_METADATA: /data/wis2box/metadata/discovery/cn-grapes-geps-global.yml
DISCOVERY_METADATA_ID: urn:x-wmo:md:cn-cma:grapes-geps-global
DISCOVERY_METADATA_ID: urn:wmo:md:cn-cma:grapes-geps-global
TEST_DATA: /data/wis2box/observations/china
run: |
python3 wis2box-ctl.py execute wis2box dataset publish $DISCOVERY_METADATA
Expand Down
4 changes: 2 additions & 2 deletions config-templates/metadata-synop.yml.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:$CENTRE_ID:surface-based-observations.synop
identifier: urn:wmo:md:$CENTRE_ID:surface-based-observations.synop
language: en
charset: utf8
hierarchylevel: dataset
Expand All @@ -32,7 +32,7 @@ identification:
keywords_type: theme
vocabulary:
name: WMO WIS2 Topic Hierarchy
url: https://github.com/wmo-im/wis2-topic-hierarchy/blob/main/topic-hierarchy/earth-system-discipline/index.csv
url: https://codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
extents:
spatial:
- bbox: [$BOUNDING_BOX]
Expand Down
4 changes: 2 additions & 2 deletions config-templates/metadata-temp.yml.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:$CENTRE_ID:surface-based-observations.temp
identifier: urn:wmo:md:$CENTRE_ID:surface-based-observations.temp
language: en
charset: utf8
hierarchylevel: dataset
Expand All @@ -33,7 +33,7 @@ identification:
keywords_type: theme
vocabulary:
name: WMO WIS2 Topic Hierarchy
url: https://github.com/wmo-im/wis2-topic-hierarchy/blob/main/topic-hierarchy/earth-system-discipline/index.csv
url: https://codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
extents:
spatial:
- bbox: [$BOUNDING_BOX]
Expand Down
4 changes: 2 additions & 2 deletions examples/config/surface-weather-observations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:centre_id:surface-weather-observations
identifier: urn:wmo:md:centre_id:surface-weather-observations
language: en
charset: utf8
hierarchylevel: dataset
Expand All @@ -33,7 +33,7 @@ identification:
keywords_type: theme
vocabulary:
name: WMO WIS2 Topic Hierarchy
url: https://github.com/wmo-im/wis2-topic-hierarchy/blob/main/topic-hierarchy/earth-system-discipline/index.csv
url: https://codes.wmo.int/topic-hierarchy/earth-system-discipline
extents:
spatial:
- bbox: [-180, -90, 190, 90]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:cd-brazza_met_centre:surface-weather-observations
identifier: urn:wmo:md:cd-brazza_met_centre:surface-weather-observations
hierarchylevel: dataset

identification:
Expand All @@ -39,7 +39,7 @@ identification:
keywords_type: theme
vocabulary:
name: Earth system disciplines as defined by the WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021), Annex 1.
url: https://github.com/wmo-im/wis2-topic-hierarchy/blob/main/topic-hierarchy/earth-system-discipline/index.csv
url: https://codes.wmo.int/topic-hierarchy/earth-system-discipline
extents:
spatial:
- bbox: [11.0937728207,-5.03798674888,18.4530652198,3.72819651938]
Expand Down
6 changes: 3 additions & 3 deletions tests/data/metadata/discovery/cn-grapes-geps-global.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:cn-cma:grapes-geps-global
identifier: urn:wmo:md:cn-cma:grapes-geps-global
hierarchylevel: dataset

identification:
Expand All @@ -42,7 +42,7 @@ identification:
keywords_type: theme
vocabulary:
name: Earth system disciplines as defined by the WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021), Annex 1.
url: https://github.com/wmo-im/wis2-topic-hierarchy/blob/main/topic-hierarchy/earth-system-discipline/index.csv
url: https://codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
extents:
spatial:
- bbox: [73.66000, 4.00000, 135.08000, 53.52000]
Expand All @@ -51,7 +51,7 @@ identification:
- begin: 2021-11-29
end: null
resolution: P12H
url: http://gisc.wis.cma.cn/wis/portal.pub?M_PID=urn:x-wmo:md:int.wmo.wis::CMA_GEPS
url: http://gisc.wis.cma.cn/wis/portal.pub?M_PID=urn:wmo:md:int.wmo.wis::CMA_GEPS
wmo_data_policy: core

contact:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:dz-alger_met_centre:surface-weather-observations
identifier: urn:wmo:md:dz-alger_met_centre:surface-weather-observations
hierarchylevel: dataset

identification:
Expand All @@ -40,7 +40,7 @@ identification:
keywords_type: theme
vocabulary:
name: Earth system disciplines as defined by the WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021), Annex 1.
url: https://github.com/wmo-im/wis2-topic-hierarchy/blob/main/topic-hierarchy/earth-system-discipline/index.csv
url: https://codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
extents:
spatial:
- bbox: [-8.68439978681, 19.0573642034, 11.9995056495, 37.1183806422]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:it-roma_met_centre:surface-weather-observations
identifier: urn:wmo:md:it-roma_met_centre:surface-weather-observations
hierarchylevel: dataset

identification:
Expand All @@ -37,7 +37,7 @@ identification:
keywords_type: theme
vocabulary:
name: Earth system disciplines as defined by the WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021), Annex 1.
url: https://github.com/wmo-im/wis2-topic-hierarchy/blob/main/topic-hierarchy/earth-system-discipline/index.csv
url: https://codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
extents:
spatial:
- bbox: [6.74995, 36.61998, 18.48024, 47.11539]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:mw-mw_met_centre:surface-weather-observations
identifier: urn:wmo:md:mw-mw_met_centre:surface-weather-observations
hierarchylevel: dataset

identification:
Expand All @@ -38,7 +38,7 @@ identification:
keywords_type: theme
vocabulary:
name: Earth system disciplines as defined by the WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021), Annex 1.
url: https://github.com/wmo-im/wis2-topic-hierarchy/blob/main/topic-hierarchy/earth-system-discipline/index.csv
url: https://codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
extents:
spatial:
- bbox: [32.6881653175,-16.8012997372,35.7719047381,-9.23059905359]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:ro-rnimh:synoptic-weather-observations
identifier: urn:wmo:md:ro-rnimh:synoptic-weather-observations
hierarchylevel: dataset

identification:
Expand All @@ -42,7 +42,7 @@ identification:
keywords_type: theme
vocabulary:
name: Earth system disciplines as defined by the WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021), Annex 1.
url: https://github.com/wmo-im/wis2-topic-hierarchy/blob/main/topic-hierarchy/earth-system-discipline/index.csv
url: https://codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
extents:
spatial:
- bbox: [20.2201924985,43.6884447292,29.62654341,48.2208812526]
Expand Down
2 changes: 1 addition & 1 deletion tests/integration/test_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@

URL = 'http://localhost'
API_URL = f'{URL}/oapi'
ID = 'urn:x-wmo:md:mw-mw_met_centre:surface-weather-observations'
ID = 'urn:wmo:md:mw-mw_met_centre:surface-weather-observations'
SESSION = Session()
SESSION.hooks = {
'response': lambda r, *args, **kwargs: r.raise_for_status()
Expand Down
4 changes: 2 additions & 2 deletions wis2box-management/wis2box/pubsub/subscribe.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,13 +95,13 @@ def on_message_handler(self, client, userdata, msg):
LOGGER.debug('Refreshing data mappings')
self.data_mappings = get_data_mappings()
return
elif topic == 'wis2box/dataset_publication':
elif topic == 'wis2box/dataset/publication':
LOGGER.debug('Publishing dataset')
metadata = message
discovery_metadata.publish_discovery_metadata(metadata)
data_.add_collection_data(metadata)
return
elif topic.startswith('wis2box/dataset_unpublication'):
elif topic.startswith('wis2box/dataset/unpublication'):
LOGGER.debug('Unpublishing dataset')
identifier = topic.split('/')[-1]
remove_collection(identifier)
Expand Down
Loading