Merge pull request #294 from OCHA-DAP/dev

dev to prod for 0.9.4

dev-requirements.txt

@@ -1,13 +1,13 @@
 mkdocs~=1.6.1
 mkdocs-include-markdown-plugin~=7.1.2
-mkdocs-material~=9.5.50
+mkdocs-material~=9.6.2
 mkdocs-table-reader-plugin~=3.1.0
 
 pytest~=8.3.4
 pytest-cov~=6.0.0
 pytest-asyncio~=0.25.3
 httpx~=0.28.1
 
-ruff~=0.9.3
+ruff~=0.9.4
 
 toml

docs/changelog.md

@@ -17,6 +17,22 @@ the API.
 
 ## Latest changes
 
+### Unreleased
+- Aligned category and sub-category endpoint names in the docs with
+  [HDX data grids](https://data.humdata.org/dashboards/overview-of-data-grids?)
+  - affected-people/refugees -> affected-people/refugees-persons-of-concern
+  - population-social/population -> geography-infrastructure/baseline-population
+  - coordination-context/conflict-event -> coordination-context/conflict-events
+  - food/food-security -> food-security-nutrition-poverty/food-security
+  - food/food-price -> food-security-nutrition-poverty/food-prices-market-monitor
+  - population-social/poverty-rate -> food-security-nutrition-poverty/poverty-rate
+  - population-social/population -> geography-infrastructure/baseline-population
+- Note that the new endpoints are being released under v2 of the api and 
+  previous versions are maintained under v1
+
+### 2025-01-06
+- Added functionality to filter by time period
+
 ### 2024-12-19
 
 - Increased temporal coverage for humanitarian needs

docs/data_usage_guides/coordination_and_context.md

@@ -2,9 +2,9 @@
 
 ---
 
-## Who is Doing What Where - Operational Presence <a id=”operational-presence”></a>
+## Operational Presence <a id=”operational-presence”></a>
 
-The [Who is Doing What Where (3W)](https://3w.unocha.org/) is a core
+[Operational presence](https://3w.unocha.org/) is a core
 humanitarian coordination dataset that contains the geographic and sectoral
 spread of humanitarian activities and partners. It is critical to know where
 humanitarian organizations are working and what they are doing in order to
@@ -27,16 +27,16 @@ For available query parameters, please see the
 
 * For consistency and interoperability, we aggregate to an
   [operational presence](https://humanitarian.atlassian.net/wiki/spaces/imtoolbox/pages/214499412/Who+does+What+Where+3W)
-  level (3W:OP, per org, sector, and admin 2), even if the original 3W data is
+  level (3W:OP, per org, sector, and admin 2), even if the original data is
   more detailed (e.g. the source lists individual activities)
 * Countries that are not p-coded are p-coded to the national level and include
   admin names from the providers.
 * Organization deduplication is a long-running challenge with this data, since
   there are no unique identifiers, and organization names may be spelled
   different ways by different OCHA offices, or sometimes even within the same
-  3W. See the [`org`](metadata.md#org) section below for more information on how we handle
-  these details.
-* The sector name strings in the 3W data are normalised and then aligned to the
+  dataset. See the [`org`](metadata.md#org) section below for more information
+  on how we handle these details.
+* The sector name strings in the data are normalised and then aligned to the
   [sector](metadata.md#sector) table, using the “sector_map” section of
   [this configuration file](https://github.com/OCHA-DAP/hapi-pipelines/blob/main/src/hapi/pipelines/configs/core.yaml)
   if needed. In the absence of a direct match, phonetic matching is used for
@@ -50,7 +50,7 @@ For available query parameters, please see the
 * Available at either Admin 0, 1, or 2 depending on the country. Please check
   data coverage for further information
 * We cannot guarantee org consistency over time; for example, if an IMO
-  changes, the 3W might change the spelling or the whole name of an org between
+  changes, the spelling or the whole name of an org might change between
   releases
 
 ## Funding <a id=”funding”></a>

docs/data_usage_guides/endpoint_parameters/humanitarian_needs_parameters.yaml

@@ -1,6 +1,11 @@
 - Parameter: '`resource_hdx_id`'
   Description: Unique resource UUID on HDX
   Source: '[`Resource`](metadata.md#resource)'
+- Parameter: '`category`'
+  Description: >
+    A category combining gender, age range, disability marker and 
+    population group information
+  Source:
 - Parameter: '`sector_code`'
   Description: >
     The sector code, derived either from the

docs/data_usage_guides/enums.md

@@ -18,7 +18,7 @@ the IDP data coming from the IOM DTM. For more detail, see the
 ## Commodity Category <a id="commodity-category"></a>
 
 **Used in:**
-[`Food Prices`](food_security_and_nutrition.md#food-price),
+[`Food Prices & Market Monitor`](food_security_nutrition_and_poverty.md#food-price),
 [`WFP Commodity`](metadata.md#wfp-commodity)
 
 The commodity categories are used in the WFP food prices data to organize
@@ -52,7 +52,7 @@ for their methodology and more detailed descriptions of the sub-event types.
 ## Gender <a id="gender"></a>
 
 **Used in:**
-[`Baseline Population`](population_and_socio-economy.md#population),
+[`Baseline Population`](geography_and_infrastructure.md#population),
 [`Humanitarian Needs`](affected_people.md#humanitarian-needs),
 [`Refugees & Persons of Concern`](affected_people.md#refugees)
 
@@ -62,7 +62,7 @@ Several sub-categories in HDX-HAPI are disaggregated by gender.
 
 ## IPC Code <a id="ipc-code"></a>
 
-**Used in:** [`Food Security`](food_security_and_nutrition.md#food-security)
+**Used in:** [`Food Security`](food_security_nutrition_and_poverty.md#food-security)
 
 The IPC/CH classification includes 5 different phases of increasing severity,
 described in detail on page 53 of
@@ -75,7 +75,7 @@ and total population, used to compute fractions.
 
 ## IPC Type <a id="ipc_type"></a>
 
-**Used in:** [`Food Security`](food_security_and_nutrition.md#food-security)
+**Used in:** [`Food Security`](food_security_nutrition_and_poverty.md#food-security)
 
 The IPC and Cadre Harmonisé provide different projections to aid in planning
 and response efforts.
@@ -113,15 +113,15 @@ multiple groups.
 
 ## Price Flag <a id="price-flag"></a>
 
-**Used in:** [`Food Prices`](food_security_and_nutrition.md#food-price)
+**Used in:** [`Food Prices & Market Monitor`](food_security_nutrition_and_poverty.md#food-price)
 
 Pre-processing characteristics of food prices.
 
 {{ read_yaml('data_usage_guides/enum_parameters/price_flag_parameters.yaml') }}
 
 ## Price Type <a id="price-type"></a>
 
-**Used in:** [`Food Prices`](food_security_and_nutrition.md#food-price)
+**Used in:** [`Food Prices & Market Monitor`](food_security_nutrition_and_poverty.md#food-price)
 
 The point in the supply chain at which the price is determined.
 See FAO's

docs/data_usage_guides/food_security_and_nutrition.md → docs/data_usage_guides/food_security_nutrition_and_poverty.md

@@ -1,4 +1,4 @@
-# Food Security & Nutrition
+# Food Security, Nutrition & Poverty
 
 ---
 
@@ -21,7 +21,7 @@ it aligns with IPC standards in terms of assessment processes and outcomes.
 
 The table below describes the parameters returned from this endpoint.
 For available query parameters, please see the
-[API sandbox](https://hapi.humdata.org/docs#/Food%20Security%20%26%20Nutrition/get_food_security_api_v1_food_food_security_get).
+[API sandbox](https://hapi.humdata.org/docs#/Food%20Security%20Nutrition%20%26%20Poverty/get_food_security_api_v1_food_food_security_get).
 
 {{ read_yaml('data_usage_guides/endpoint_parameters/food_security_parameters.yaml') }}
 
@@ -66,7 +66,7 @@ For available query parameters, please see the
 | SOM | Somalia | Admin 2 regions in Somalia are sub-divided, thus we to not attempt to assign p-codes |
 | ZAF | South Africa | Admin 2 regions are a mix of admin levels, thus we do not attempt to assign p-codes at admin 2 |
 
-## Food Prices <a id="food-price"></a>
+## Food Prices & Market Monitor <a id="food-price"></a>
 
 The World Food Programme Price Database covers foods such as maize, rice,
 beans, fish, and sugar for 98 countries and some 3000 markets. It is updated
@@ -82,7 +82,7 @@ detailed methodology, see WFP's
 
 The table below describes the parameters returned from this endpoint.
 For available query parameters, please see the
-[API sandbox](https://hapi.humdata.org/docs#/Food%20Security%20%26%20Nutrition/get_food_price_api_v1_food_food_price_get).
+[API sandbox](https://hapi.humdata.org/docs#/Food%20Security%20Nutrition%20%26%20Poverty/get_food_price_api_v1_food_food_price_get).
 
 {{ read_yaml('data_usage_guides/endpoint_parameters/food_price_parameters.yaml') }}
 
@@ -94,3 +94,50 @@ For available query parameters, please see the
 * The source data is not p-coded, however we have used the admin 1 and 2 names
   to p-code most markets. See [WFP Market](metadata.md#wfp-market)
   for more details.
+
+## Poverty Rate <a id="poverty-rate"></a>
+
+The global [Oxford Multidimensional Poverty Index](https://ophi.org.uk/global-mpi)
+(MPI) measures multidimensional poverty in over 100 developing countries,
+using internationally comparable datasets. The MPI assesses poverty through
+three main dimensions: health, education, and living standards, each of which
+is represented by specific indicators. For each country, MPI trends over time 
+are supplied if available. Relevant OPHI methodological notes are 
+[58](https://ophi.org.uk/publications/MN-58), 
+[59](https://ophi.org.uk/publications/MN-59) and 
+[60](https://ophi.org.uk/publications/MN-60).
+
+### Summary
+
+{{ read_yaml('data_usage_guides/subcategory_details/poverty_rate_details.yaml') }}
+
+### Parameters Returned
+
+The table below describes the parameters returned from this endpoint.
+For available query parameters, please see the
+[API sandbox](https://hapi.humdata.org/docs#/Food%20Security%20Nutrition%20%26%20Poverty/get_poverty_rate_api_v1_population_social_poverty_rate_get).
+
+{{ read_yaml('data_usage_guides/endpoint_parameters/poverty_rate_parameters.yaml') }}
+
+### Transformations applied
+
+* For rows in the original data with two timepoints, we take each timepoint as
+  single entry into HDX HAPI
+* The reference period is constructed using the full range of the year or year
+  range presented in the “year” column, pertaining to the timepoint in
+  question, of the original data
+
+### Usage Notes
+
+* The data are available at the national and admin 1 level
+* The admin name from the provider is supplied along with p-codes and 
+  corresponding standardised admin names where available
+* We use p-codes from the source data which was p-coded by taking the admin 1 
+  names, and applying the algorithm from [`hdx-python-country`](https://hdx-python-country.readthedocs.io/en/latest/)
+* Where admin 1 names could not be p-coded, the provided p-codes from the 
+  source data are at national level
+* Trends are estimated using indicators in the global MPI that are harmonised 
+  across the time periods and are used where data are available for a country
+* For any country where trends are unavailable in the source, the latest data
+  (which is not harmonised across time) are used instead
+

docs/data_usage_guides/geography_and_infrastructure.md

@@ -0,0 +1,43 @@
+# Geography & Infrastructure
+
+---
+
+## Baseline Population <a id="population"></a>
+
+The population statistics presented here are sourced from the
+[common operational datasets](https://cod.unocha.org/) (CODs), and are
+typically disaggregated by age and/or gender, and extend to administrative
+levels 1 or 2. Primary data providers include the UNFPA and OCHA country
+offices. These data are projections based on demographic indicators, and
+therefore caution should be used when comparing to population data from other
+sources.
+
+### Summary
+
+{{ read_yaml('data_usage_guides/subcategory_details/population_details.yaml') }}
+
+### Parameters Returned
+
+The table below describes the parameters returned from this endpoint.
+For available query parameters, please see the
+[API sandbox](https://hapi.humdata.org/docs#/Geography%20%26%20Infrastructure/get_population_api_v1_population_social_population_get).
+
+{{ read_yaml('data_usage_guides/endpoint_parameters/population_parameters.yaml') }}
+
+### Transformations applied
+
+* The table has been reshaped from wide to long: demographic-specific columns
+  have been cast to `gender`, `age_range`, and `population`
+* The reference period is obtained from the HDX dataset
+
+### Usage Notes
+
+* Age disaggregation ranges are not consistent across countries
+* Any aggregation to a higher administrative level (e.g., admin 1 for a
+  country where admin 2 is also available) has been taken directly from the
+  data provided, and was not computed in the API pipeline
+* An “all” value in the `gender` and `age_range` columns indicates no
+  disaggregation
+* `age_range` is expressed as "[`min_age`]-[`max_age`]", where `max_age` is
+  inclusive, or "[`min_age`]+" for an age range starting at `min_age` or above
+

docs/data_usage_guides/metadata.md

@@ -127,7 +127,7 @@ For available query parameters, please see the
 ### Currency <a id="currency"></a>
 
 **Used in:**
-[`Food Prices`](food_security_and_nutrition.md#food-price)
+[`Food Prices & Market Monitor`](food_security_nutrition_and_poverty.md#food-price)
 
 The currency table is populated using the WFP VAM Data Bridges API.
 
@@ -142,20 +142,20 @@ For available query parameters, please see the
 ### Org <a id="org"></a>
 
 **Used in:**
-[`Who is Doing What Where - Operational Presence`](coordination_and_context.md#operational-presence)
+[`Operational Presence`](coordination_and_context.md#operational-presence)
 
-The organization table is populated from the 3W data, using the following
-methodology:
+The organization table is populated from the operational presence data, using
+the following methodology:
 
 * Organization name and acronym strings are normalised. If an acronym isn’t
   available, the first 32 characters of the name are used.
 * This [organization mapping](https://docs.google.com/spreadsheets/d/e/2PACX-1vSfBWvSu3fKA743VvHtgf-pIGkYH7zhy-NP7DZgEV9_a6YU7vtCeWhbLM56aUL1iIfrfv5UBvvjVt7B/pub?gid=1040329566&single=true&output=csv)
   is used for common alternative names
 * Organizations must have an associated organization type. If available, the
-  organization type code is taken directly from the 3W data, otherwise the name
-  string is normalised and matched to the org type names. In the absence of a
-  direct match, phonetic matching is used for strings > 5 characters. If no
-  match is found, the organization is skipped.
+  organization type code is taken directly from the operational presence data,
+  otherwise the name string is normalised and matched to the org type names.
+  In the absence of a direct match, phonetic matching is used for strings > 5 
+  characters. If no match is found, the organization is skipped.
 
 <h4> Parameters Returned </h4>
 
@@ -169,7 +169,7 @@ For available query parameters, please see the
 
 **Used in:**
 [`Org`](metadata.md#org),
-[`Who is Doing What Where - Operational Presence`](coordination_and_context.md#operational-presence)
+[`Operational Presence`](coordination_and_context.md#operational-presence)
 
 The table is initially populated using the
 [OCHA Digital Services organization types list](https://data.humdata.org/dataset/organization-types-beta).
@@ -195,7 +195,7 @@ For available query parameters, please see the
 ### Sector <a id="sector"></a>
 
 **Used in:**
-[`Who is Doing What Where - Operational Presence`](coordination_and_context.md#operational-presence),
+[`Operational Presence`](coordination_and_context.md#operational-presence),
 [`Humanitarian Needs`](affected_people.md#humanitarian-needs)
 
 This table is initially populated using the
@@ -222,7 +222,7 @@ For available query parameters, please see the
 ### WFP Commodity <a id="wfp-commodity"></a>
 
 **Used in:**
-[`Food Prices`](food_security_and_nutrition.md#food-price)
+[`Food Prices & Market Monitor`](food_security_nutrition_and_poverty.md#food-price)
 
 The commodity table tracks all food items, and their associated
 commodity category, present in the food prices data.
@@ -238,7 +238,7 @@ For available query parameters, please see the
 ### WFP Market <a id="wfp-market"></a>
 
 **Used in:**
-[`Food Prices`](food_security_and_nutrition.md#food-price)
+[`Food Prices & Market Monitor`](food_security_nutrition_and_poverty.md#food-price)
 
 Markets are defined as the physical locations where buyers and sellers
 come together to trade goods and services.