Merge pull request #573 from ecmwf/develop

Release 0.12.0

Author: sandorkertesz

docs/examples/grib_metadata_object.ipynb

@@ -26,7 +26,7 @@
     "tags": []
    },
    "source": [
-    "We will work with a GRIB file containing 6 messages. First we ensure the example file is available, then read the file with :ref:`from_source() <data-sources-file>`."
+    "In this notebook we will work with a GRIB file containing 6 messages. First we ensure the example file is available, then read the file with :ref:`from_source() <data-sources-file>`."
    ]
   },
   {
@@ -137,7 +137,7 @@
     {
      "data": {
       "text/plain": [
-       "<earthkit.data.readers.grib.metadata.GribFieldMetadata at 0x294e37ee0>"
+       "<earthkit.data.readers.grib.metadata.GribFieldMetadata at 0x2aba44130>"
       ]
      },
      "execution_count": 3,
@@ -246,21 +246,84 @@
     },
     "tags": []
    },
+   "outputs": [],
+   "source": [
+    "md_copy = md.override()"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "1c511db3-f841-4936-a7cb-25dd3e3a1010",
+   "metadata": {
+    "editable": true,
+    "raw_mimetype": "text/restructuredtext",
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "By default :py:meth:`~data.readers.grib.metadata.GribMetadata.override` is called with the ``headers_only_clone=True`` option to clone a new GRIB handle with all the data values (and some related information) removed. With this the resulting object can be significantly smaller, especially if the data section is large. The downside is that now the value related keys either cannot be accessed or give back wrong values. E.g when using the \"average\" key we get:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "c9ab97f0-40b7-403e-b7c9-6ce16c894d7d",
+   "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "<earthkit.data.readers.grib.metadata.StandAloneGribMetadata at 0x294e7cee0>"
+       "(279.70703560965404, 47485.4296875)"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "md_copy = md.override()\n",
-    "md_copy"
+    "md[\"average\"], md_copy[\"average\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fe340c05-ac04-4f7d-8c43-3ed1f415a979",
+   "metadata": {},
+   "source": [
+    "To get a copy without shrinking the GRIB handle use ``headers_only_clone=False``."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "a789bb5e-b493-45cd-b7ce-e5cb8b34e125",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "md_copy_full = md.override(headers_only_clone=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "3b030549-2a7e-494e-bd46-0a4cd19a09ad",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "279.70703560965404"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "md_copy_full[\"average\"]"
    ]
   },
   {
@@ -294,7 +357,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 9,
    "id": "40c6d232-03de-402b-82bf-8647e8a7bece",
    "metadata": {
     "editable": true,
@@ -310,7 +373,7 @@
        "('z', 850)"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -336,7 +399,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 10,
    "id": "ef78a3ec-4ea2-4ff5-8c90-e60b5e07e77f",
    "metadata": {
     "editable": true,
@@ -352,7 +415,7 @@
        "('t', 1000)"
       ]
      },
-     "execution_count": 7,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -386,7 +449,7 @@
     "tags": []
    },
    "source": [
-    "GRIB metadata objects play a part in building new fieldlist from (altered) values and metadata."
+    "GRIB metadata objects play a part in building new fieldlists from (altered) values and metadata."
    ]
   },
   {
@@ -405,7 +468,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 11,
    "id": "cb59ad5f-c48b-4943-984d-3abdf48fda8d",
    "metadata": {
     "editable": true,
@@ -427,7 +490,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 12,
    "id": "85c32bfb-c929-404f-add9-9adae40418d2",
    "metadata": {
     "editable": true,
@@ -496,7 +559,7 @@
        "0       an       0  regular_ll  "
       ]
      },
-     "execution_count": 9,
+     "execution_count": 12,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -520,12 +583,12 @@
     }
    },
    "source": [
-    "Please note that the resulting :py:class:`~data.sources.array_list.ArrayFieldList` always contains a :py:class:`~data.readers.grib.metadata.RestrictedGribMetadata` object for each field. These objects possess their own GRIB handles, which is ensured by creating a copy with ``override()`` when needed. On top of that metadata access is limited to keys not related to data values. Getting metadata on any other keys will throw an exception. "
+    "The resulting fieldlist contains an :py:class:`~data.sources.array_list.ArrayField`, which is  composed of a numpy array storing the values and a metadata object owning its own GRIB handle with a trimmed down data section. Since the values array is decoupled from the GRIB handle stored in the metadata object, accessing metadata keys related to the data values is forbidden. Getting metadata on these keys will throw an exception. "
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 13,
    "id": "c6fe87ed-ee88-4f4d-a2b6-9401b364e2df",
    "metadata": {
     "editable": true,
@@ -538,10 +601,10 @@
     {
      "data": {
       "text/plain": [
-       "<earthkit.data.readers.grib.metadata.RestrictedGribMetadata at 0x294e7dc60>"
+       "<earthkit.data.readers.grib.metadata.RestrictedGribMetadata at 0x2aba46ec0>"
       ]
      },
-     "execution_count": 10,
+     "execution_count": 13,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -552,7 +615,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 14,
    "id": "27686ac4-9382-4916-ad0e-be96a649d034",
    "metadata": {
     "editable": true,
@@ -568,7 +631,7 @@
        "'Wind speed'"
       ]
      },
-     "execution_count": 11,
+     "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -579,7 +642,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 15,
    "id": "dc28fa77-4020-431f-ad37-e480a69f9d7f",
    "metadata": {
     "editable": true,
@@ -620,7 +683,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 16,
    "id": "8eab3462-3661-4fc1-9d23-8be05dc99cd8",
    "metadata": {
     "editable": true,
@@ -636,7 +699,7 @@
        "7.450183054360252"
       ]
      },
-     "execution_count": 13,
+     "execution_count": 16,
      "metadata": {},
      "output_type": "execute_result"
     }

docs/examples/index.rst

@@ -172,6 +172,7 @@ Miscellaneous
     :glob:
 
     settings.ipynb
+    settings_env_vars.ipynb
     cache.ipynb
     projection.ipynb
     metadata.ipynb

docs/examples/settings.ipynb

@@ -72,7 +72,13 @@
    "cell_type": "code",
    "execution_count": 2,
    "id": "c5d47016-6f5b-4cbc-ac15-1d231546ac10",
-   "metadata": {},
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "settings.auto_save_settings = False"