Add config object to keep config in sync at all times

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ### User-facing changes
 
+|changed| to ensure the model configuration always remains in sync with the results, `kwargs` in `model.build()` and `model.solve()` now directly affect `model.config`
+
 |changed| `template:` can now be used anywhere within YAML definition files, not just in the `nodes`, `techs` and `data_tables` sections.
 
 |changed| "An overview of the Calliope terminology" information admonition to remove self-references and improve understandability.
@@ -33,6 +35,8 @@ This change has occurred to avoid confusion between data "sources" and model ene
 
 ### Internal changes
 
+|changed| Model configuration now uses `pydantic`.
+
 |changed| Model definition reading is now defined in a single place (preprocess/model_definition.py).
 
 |changed| Moved YAML reading/importing functionality out of `AttrDict`. It is now part of our `io` functionality.

docs/creating/config.md

@@ -83,7 +83,7 @@ In `plan` mode, capacities are determined by the model, whereas in `operate` mod
 In `spores` mode, the model is first run in `plan` mode, then run `N` number of times to find alternative system configurations with similar monetary cost, but maximally different choice of technology capacity and location (node).
 
 In most cases, you will want to use the `plan` mode.
-In fact, you can use a set of results from using `plan` model to initialise both the `operate` (`config.build.operate_use_cap_results`) and `spores` modes.
+In fact, you can use a set of results from using `plan` model to initialise both the `operate` (`config.build.operate.use_cap_results`) and `spores` modes.
 
 ### `config.solve.solver`
 

docs/creating/parameters.md

@@ -52,7 +52,7 @@ Which will add the new dimension `my_new_dim` to your model: `model.inputs.my_ne
 `foreach: [my_new_dim]`.
 
 !!! warning
-    The `parameter` section should not be used for large datasets (e.g., indexing over the time dimension) as it will have a high memory overhead on loading the data.
+    The `parameter` section should not be used for large datasets (e.g., indexing over the time dimension) as it will have a high memory overhead when loading the data.
 
 ## Broadcasting data along indexed dimensions
 

docs/creating/scenarios.md

@@ -37,33 +37,6 @@ Scenarios consist of a name and a list of override names which together form tha
 Scenarios and overrides can be used to generate scripts that run a single Calliope model many times, either sequentially, or in parallel on a high-performance cluster
 (see the section on [generating scripts to repeatedly run variations of a model](../advanced/scripts.md)).
 
-## Importing other YAML files in overrides
-
-When using overrides, it is possible to have [`import` statements](yaml.md#relative-file-imports) for more flexibility.
-This can be useful if many overrides are defined which share large parts of model configuration, such as different levels of interconnection between model zones
-The following example illustrates this:
-
-```yaml
-overrides:
-  some_override:
-    techs:
-      some_tech.constraints.flow_cap_max: 10
-    import: [additional_definitions.yaml]
-```
-
-`additional_definitions.yaml`:
-
-```yaml
-techs:
-  some_other_tech.constraints.flow_out_eff: 0.1
-```
-
-This is equivalent to the following override:
-
-```yaml
-overrides:
-  some_override:
-    techs:
-      some_tech.constraints.flow_cap_max: 10
-      some_other_tech.constraints.flow_out_eff: 0.1
-```
+???+ warning
+    Overrides are executed _after_ `imports:` but _before_ `templates:`.
+    This means it is possible to override template values, but not the files imported in your model definition.

docs/hooks/generate_readable_schema.py

@@ -14,12 +14,13 @@
 import jsonschema2md
 from mkdocs.structure.files import File
 
+from calliope import config
 from calliope.util import schema
 
 TEMPDIR = tempfile.TemporaryDirectory()
 
 SCHEMAS = {
-    "config_schema": schema.CONFIG_SCHEMA,
+    "config_schema": config.CalliopeConfig().model_no_ref_schema(),
     "model_schema": schema.MODEL_SCHEMA,
     "math_schema": schema.MATH_SCHEMA,
     "data_table_schema": schema.DATA_TABLE_SCHEMA,

docs/migrating.md

@@ -340,9 +340,9 @@ Along with [changing the YAML hierarchy of model configuration](#model-and-run-
 
 * `model.subset_time` → `config.init.time_subset`
 * `model.time: {function: resample, function_options: {'resolution': '6H'}}` → `config.init.time_resample`
-* `run.operation.window` → `config.build.operate_window`
-* `run.operation.horizon` → `config.build.operate_horizon`
-* `run.operation.use_cap_results` → `config.build.operate_use_cap_results`
+* `run.operation.window` → `config.build.operate.window`
+* `run.operation.horizon` → `config.build.operate.horizon`
+* `run.operation.use_cap_results` → `config.build.operate.use_cap_results`
 
 We have also moved some _data_ out of the configuration and into the [top-level `parameters` section](creating/parameters.md):
 
@@ -516,8 +516,8 @@ Therefore, `24H` is equivalent to `24` in v0.6 if you are using hourly resolutio
           init:
             time_resample: 6H
           build:
-            operate_window: 12H
-            operate_horizon: 24H
+            operate.window: 12H
+            operate.horizon: 24H
         ```
 
 !!! warning

docs/running.md

@@ -25,7 +25,7 @@ The `calliope run` command takes the following options:
 * `--scenario={scenario}` and `--override_dict={yaml_string}`: Specify a scenario, or one or several overrides, to apply to the model, or apply specific overrides from a YAML string (see below for more information).
 * `--help`: Show all available options.
 
-Multiple options can be specified, for example, saving NetCDF, CSV, and HTML plots simultaneously.
+Multiple options can be specified, for example, saving NetCDF and CSV simultaneously.
 
 ```shell
 $ calliope run testmodel/model.yaml --save_netcdf=results.nc --save_csv=outputs

requirements/base.txt

@@ -4,6 +4,7 @@ geographiclib >= 2, < 3
 ipdb >= 0.13, < 0.14
 ipykernel < 7
 jinja2 >= 3, < 4
+jsonref >= 1.1, < 2
 jsonschema >= 4, < 5
 natsort >= 8, < 9
 netcdf4 >= 1.2, < 1.7
@@ -14,3 +15,4 @@ pyparsing >= 3.0, < 3.1
 ruamel.yaml >= 0.18, < 0.19
 typing-extensions >= 4, < 5
 xarray >= 2024.1, < 2024.4
+pydantic >= 2.9.2

src/calliope/attrdict.py

@@ -232,7 +232,6 @@ def union(
             KeyError: `other` has an already defined key and `allow_override == False`
         """
         if not isinstance(other, AttrDict):
-            # FIXME-yaml: remove AttrDict wrapping in uses of this function.
             other = AttrDict(other)
         self_keys = self.keys_nested()
         other_keys = other.keys_nested()

src/calliope/backend/__init__.py

@@ -15,30 +15,30 @@
 from calliope.preprocess import CalliopeMath
 
 if TYPE_CHECKING:
+    from calliope import config
     from calliope.backend.backend_model import BackendModel
 
 
 def get_model_backend(
-    name: str, data: xr.Dataset, math: CalliopeMath, **kwargs
+    build_config: "config.Build", data: xr.Dataset, math: CalliopeMath
 ) -> "BackendModel":
     """Assign a backend using the given configuration.
 
     Args:
-        name (str): name of the backend to use.
+        build_config: Build configuration options.
         data (Dataset): model data for the backend.
         math (CalliopeMath): Calliope math.
-        **kwargs: backend keyword arguments corresponding to model.config.build.
 
     Raises:
         exceptions.BackendError: If invalid backend was requested.
 
     Returns:
         BackendModel: Initialized backend object.
     """
-    match name:
+    match build_config.backend:
         case "pyomo":
-            return PyomoBackendModel(data, math, **kwargs)
+            return PyomoBackendModel(data, math, build_config)
         case "gurobi":
-            return GurobiBackendModel(data, math, **kwargs)
+            return GurobiBackendModel(data, math, build_config)
         case _:
-            raise BackendError(f"Incorrect backend '{name}' requested.")
+            raise BackendError(f"Incorrect backend '{build_config.backend}' requested.")

src/calliope/backend/backend_model.py

@@ -26,17 +26,13 @@
 import numpy as np
 import xarray as xr
 
-from calliope import exceptions
+from calliope import config, exceptions
 from calliope.attrdict import AttrDict
 from calliope.backend import helper_functions, parsing
 from calliope.exceptions import warn as model_warn
 from calliope.io import load_config, to_yaml
 from calliope.preprocess.model_math import ORDERED_COMPONENTS_T, CalliopeMath
-from calliope.util.schema import (
-    MODEL_SCHEMA,
-    extract_from_schema,
-    update_then_validate_config,
-)
+from calliope.util.schema import MODEL_SCHEMA, extract_from_schema
 
 if TYPE_CHECKING:
     from calliope.backend.parsing import T as Tp
@@ -69,20 +65,20 @@ class BackendModelGenerator(ABC):
     _PARAM_UNITS = extract_from_schema(MODEL_SCHEMA, "x-unit")
     _PARAM_TYPE = extract_from_schema(MODEL_SCHEMA, "x-type")
 
-    def __init__(self, inputs: xr.Dataset, math: CalliopeMath, **kwargs):
+    def __init__(
+        self, inputs: xr.Dataset, math: CalliopeMath, build_config: config.Build
+    ):
         """Abstract base class to build a representation of the optimisation problem.
 
         Args:
             inputs (xr.Dataset): Calliope model data.
             math (CalliopeMath): Calliope math.
-            **kwargs (Any): build configuration overrides.
+            build_config: Build configuration options.
         """
         self._dataset = xr.Dataset()
         self.inputs = inputs.copy()
         self.inputs.attrs = deepcopy(inputs.attrs)
-        self.inputs.attrs["config"]["build"] = update_then_validate_config(
-            "build", self.inputs.attrs["config"], **kwargs
-        )
+        self.config = build_config
         self.math: CalliopeMath = deepcopy(math)
         self._solve_logger = logging.getLogger(__name__ + ".<solve>")
 
@@ -200,6 +196,7 @@ def _check_inputs(self):
             "equation_name": "",
             "backend_interface": self,
             "input_data": self.inputs,
+            "build_config": self.config,
             "helper_functions": helper_functions._registry["where"],
             "apply_where": True,
             "references": set(),
@@ -246,7 +243,7 @@ def add_optimisation_components(self) -> None:
         # The order of adding components matters!
         # 1. Variables, 2. Global Expressions, 3. Constraints, 4. Objectives
         self._add_all_inputs_as_parameters()
-        if self.inputs.attrs["config"]["build"]["pre_validate_math_strings"]:
+        if self.config.pre_validate_math_strings:
             self._validate_math_string_parsing()
         for components in typing.get_args(ORDERED_COMPONENTS_T):
             component = components.removesuffix("s")
@@ -399,7 +396,7 @@ def _add_all_inputs_as_parameters(self) -> None:
             if param_name in self.parameters.keys():
                 continue
             elif (
-                self.inputs.attrs["config"]["build"]["mode"] != "operate"
+                self.config.mode != "operate"
                 and param_name
                 in extract_from_schema(MODEL_SCHEMA, "x-operate-param").keys()
             ):
@@ -606,17 +603,21 @@ class BackendModel(BackendModelGenerator, Generic[T]):
     """Calliope's backend model functionality."""
 
     def __init__(
-        self, inputs: xr.Dataset, math: CalliopeMath, instance: T, **kwargs
+        self,
+        inputs: xr.Dataset,
+        math: CalliopeMath,
+        build_config: config.Build,
+        instance: T,
     ) -> None:
         """Abstract base class to build backend models that interface with solvers.
 
         Args:
             inputs (xr.Dataset): Calliope model data.
             math (CalliopeMath): Calliope math.
             instance (T): Interface model instance.
-            **kwargs: build configuration overrides.
+            build_config: Build configuration options.
         """
-        super().__init__(inputs, math, **kwargs)
+        super().__init__(inputs, math, build_config)
         self._instance = instance
         self.shadow_prices: ShadowPrices
         self._has_verbose_strings: bool = False

src/calliope/backend/gurobi_backend_model.py

@@ -14,6 +14,7 @@
 import pandas as pd
 import xarray as xr
 
+from calliope import config
 from calliope.backend import backend_model, parsing
 from calliope.exceptions import BackendError, BackendWarning
 from calliope.exceptions import warn as model_warn
@@ -41,19 +42,21 @@
 class GurobiBackendModel(backend_model.BackendModel):
     """gurobipy-specific backend functionality."""
 
-    def __init__(self, inputs: xr.Dataset, math: CalliopeMath, **kwargs) -> None:
+    def __init__(
+        self, inputs: xr.Dataset, math: CalliopeMath, build_config: config.Build
+    ) -> None:
         """Gurobi solver interface class.
 
         Args:
             inputs (xr.Dataset): Calliope model data.
             math (CalliopeMath): Calliope math.
-            **kwargs: passed directly to the solver.
+            build_config: Build configuration options.
         """
         if importlib.util.find_spec("gurobipy") is None:
             raise ImportError(
                 "Install the `gurobipy` package to build the optimisation problem with the Gurobi backend."
             )
-        super().__init__(inputs, math, gurobipy.Model(), **kwargs)
+        super().__init__(inputs, math, build_config, gurobipy.Model())
         self._instance: gurobipy.Model
         self.shadow_prices = GurobiShadowPrices(self)
 
@@ -144,7 +147,7 @@ def _objective_setter(
         ) -> xr.DataArray:
             expr = element.evaluate_expression(self, references=references)
 
-            if name == self.inputs.attrs["config"].build.objective:
+            if name == self.config.objective:
                 self._instance.setObjective(expr.item(), sense=sense)
 
                 self.log("objectives", name, "Objective activated.")

src/calliope/backend/latex_backend_model.py

@@ -12,6 +12,7 @@
 import pandas as pd
 import xarray as xr
 
+from calliope import config
 from calliope.backend import backend_model, parsing
 from calliope.exceptions import ModelError
 from calliope.preprocess import CalliopeMath
@@ -305,19 +306,19 @@ def __init__(
         self,
         inputs: xr.Dataset,
         math: CalliopeMath,
+        build_config: config.Build,
         include: Literal["all", "valid"] = "all",
-        **kwargs,
     ) -> None:
         """Interface to build a string representation of the mathematical formulation using LaTeX math notation.
 
         Args:
             inputs (xr.Dataset): model data.
             math (CalliopeMath): Calliope math.
+            build_config: Build configuration options.
             include (Literal["all", "valid"], optional):
                 Defines whether to include all possible math equations ("all") or only those for which at least one index item in the "where" string is valid ("valid"). Defaults to "all".
-            **kwargs: for the backend model generator.
         """
-        super().__init__(inputs, math, **kwargs)
+        super().__init__(inputs, math, build_config)
         self.include = include
 
     def add_parameter(  # noqa: D102, override

src/calliope/backend/parsing.py

@@ -311,6 +311,7 @@ def evaluate_where(
                 helper_functions=helper_functions._registry["where"],
                 input_data=backend_interface.inputs,
                 backend_interface=backend_interface,
+                build_config=backend_interface.config,
                 references=references if references is not None else set(),
                 apply_where=True,
             )

src/calliope/backend/pyomo_backend_model.py

@@ -26,6 +26,7 @@
 from pyomo.opt import SolverFactory  # type: ignore
 from pyomo.util.model_size import build_model_size_report  # type: ignore
 
+from calliope import config
 from calliope.exceptions import BackendError, BackendWarning
 from calliope.exceptions import warn as model_warn
 from calliope.preprocess import CalliopeMath
@@ -58,15 +59,17 @@
 class PyomoBackendModel(backend_model.BackendModel):
     """Pyomo-specific backend functionality."""
 
-    def __init__(self, inputs: xr.Dataset, math: CalliopeMath, **kwargs) -> None:
+    def __init__(
+        self, inputs: xr.Dataset, math: CalliopeMath, build_config: config.Build
+    ) -> None:
         """Pyomo solver interface class.
 
         Args:
             inputs (xr.Dataset): Calliope model data.
             math (CalliopeMath): Calliope math.
-            **kwargs: passed directly to the solver.
+            build_config: Build configuration options.
         """
-        super().__init__(inputs, math, pmo.block(), **kwargs)
+        super().__init__(inputs, math, build_config, pmo.block())
 
         self._instance.parameters = pmo.parameter_dict()
         self._instance.variables = pmo.variable_dict()
@@ -185,7 +188,7 @@ def _objective_setter(
         ) -> xr.DataArray:
             expr = element.evaluate_expression(self, references=references)
             objective = pmo.objective(expr.item(), sense=sense)
-            if name == self.inputs.attrs["config"].build.objective:
+            if name == self.config.objective:
                 text = "activated"
                 objective.activate()
             else: