Introduce SPORES in v0.7.0 (#716)

* Add working SPORES mode back into core code.
* Add `set_objective` as a backend method.
* Add spores scenarios to example models and add run mode comparison example notebook.
* (Compared to v0.6) add config option to point to another, boolean, parameter which will define which techs, and at which nodes, the SPORES scoring should be applied.
* (Compared to v0.6) add config option to choose from a selection of scoring algorithms.
* (Compared to v0.6) apply SPORES score objective as a new objective, rather than an update to the base cost minimisation objective.

---------

Co-authored-by: Bryn Pickering <17178478+brynpickering@users.noreply.github.com>

Author: FLomb

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ### User-facing changes
 
+|new| working SPORES mode, upgraded from v0.6 to include a selection of scoring algorithms.
+
+|new| backend `set_objective` method, to switch between pre-defined objectives.
+
 |changed| coin-or-cbc is now available cross-platform on conda-forge and so is the recommended open-source solver to install a user environment with.
 
 |changed| Upper bound pins for dependencies removed where possible, to minimise clashes when using calliope as a dependency in a project.

docs/advanced/mode.md

@@ -65,30 +65,19 @@ For this reason, `horizon` must always be equal to or larger than `window`.
 
 ## SPORES mode
 
-!!! warning
-    SPORES mode has not yet been re-implemented in Calliope v0.7.
-
 `SPORES` refers to Spatially-explicit Practically Optimal REsultS.
 This run mode allows a user to generate any number of alternative results which are within a certain range of the optimal cost.
 It follows on from previous work in the field of `modelling to generate alternatives` (MGA), with a particular emphasis on alternatives that vary maximally in the spatial dimension.
-This run mode was developed for and implemented in a [study on the future Italian energy system](https://doi.org/10.1016/j.joule.2020.08.002).
+This run mode was developed for and first implemented in a [study on the future Italian energy system](https://doi.org/10.1016/j.joule.2020.08.002). We later expanded it to enable greater [computational efficiency and versatility](https://doi.org/10.1016/j.apenergy.2023.121002) in what kind of alternative results are prioritised.
 
 As an example, if you wanted to generate 10 SPORES, all of which are within 10% of the optimal system cost, you would define the following in your model configuration:
 
 ```yaml
 config.build.mode: spores
-config.solve:
-    # The number of SPORES to generate:
-    spores_number: 10
-    # The cost class to optimise against when generating SPORES:
-    spores_score_cost_class: spores_score
-    # The initial system cost to limit the SPORES to fit within:
-    spores_cost_max: .inf
-    # The cost class to constrain to be less than or equal to `spores_cost_max`:
-    spores_slack_cost_group: monetary
-parameters:
-    # The fraction above the cost-optimal cost to set the maximum cost during SPORES:
-    slack: 0.1
+# The number of SPORES to generate:
+config.solve.spores.number: 10:
+# The fraction above the cost-optimal cost to set the maximum cost during SPORES:
+parameters.spores_slack: 0.1
 ```
 
 You will now also need a `spores_score` cost class in your model.
@@ -128,3 +117,6 @@ techs:
 !!! note
     We ourselves use and recommend using `spores_score` to define the cost class that you will now optimise against.
     However, it is user-defined, allowing you to choose any terminology that best fits your use case.
+
+To get a glimpse of how the results generated via SPORES compare to simple cost optimisation, check out our documentation
+on [comparing run modes](../examples/modes.py).

docs/examples/loading_tabular_data.py

@@ -1,7 +1,6 @@
 # ---
 # jupyter:
 #   jupytext:
-#     custom_cell_magics: kql
 #     text_representation:
 #       extension: .py
 #       format_name: percent

docs/examples/modes.py

@@ -0,0 +1,227 @@
+# ---
+# jupyter:
+#   jupytext:
+#     text_representation:
+#       extension: .py
+#       format_name: percent
+#       format_version: '1.3'
+#       jupytext_version: 1.16.4
+#   kernelspec:
+#     display_name: calliope_docs_build
+#     language: python
+#     name: calliope_docs_build
+# ---
+
+# %% [markdown]
+# # Running models in different modes
+#
+# Models can be built and solved in different modes:
+
+# - `plan` mode.
+# In `plan` mode, the user defines upper and lower boundaries for technology capacities and the model decides on an optimal system configuration.
+# In this configuration, the total cost of investing in technologies and then using them to meet demand in every _timestep_ (e.g., every hour) is as low as possible.
+# - `operate` mode.
+# In `operate` mode, all capacity constraints are fixed and the system is operated with a receding horizon control algorithm.
+# This is sometimes known as a `dispatch` model - we're only concerned with the _dispatch_ of technologies whose capacities are already fixed.
+# Optimisation is limited to a time horizon which
+# - `spores` mode.
+# `SPORES` refers to Spatially-explicit Practically Optimal REsultS.
+# This run mode allows a user to generate any number of alternative results which are within a certain range of the optimal cost.
+
+# In this notebook we will run the Calliope national scale example model in these three modes.
+
+# More detail on these modes is given in the [_advanced_ section of the Calliope documentation](https://calliope.readthedocs.io/en/latest/advanced/mode/).
+
+# %%
+
+import plotly.express as px
+import plotly.graph_objects as go
+import xarray as xr
+
+import calliope
+
+# We update logging to show a bit more information but to hide the solver output, which can be long.
+calliope.set_log_verbosity("INFO", include_solver_output=False)
+
+# %% [markdown]
+# ## Running in `plan` mode.
+
+# %%
+# We subset to the same time range as operate mode
+model_plan = calliope.examples.national_scale(time_subset=["2005-01-01", "2005-01-10"])
+model_plan.build()
+model_plan.solve()
+
+# %% [markdown]
+# ## Running in `operate` mode.
+
+# %%
+model_operate = calliope.examples.national_scale(scenario="operate")
+model_operate.build()
+model_operate.solve()
+
+# %% [markdown]
+# Note how we have capacity variables as parameters in the inputs and only dispatch variables in the results
+
+# %%
+model_operate.inputs[["flow_cap", "storage_cap", "area_use"]]
+
+# %%
+model_operate.results
+
+# %% [markdown]
+# ## Running in `spores` mode.
+
+# %%
+# We subset to the same time range as operate/plan mode
+model_spores = calliope.examples.national_scale(
+    scenario="spores", time_subset=["2005-01-01", "2005-01-10"]
+)
+model_spores.build()
+model_spores.solve()
+
+# %% [markdown]
+# Note how we have a new `spores` dimension in our results.
+
+# %%
+model_spores.results
+
+# %% [markdown]
+# We can track the SPORES scores used between iterations using the `spores_score_cumulative` result.
+# This scoring mechanism is based on increasing the score of any technology-node combination where the
+
+# %%
+# We do some prettification of the outputs
+model_spores.results.spores_score_cumulative.to_series().where(
+    lambda x: x > 0
+).dropna().unstack("spores")
+
+# %% [markdown]
+# ## Visualising results
+#
+# We can use [plotly](https://plotly.com/) to quickly examine our results.
+# These are just some examples of how to visualise Calliope data.
+
+# %%
+# We set the color mapping to use in all our plots by extracting the colors defined in the technology definitions of our model.
+# We also create some reusable plotting functions.
+colors = model_plan.inputs.color.to_series().to_dict()
+
+
+def plot_flows(results: xr.Dataset) -> go.Figure:
+    df_electricity = (
+        (results.flow_out.fillna(0) - results.flow_in.fillna(0))
+        .sel(carriers="power")
+        .sum("nodes")
+        .to_series()
+        .where(lambda x: x != 0)
+        .dropna()
+        .to_frame("Flow in/out (kWh)")
+        .reset_index()
+    )
+    df_electricity_demand = df_electricity[df_electricity.techs == "demand_power"]
+    df_electricity_other = df_electricity[df_electricity.techs != "demand_power"]
+
+    fig = px.bar(
+        df_electricity_other,
+        x="timesteps",
+        y="Flow in/out (kWh)",
+        color="techs",
+        color_discrete_map=colors,
+    )
+    fig.add_scatter(
+        x=df_electricity_demand.timesteps,
+        y=-1 * df_electricity_demand["Flow in/out (kWh)"],
+        marker_color="black",
+        name="demand",
+    )
+    return fig
+
+
+def plot_capacity(results: xr.Dataset, **plotly_kwargs) -> go.Figure:
+    df_capacity = (
+        results.flow_cap.where(results.techs != "demand_power")
+        .sel(carriers="power")
+        .to_series()
+        .where(lambda x: x != 0)
+        .dropna()
+        .to_frame("Flow capacity (kW)")
+        .reset_index()
+    )
+
+    fig = px.bar(
+        df_capacity,
+        x="nodes",
+        y="Flow capacity (kW)",
+        color="techs",
+        color_discrete_map=colors,
+        **plotly_kwargs,
+    )
+    return fig
+
+
+# %% [markdown]
+# ### Using different `spores` scoring algorithms.
+#
+# We make a number of scoring algorithms accessible out-of-the-box, based on those we present in [Lombardi et al. (2023)](https://doi.org/10.1016/j.apenergy.2023.121002).
+# You can call them on solving the model.
+# Here, we'll compare the result on `flow_cap` from running each.
+
+# %%
+# We subset to the same time range as operate/plan mode
+model_spores = calliope.examples.national_scale(
+    scenario="spores", time_subset=["2005-01-01", "2005-01-10"]
+)
+model_spores.build()
+
+spores_results = []
+for algorithm in ["integer", "evolving_average", "random", "relative_deployment"]:
+    model_spores.solve(**{"spores.scoring_algorithm": algorithm}, force=True)
+    spores_results.append(model_spores.results.expand_dims(algorithm=[algorithm]))
+
+spores_results_da = xr.concat(spores_results, dim="algorithm")
+
+spores_results_da.flow_cap.to_series().dropna().unstack("spores")
+
+# %% [markdown]
+# ## `plan` vs `operate`
+# Here, we compare flows over the 10 days.
+# Note how flows do not match as the rolling horizon makes it difficult to make the correct storage charge/discharge decisions.
+
+# %%
+fig_flows_plan = plot_flows(
+    model_plan.results.sel(timesteps=model_operate.results.timesteps)
+)
+fig_flows_plan.update_layout(title="Plan mode flows")
+
+
+# %%
+fig_flows_operate = plot_flows(model_operate.results)
+fig_flows_operate.update_layout(title="Operate mode flows")
+
+# %% [markdown]
+# ## `plan` vs `spores`
+# Here, we compare installed capacities between the baseline run (== `plan` mode) and the SPORES.
+# Note how the baseline SPORE is the same as `plan` mode and then results deviate considerably.
+
+# %%
+fig_flows_plan = plot_capacity(model_plan.results)
+fig_flows_plan.update_layout(title="Plan mode capacities")
+
+# %%
+fig_flows_spores = plot_capacity(model_spores.results, facet_col="spores")
+fig_flows_spores.update_layout(title="SPORES mode capacities")
+
+# %% [markdown]
+# ## Comparing `spores` scoring algorithms
+# Here, we compare installed capacities between the different SPORES runs.
+
+# %%
+fig_flows_spores = plot_capacity(
+    spores_results_da, facet_col="spores", facet_row="algorithm"
+)
+fig_flows_spores.update_layout(
+    title="SPORES mode capacities using different scoring algorithms",
+    autosize=False,
+    height=800,
+)

docs/examples/national_scale/notebook.py

@@ -109,7 +109,7 @@
 
 # %% [markdown]
 # #### Plotting flows
-# We do this by combinging in- and out-flows and separating demand from other technologies.
+# We do this by combining in- and out-flows and separating demand from other technologies.
 # First, we look at the aggregated result across all nodes, then we look at each node separately.
 
 # %%

docs/examples/piecewise_constraints.py

@@ -129,7 +129,7 @@
 """
 
 # %% [markdown]
-# # Building and checking the optimisation problem
+# # Building and checking the piecewise constraint
 #
 # With our piecewise constraint defined, we can build our optimisation problem and inject this new math.
 

docs/hooks/dummy_model/model.yaml

@@ -5,12 +5,19 @@ overrides:
       time_cluster: cluster_days.csv
     config.build:
       add_math: ["storage_inter_cluster"]
+  spores:
+    config:
+      init.name: SPORES solve mode
+      build.mode: spores
+      solve.spores.number: 2
+    parameters:
+      spores_slack: 0.1
 
 config.init.name: base
 
 nodes:
-  A.techs: {demand_tech, conversion_tech, supply_tech, storage_tech}
-  B.techs: {demand_tech, conversion_tech, supply_tech, storage_tech}
+  A.techs: { demand_tech, conversion_tech, supply_tech, storage_tech }
+  B.techs: { demand_tech, conversion_tech, supply_tech, storage_tech }
 
 techs:
   tech_transmission:

docs/hooks/generate_math_docs.py

@@ -67,7 +67,7 @@ def on_files(files: list, config: dict, **kwargs):
             f"{override}.yaml",
             textwrap.dedent(
                 f"""
-            Pre-defined additional math to apply {custom_documentation.name} math on top of the [base mathematical formulation][base-math].
+            Pre-defined additional math to apply {custom_documentation.name} __on top of__ the [base mathematical formulation][base-math].
             This math is _only_ applied if referenced in the `config.init.add_math` list as `{override}`.
             """
             ),

mkdocs.yml

@@ -132,6 +132,7 @@ nav:
       - examples/milp/index.md
       - examples/milp/notebook.py
     - examples/loading_tabular_data.py
+    - examples/modes.py
     - examples/piecewise_constraints.py
     - examples/calliope_model_object.py
     - examples/calliope_logging.py