Add switch graph (#45)

* Fix bug in storage fixed cost calculations

In versions 2.0.0b3-2.0.5, fixed costs from all possible build years
were applied each period, instead of only using the build years that are
still in service in the current period. This increased the apparent cost
of storage by approximately (study length) / (storage life). This commit
fixes that bug.

* update version number to 2.0.6

* prevent SyntaxWarning about using `is` with literal in Python 3.8+

* minor cosmetic changes

* install only compatible versions of Pyomo and pyutilib

* update CHANGELOG.md

* Add basic graphing framework

* Plot graphs side by side

* Add subtitles, comments and refactor

* Add graph for generation build

* Improve graphing and generation

* Improve carbon_policies graphing and refactor graphing framework to include compare() option

* Fix compare mode

* Refactor + make graphs gray

* Split into switch graph and switch compare

* Start adding planning reserves

* Make planning reserves optional

* Minor improvements

* Create reserve capacity value csv

* Allow negatives and values greater than 1

* Create outline of documentation

* Start writing overview and Usage.md

* Add documentation

* Define technology type based on file

* Add gen_buildout_per_tech.png

* Add note on installation extras

* Start creating dispatch plots

* Improve dispatch and build-out graph

* Add plot for dispatched electricity

* Make graphing run automatically after solving

* Fix circular import

* Add documentation

* Add costs graphs

* Improve performance slightly

* Fix setup.py

* Migrate to pyomo 6.0.0

* Fix two more warnings

* Improve performance

* Make plotting packages mandatory

* Properly print error messages

* Make graphs work with non-wecc setup

* Make --recommend unambigious

* Add the timepoint weights to the outputs

* Create v1 of duals plot

* Fix errors in gen_buildout_per_tech plot

* Add curtailment plots

* Catch errors during saving

* Add daily matrix plots

* Fix dual values and its graph

* Sort plots

* Fix compare

* Add example to tool usage

* Small improvements based on PR review

* Give a default to output_dir

* Improve documentation

* Add transmission capacity graph

* Fix bug in matrix plots

* Improve CA policies plots

* Update CA policies costs since wasn't correct when first added

* Improve plotting for examples

docs/Graphs.md

@@ -10,37 +10,105 @@ graphs. For more options, run `switch graph --help`.
 
 ## Using `switch compare`
 
-Run `switch compare <output_folder> <path_to_scenario_1> <path_to_scenario_2> <path_to_scenario_3> ...`.
-This will create your `output_folder` and fill it with comparison graphs.
+Run `switch compare <path_to_scenario_1> <path_to_scenario_2> <path_to_scenario_3> ...`. This will create a folder with
+comparison graphs. For example, if you run
+`switch compare storage ca_policies` while in the `/examples` folder, a new folder
+called `compare_storage_to_ca_policies` will be created with comparison graphs (this assumes both `storage`
+and `ca_policies` have already been solved).
 
 ## Adding new graphs
 
 Graphs can be defined in any module by adding the following function to the file.
 
 ```python
 def graph(tools):
-    # Your graphing code
+  # Your graphing code
+  ...
 ```
 
 In `graph()` you can use the `tools` object to create graphs. Here are some important methods.
 
-- `tools.get_dataframe(csv=filename)` will return a pandas dataframe for the file called `filename`.
-You can also specify `folder=tools.folders.INPUTS` to load a csv from the inputs directory.
-
-- `tools.get_new_axes(out, title, note)` will return a matplotlib axes. This should
-be the axes used while graphing. `out` is the name of the `.png` file that will be created
-  with this graph. `title` and `note` are optional and will be the title and footnote for the graph.
-
-- `tools.pd`, `tools.sns`, `tools.np`, `tools.mplt` are references to the pandas, seaborn, numpy and matplotlib libraries.
-This is useful if your graphing code needs to access these libraries since it doesn't require adding an import to your file.
-
-- `tools.add_gen_type_column(df)` add a column called `gen_type` to a dataframe with columns
-`gen_tech` and `gen_energy_source`. `gen_type` is a user-friendly name for the technology (e.g. Nuclear instead of Uranium).
-  The mapping of `gen_energy_source` and `gen_tech` to `gen_type` is defined in a `.csv` file in
-  `switch_model.tools.graphing`. You can add more mappings with a different `map_name` and then
-  use those mappings by specifying `map_name=` when calling `add_gen_type_colum()`.
-
-- `tools.get_colors()` returns a mapping of `gen_type` to its color. This is useful for graphing
-and can normally be passed straight to `color=` in standard plotting libraries. You can also
-  specify a different color mapping using a similar process to above (`map_name=`)
-
+- `tools.get_dataframe(csv=filename)` will return a pandas dataframe for the file called `filename`. You can also
+  specify `folder=tools.folders.INPUTS` to load a csv from the inputs directory.
+
+- `tools.get_new_axes(out, title, note)` will return a matplotlib axes. This should be the axes used while
+  graphing. `out` is the name of the `.png` file that will be created with this graph. `title` and `note` are optional
+  and will be the title and footnote for the graph.
+
+- `tools.pd`, `tools.sns`, `tools.np`, `tools.mplt` are references to the pandas, seaborn, numpy and matplotlib
+  libraries. This is useful if your graphing code needs to access these libraries since it doesn't require adding an
+  import to your file.
+
+- `tools.add_gen_type_column(df)` adds a column called `gen_type` to a dataframe with columns
+  `gen_tech` and `gen_energy_source`. `gen_type` is a user-friendly name for the technology (e.g. Nuclear instead of
+  Uranium). The mapping of `gen_energy_source` and `gen_tech` to `gen_type` is defined in
+  a `inputs/graph_tech_types.csv`. If this file isn't present, a default mapping will be used. You can also use other
+  mappings found in `graph_tech_types.csv` by specifying `map_name=` when calling `add_gen_type_column()`.
+
+- `tools.get_colors()` returns a mapping of `gen_type` to its color. This is useful for graphing and can normally be
+  passed straight to `color=` in standard plotting libraries. You can also specify a different color mapping using a
+  similar process to above (`map_name=`)
+
+## Adding a comparison graph
+
+By default, `tools.get_dataframe` will return the data for only one scenario (the one you are graphing).
+
+Sometimes, you may wish to create a graph that compares multiple scenarios. To do this create a function
+called `compare`.
+
+```python
+def compare(tools):
+  # Your graphing code
+  ...
+```
+
+If you call `tools.get_dataframe(...)` from within `compare`, then
+`tools.get_dataframe` will return a dataframe containing the data from *all*
+the scenarios. The dataframe will contain a column called `scenario` to indicate which rows correspond to which
+scenarios. You can then use this column to create a graph comparing the different scenarios (still
+using `tools.get_new_axes`).
+
+At this point, when you run `switch compare`, your `compare(tools)` function will be called and your comparison graph
+will be generated.
+
+## Example
+
+In this example we create a graph that shows the power capacity during each period broken down by technology.
+
+```python
+def graph(tools):
+  # Get a dataframe of gen_cap.csv
+  gen_cap = tools.get_dataframe(csv="gen_cap")
+
+  # Add a 'gen_type' column to your dataframe
+  gen_cap = tools.add_gen_type_column(gen_cap)
+
+  # Aggregate the generation capacity by gen_type and PERIOD
+  capacity_df = gen_cap.pivot_table(
+    index='PERIOD',
+    columns='gen_type',
+    values='GenCapacity',
+    aggfunc=tools.np.sum,
+    fill_value=0  # Missing values become 0
+  )
+
+  # Get a new pair of axis to plot onto
+  ax = tools.get_new_axes(out="capacity_per_period")
+
+  # Plot
+  capacity_df.plot(
+    kind='bar',
+    ax=ax, # Notice we pass in the axis
+    stacked=True,
+    ylabel="Capacity Online (MW)",
+    xlabel="Period",
+    color=tools.get_colors(len(capacity_df.index))
+  )
+```
+
+Running `switch graph` would run the `graph()` function above and create 
+`capacity_per_period.png` containing your plot.
+
+Running `switch compare` would create `capacity_per_period.png` containing
+your plot side-by-side with the same plot but for the scenario you're comparing to.
+

examples/ca_policies/outputs/total_cost.txt

@@ -1 +1 @@
-139632004.93414935
+136941837.16

switch_model/balancing/load_zones.py

@@ -310,13 +310,18 @@ def graph(tools):
         "energy_balance_duals", title="Energy balance duals per period"
     )
     load_balance["energy_balance_duals"] = (
-        load_balance["normalized_energy_balance_duals_dollar_per_mwh"] / 10
+        tools.pd.to_numeric(
+            load_balance["normalized_energy_balance_duals_dollar_per_mwh"],
+            errors="coerce",
+        )
+        / 10
     )
     load_balance = load_balance[["energy_balance_duals", "time_row"]]
     load_balance = load_balance.pivot(columns="time_row", values="energy_balance_duals")
-    load_balance.plot.box(
-        ax=ax,
-        xlabel="Period",
-        ylabel="Energy balance duals (cents/kWh)",
-        logy=True,
-    )
+    if load_balance.count().sum() != 0:
+        load_balance.plot.box(
+            ax=ax,
+            xlabel="Period",
+            ylabel="Energy balance duals (cents/kWh)",
+            logy=True,
+        )

switch_model/generators/core/dispatch.py

@@ -673,7 +673,9 @@ def graph_curtailment_per_tech(tools):
     df["Percent Curtailed"] = df["Curtailment_MW"] / (
         df["DispatchGen_MW"] + df["Curtailment_MW"]
     )
-    df = df.pivot_table(index="period", columns="gen_type", values="Percent Curtailed")
+    df = df.pivot(
+        index="period", columns="gen_type", values="Percent Curtailed"
+    ).fillna(0)
     if len(df) == 0:  # No dispatch from renewable technologies
         return
     # Set the name of the legend.

switch_model/policies/CA_policies.py

@@ -237,6 +237,9 @@ def post_solve(model, outdir):
 
 def graph(tools):
     # Plot emissions over time
-    ax = tools.get_new_axes(out="emissions_CA")
+    ax = tools.get_new_axes(out="emissions_CA", title="California's Total Emissions")
     df = tools.get_dataframe(csv="ca_policies")
-    tools.sns.barplot(x="PERIOD", y="AnnualEmissions_tCO2_per_yr_CA", data=df, ax=ax)
+    df = df.set_index("PERIOD")
+    df["AnnualEmissions_tCO2_per_yr_CA"].plot(
+        ax=ax, kind="bar", ylabel="Annual Emissions (tCO2)", xlabel="Year"
+    )

switch_model/reporting/__init__.py

@@ -81,11 +81,12 @@ def format_row(row, sig_digits):
     return tuple(cell_formatter(get_value(v)) for v in row)
 
 
-def write_table(instance, *indexes, **kwargs):
+def write_table(instance, *indexes, output_file=None, **kwargs):
     # there must be a way to accept specific named keyword arguments and
     # also an open-ended list of positional arguments (*indexes), but I
     # don't know what that is.
-    output_file = kwargs.pop("output_file")
+    if output_file is None:
+        raise Exception("Must specify output_file in write_table()")
     digits = instance.options.sig_figs_output
 
     if "df" in kwargs:

switch_model/tools/graphing/compare.py

@@ -1,12 +1,12 @@
-import argparse, os
-from switch_model.tools.graphing.main import Scenario, graph_scenarios
-from switch_model.utilities import query_yes_no
-
 """
 Tool to generate graphs that compare multiple scenario outputs.
 Run 'switch compare -h' for details.
 """
 
+import argparse, os
+from switch_model.tools.graphing.main import Scenario, graph_scenarios
+from switch_model.utilities import query_yes_no
+
 
 def main():
     # Create the command line interface
@@ -19,12 +19,13 @@ def main():
         formatter_class=argparse.RawTextHelpFormatter,
     )
     parser.add_argument(
-        "graph_dir",
-        type=str,
-        help="Name of the folder where the graphs should be saved",
+        "scenarios", nargs="+", help="Specify a list of runs to compare"
     )
     parser.add_argument(
-        "scenarios", nargs="+", help="Specify a list of runs to compare"
+        "--graph-dir",
+        type=str,
+        default=None,
+        help="Name of the folder where the graphs should be saved",
     )
     parser.add_argument(
         "--overwrite",
@@ -45,7 +46,7 @@ def main():
 
     # If names is not set, make the names the scenario path
     if args.names is None:
-        args.names = args.scenarios
+        args.names = list(map(lambda p: os.path.normpath(p), args.scenarios))
         print(
             "NOTE: For better graphs, use the flag '--names' to specify descriptive scenario names (e.g. baseline)"
         )
@@ -56,6 +57,10 @@ def main():
                 f"Gave {len(args.names)} scenario names but there were {len(args.scenarios)} scenarios."
             )
 
+    # If graph_dir is not set, make it 'compare_<name_1>_to_<name2>_to_<name3>...'
+    if args.graph_dir is None:
+        args.graph_dir = f"compare_{'_to_'.join(args.names)}"
+
     # Create a list of Scenario objects for each scenario
     scenarios = [
         Scenario(rel_path, args.names[i]) for i, rel_path in enumerate(args.scenarios)

switch_model/tools/graphing/graph.py

@@ -1,3 +1,9 @@
+"""
+Tool to generate graphs for a scenario.
+
+Run "switch graph -h" for details.
+"""
+
 import os, argparse
 
 from switch_model.utilities import query_yes_no