Can not copy a GymEnv environment when using a LightSim2Grid backend

[EBoguslawski]

Environment

• Python version: 3.12
• Grid2op version: 1.10.5
• LightSim2Grid version: 0.9.2.post2
• System: ubuntu

Bug description

Hello,

I create a GymEnv environment from a grid2op environment. If I use a LightSimBackend backend when creating the grid2op environment, then I obtain an error if I try to copy the GymEnv environment with copy.deepcopy.
It works on an older environment with older versions (python 3.8, Grid2op 1.9.8, LightSim2Grid 0.7.0.post1)

Notes:

• In the l2rpn_baselines.PPO_SB3.utils.SB3Agent class, a copy.deepcopy is performed on the nn_kwargs argument during initialization, the latter containing a GymEnv in its keys / values (needed to train the agent). Consequently, it now raises this error.
• I cannot find any copy method for the GymEnv class
• I firstly put this issue on the Grid2op repo, but was asked to copy it here. So this issue is a replicate of this one: Can not copy a GymEnv environment when using a LightSim2Grid backend grid2op#672

How to reproduce

Code snippet

import grid2op
from grid2op.gym_compat import GymEnv
from lightsim2grid import LightSimBackend
import copy

env = grid2op.make("l2rpn_case14_sandbox", backend=LightSimBackend())
env_gym = GymEnv(env)

# This line raises an error
copy.deepcopy(env_gym)

Current output

/data/boguslawskieva/virtual_environments/venv_multi_agent/lib/python3.12/site-packages/lightsim2grid/gridmodel/from_pandapower/_aux_add_trafo.py:38: UserWarning: There were some Nan in the pp_net.trafo["tap_neutral"], they have been replaced by 0
  warnings.warn("There were some Nan in the pp_net.trafo[\"tap_neutral\"], they have been replaced by 0")
/data/boguslawskieva/virtual_environments/venv_multi_agent/lib/python3.12/site-packages/lightsim2grid/gridmodel/from_pandapower/_aux_add_trafo.py:46: UserWarning: There were some Nan in the pp_net.trafo["tap_step_percent"], they have been replaced by 0
  warnings.warn("There were some Nan in the pp_net.trafo[\"tap_step_percent\"], they have been replaced by 0")
/data/boguslawskieva/virtual_environments/venv_multi_agent/lib/python3.12/site-packages/lightsim2grid/gridmodel/from_pandapower/_aux_add_trafo.py:51: UserWarning: There were some Nan in the pp_net.trafo["tap_pos"], they have been replaced by 0
  warnings.warn("There were some Nan in the pp_net.trafo[\"tap_pos\"], they have been replaced by 0")
/data/boguslawskieva/virtual_environments/venv_multi_agent/lib/python3.12/site-packages/lightsim2grid/gridmodel/from_pandapower/_aux_add_trafo.py:70: UserWarning: There were some Nan in the pp_net.trafo["tap_step_degree"], they have been replaced by 0
  warnings.warn("There were some Nan in the pp_net.trafo[\"tap_step_degree\"], they have been replaced by 0")
/data/boguslawskieva/virtual_environments/venv_multi_agent/lib/python3.12/site-packages/lightsim2grid/gridmodel/from_pandapower/_aux_add_slack.py:56: UserWarning: LightSim will not consider the pandapower "ext_grid" as there are already generators tagged as slack bus
  warnings.warn("LightSim will not consider the pandapower \"ext_grid\" as there "
Traceback (most recent call last):
  File "/home/boguslawskieva/env_with_setpoint_new_implementation/test.py", line 200, in <module>
    copy.deepcopy(env_gym)
  File "/usr/lib/python3.12/copy.py", line 162, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 259, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 136, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 221, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 162, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 259, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 136, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 221, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 162, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 259, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 136, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 221, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/copy.py", line 143, in deepcopy
    y = copier(memo)
        ^^^^^^^^^^^^
  File "/data/boguslawskieva/virtual_environments/venv_multi_agent/lib/python3.12/site-packages/lightsim2grid/lightSimBackend.py", line 1418, in __deepcopy__
    result = self.copy()
             ^^^^^^^^^^^
  File "/data/boguslawskieva/virtual_environments/venv_multi_agent/lib/python3.12/site-packages/lightsim2grid/lightSimBackend.py", line 1531, in copy
    res._grid = mygrid.copy()
                ^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'copy'

[Borroot]

Hi,

I can confirm this issue for the following environment.

• Python version 3.12.9
• Grid2op version 1.10.5.post1
• LightSim2Grid version 0.10.2
• OS Arch

This issue has been open for about four months. I was wondering if there are any plans to work on it or if there is an estimated timeline for a fix. Any information would be greatly appreciated, as it would help me plan accordingly.

Thank you for your time and attention to this matter!

[BDonnot]

Hello,

Indeed grid2op and deepcopy does work really well together.

At the moment, grid2op uses the "env.copy()" which is rather well tested currently. I have no "bandwidth" at the moment to fix this. I'll see if I can reproduce and provide a quick fix.

[EBoguslawski]

Hello @Borroot ,

Do you try to initialize a l2rpn_baselines.PPO_SB3.utils.SB3Agent instance ? (see first item of my notes)
For this case, I proposed a fix to avoid copying the GymEnv in the PR Grid2op/l2rpn-baselines#70.

Have a good day

[Borroot]

Thank you both for your fast responses!

@EBoguslawski Thank you for pointing this out, however it concerns a modified version of the l2rpn_baselines.PPO_RLLIB.Env_RLLIB instance. I need to copy the environment so I can execute all actions on the same state. I do this to compute action similarity so I can mask similar actions.

@BDonnot Thank you for your response. I will use env.copy() for now to make a workaround. Note that in the provided example deepcopy also does not work for the regular environment, i.e.

import grid2op
from lightsim2grid import LightSimBackend
import copy

env = grid2op.make("l2rpn_case14_sandbox", backend=LightSimBackend())

# This line raises an error
copy.deepcopy(env_gym)

[BDonnot]

Hello,

I made a quick fix which will work correctly for lightsim2grid. But it's not efficient from grid2op perspective (it will un necessarily copy things that should not be).
Beside being slower, I don't see any major drawbacks.

I'll push the modifications on pypi tomorrow if all tests passed correctly. And you'll be able to install it with

pip install lightsim2grid==0.10.3.dev0

This fix will be part of next release of lightsim2grid

[Borroot]

Thank you @BDonnot, I will check it out tomorrow.

I know this is off-topic, and let me know if I should ask this question elsewhere, but do you have another recommendation on how to execute multiple actions in the same state? Currently I copy the environment, but since this inefficient like you said, another method would be preferred.

I know it is also possible using simulate(), but as I understand this is not equivalent to step() and a little less accurate.

[BDonnot]

Without a proper understanding, and depending on what you want exactly to do, there is:

• copy of the environment
• obs.simulate
• the "simulator" (obs.get_simulator() https://grid2op.readthedocs.io/en/latest/user/simulator.html)

All have pros and cons and depending on what you want to do it's more efficient to use any of the above.

Copy of the env is the most expensive and the things that is the less "reinforcement learning" as I don't think it's easy to fit it in a MDP framework.
And most importantly it's not usable for a "real" use case (it requires perfect knowledge of the future)

Obs.simulate is faster than the copy. The modeling is the same as the one in the environment. The major difference between obs.simulate and env copy is that obs.simulate does not assumer perfect knowledge of the future. It relies on provided forecast (part of the env and part of what the agent can observe)

The simulator is even more simple. In the sense it only provide an access to the... Power grid simulator. It does not assume anything about the future or whatever. It is only a simple object that can perform actions on a given grid state.
If I remember correctly, this is probably a little bit faster than obs.simulate (it should be at least)

[BDonnot]

Hello @Borroot , I just made the release on pypi, you can use it with:

pip install ightsim2grid==0.10.3.dev0

(as it is a development version, pypi will not automatically download it, so you have to add the ==0.10.3.dev0 in the pip command)
(when version 0.10.3 will be out you won't be forced to do that. I don't know when i'll have time to do it)

[Borroot]

Hi @BDonnot

Thank you so much for your quick action, I can confirm it works now!

Also thanks for elaborating on the options available, this helps!