Add method to print a domain graph to JSON file

[maartendecat]

No description provided.

[jdeflander]

Is this functionality of the client that we want to expose and maintain? Or would it also be okay for now to deal with this in the agents that need it?

[maartendecat]

I think that this definitely is useful functionality for developing with the client. I also though about whether the client is the best place to put this and my reasoning for doing so was that the JSON output should be valid for transmitting to the server.

I would say that ideally this indeed is a method that we maintain in the client, but at a low cost (could it be a method that the _encode() uses itself maybe?).

What do you think?

[jdeflander]

What do you think?

If we add it to the client then we have to maintain it even if at some point we internally use a more efficient representation (to remain backwards compatible). If we are sure that we need the exact same logic in multiple agents, then it makes sense to add it here though.

[maartendecat]

If we add it to the client then we have to maintain it even if at some point we internally use a more efficient representation (to remain backwards compatible).

Good point, that's not what I was thinking off for this method. What would be useful for me is to have a method that can be used to store the domain graph in the format that will be posted to the backend later on. If that format changes, the output of this method can change as well. And if at some point we decide to completely change the format so that there is no intermediary data representation anymore, that's okay as well. I'm definitely not looking for yet another input/output/store-for-later format that we have to maintain.

Would it be an option to clarify this in the documentation of this method to manage the expectations of users?

[jdeflander]

What would be useful for me is to have a method that can be used to store the domain graph in the format that will be posted to the backend later on. If that format changes, the output of this method can change as well.

I think that would be useful, yes. Unless we bump the major version of our backend itself, the endpoint for reloading domain graphs should remain backwards compatible anyway (in theory). If at some point we no longer wish to maintain a deprecated format in this client, I would just bump the major version.

Would it be an option to clarify this in the documentation of this method to manage the expectations of users?

I would just bump our major version if at some point we want to change the format. We should maintain a compatibility table with regard to our backend anyway.

[jdeflander]

As an alternative, you can also save the DomainGraph object itself to a file (using pickle, for example). I think @GillesVanc has used a similar strategy in the past.

[jdeflander]

After thinking about it a bit more, isn't it even simpler to just use pickle in the agents for this use case? Then we don't need an additional method in the client and it's easy to parse back into a DomainGraph and send it with the client.

[maartendecat]

Going from the domain graph object to dict to JSON to the server takes 10min at NMBS scale. My goal was to avoid repeating those 10min when something went wrong. So ideally the output format is as close to the API as possible.

[jdeflander]

Going from the domain graph object to dict to JSON to the server takes 10min at NMBS scale. My goal was to avoid repeating those 10min when something went wrong. So ideally the output format is as close to the API as possible.

I understand, but what part of those 10 minutes is actually spent on converting the domain graph object to the dict? After that step, the dict is encoded and sent to the server in chunks, which hopefully doesn't take much more time than sending the JSON directly (?). Anyway, if we add a method to output a domain graph JSON, shouldn't we also add a method to compress and send it in chunks? I assume we would need that logic as well in multiple agents.

[jdeflander]

We should definitely come back to if it turns out that printing a domain graph to JSON is still relevant even if it is streamed as described in #37. At that point we should also consider whether our Python client should expose development utilities or just an API for production agents.