Open62541 server (with rust bindings) crashes when Ignition subscribes for large number of variables (~250K variables) to monitor

[ndchandar]

Looking for inputs on how to triage this issue.

Our OPCUA server has ~250K variables and three levels of hierarchy (3 object folders + variables at the leaf level + smaller number of variables at various levels in hierarchy). The server (with rust bindings) itself loads the full hierarchy very quickly (< 7 seconds). Writes to variables from the server side also happen very quickly (in milliseconds) for a fairly decent write rate (~1K writes per second). I can verify through Node OPCUA commander that that couple of monitored value shows up ok.

However when I try to connect the server through Ignition (which creates a monitoring request for each of the ~250K variables), the write latency on the server side increases exponentially (sometimes > 100 seconds). Eventually the server goes unresponsive and it was not able to respond to clients. I took a flamegraph but nothing obvious popped and server memory/cpu looks ok.

I am currently using 0.7.0 and can update to 0.8.0 if required. I am wondering how we could triage this issue with the way Ignition works.

Regards
Dinesh Narayanan

[sgoll]

Interesting. I am unsure whether this could possibly be an issue of the Rust bindings: when issuing server-side write requests, all the remaining handling—internally updating the value, sending updates to all subscribed monitored items, etc.—happens inside the base library.

Can you create a reproduction in C, using the open62541 library directly, with a similarly large number of (dummy) nodes?

[ndchandar]

We tried simulating with open62541 C library and could reproduce the same issue when Ignition uses Subscriptions to monitor ~250K some variables. There is an option to use "Poll" in Ignition where the clients polls for new values periodically (E.g every 3 seconds) and we see that the server was mostly stable. But the latency for each write is on the higher side when Ignition polls periodically. When we poll/subscribe for only a handful set of variables (e.g. <50), there is no overhead at all. Any suggestions on how we could approach to understand this more.

Regards
Dinesh Narayanan

[sgoll]

As the issue seems to occur also when using open62541 directly, i.e. without the Rust bindings, then it is probably the library itself that reaches some kind of limit when managing this amount of subscriptions.

I am afraid there is not much we can do inside the bindings to mitigate the problem. I would suggest that you open an issue over in the open62541/open62541 repository. They will probably know what to do—maybe it is a known problem.