Delayed update after OrchestratorUpdateCallback parse error

[D-r-P-3-p-p-3-r]

Expected Behavior

After an update is deployed it starts more or less immediately on a device.

Current Behavior

In about 20-25% of the cases, a parse error message appears in the log. If that happens, the update is delayed by 5+ minutes.

Device Information

• Architecture: arm32
• DU Agent Version: 1.1

Logs

Update deployed:

2024-10-28T12:08:43.6769Z 926[926] [D] ComponentName:deviceUpdate, propertyName:service [ADUC_PnP_ComponentClient_PropertyUpdate_Callback:536]
2024-10-28T12:08:43.6774Z 926[926] [D] Update Action info string ({"workflow":{"action":255,"id":"nodeployment"},"updateManifestSignature":null,"fileUrls":null}), property version (39) [OrchestratorUpdateCallback:433]
2024-10-28T12:08:43.6774Z 926[926] [E] Parse failed for unprotected properties, erc: 0x80300011 [OrchestratorUpdateCallback:439]
2024-10-28T12:08:43.6776Z 926[926] [I] OrchestratorPropertyUpdateCallback ended [OrchestratorUpdateCallback:517]

Update starts:

2024-10-28T12:14:13.8827Z 926[926] [D] ComponentName:deviceUpdate, propertyName:service [ADUC_PnP_ComponentClient_PropertyUpdate_Callback:536]

[jw-msft]

The agent for "nodeployment" workflow id for which it should instead ignore the normally required signature and file urls fields. As for the 5-min delay, @josephmsft would you know if it's by design to have delays on the order of minutes where the svc first sends nodeployment workflow id before pushing C2D msg for UUID-based workflow id?

[josephmsft]

It's expected because it's internal behavior from IoT Hub. The ADM configurations we're using to send the twin changes don't re-evaluate for 5 minutes after a change is detected, so if you get "nodeployment", it won't send you the configuration with the twin changes for the next deployment for 5 minutes.

[josephmsft]

This should only happen if there's not a deployment for the device initially though, so if you are connecting the very first device that will get a deployment (before the deployment is created), it will probably get "nodeployment" initially while you're creating the deployment. It won't give you that new deployment for 5 minutes after it gives you "nodeployment", even if you created it immediately after connecting the device. However, any new device created in that group after the deployment has been created should get the deployment right away, assuming it was tagged as being in the group before connecting to IoT Hub.

[D-r-P-3-p-p-3-r]

This should only happen if there's not a deployment for the device initially though, so if you are connecting the very first device that will get a deployment (before the deployment is created), it will probably get "nodeployment" initially while you're creating the deployment.

I just confirmed with the responsible tester. She is sure it happened for groups for which updates already had been deployed. She stated that she normally did not create new groups but pushed new updates for the same device to an existing group. Still, this situation occurred.

[josephmsft]

"However, any new device created in that group after the deployment has been created should get the deployment right away, assuming it was tagged as being in the group before connecting to IoT Hub."

Sorry I was wrong to say "right away". I looked up some old communication on this, it looks like the 5 minute re-evaluation window is tracked within IoT Hub on a per-device basis, so if your device was checked 10 seconds before you create the deployment, it still won't get the new deployment until the 5 minute re-evaluation wait time passes, which would look like almost 5 minutes. So it could be up to 5 minutes for any device in the group, although it should average less than that if the device is just getting an additional deployment to the same group. This internal IoT Hub technology was optimized for very large numbers (millions) of devices to get twin changes as quickly as possible without overloading the internal messaging system, but it can lead to higher than desired latency when dealing with small numbers of devices.

[jw-msft]

Moved this to discussion since one issue was by design and the other has a new more targeted issue created for it here:
#694

[jw-msft]

Here is PR #705