Maestro sometimes doesn't open dependency update PRs

[vseanreesermsft]

Duplicate of #4354

I'm not able to reopen that issue.

See more examples in comments there

runtime is not flowing into any repos:

• completed build for internal/release/8.0: https://dev.azure.com/dnceng/internal/_build/results?buildId=2640520&view=results
• completed build for internal/release/9.0: https://dev.azure.com/dnceng/internal/_build/results?buildId=2640557&view=results

No PRs created anywhere:
https://dev.azure.com/dnceng/internal/_git/dotnet-aspnetcore/pullrequests?_a=active
https://dev.azure.com/dnceng/internal/_git/dotnet-efcore/pullrequests?_a=active
https://dev.azure.com/dnceng/internal/_git/dotnet-winforms/pullrequests?_a=active
https://dev.azure.com/dnceng/internal/_git/dotnet-sdk/pullrequests?_a=active

logs show the build should be applied (eg from 8.0):

BAR Build Id:  256243
Released:      False
Channels:
- .NET 8 Internal
The following repos/branches will apply this build immediately:
  https://dev.azure.com/dnceng/internal/_git/dotnet-aspnetcore @ internal/release/8.0
  https://dev.azure.com/dnceng/internal/_git/dotnet-winforms @ internal/release/8.0
  https://dev.azure.com/dnceng/internal/_git/dotnet-efcore @ internal/release/8.0
  https://dev.azure.com/dnceng/internal/_git/dotnet-sdk @ internal/release/8.0.1xx
  https://dev.azure.com/dnceng/internal/_git/dotnet-sdk @ internal/release/8.0.3xx
  https://dev.azure.com/dnceng/internal/_git/dotnet-sdk @ internal/release/8.0.4xx

Triggering subscriptions manually with the darc cli made all the PRs:
https://dev.azure.com/dnceng/internal/_git/dotnet-aspnetcore/pullrequest/47540
https://dev.azure.com/dnceng/internal/_git/dotnet-aspnetcore/pullrequest/47534
https://dev.azure.com/dnceng/internal/_git/dotnet-efcore/pullrequest/47535
https://dev.azure.com/dnceng/internal/_git/dotnet-efcore/pullrequest/47531
https://dev.azure.com/dnceng/internal/_git/dotnet-winforms/pullrequest/47536
https://dev.azure.com/dnceng/internal/_git/dotnet-winforms/pullrequest/47530
https://dev.azure.com/dnceng/internal/_git/dotnet-sdk/pullrequest/47539
https://dev.azure.com/dnceng/internal/_git/dotnet-sdk/pullrequest/47538
https://dev.azure.com/dnceng/internal/_git/dotnet-sdk/pullrequest/47537
https://dev.azure.com/dnceng/internal/_git/dotnet-sdk/pullrequest/47533
https://dev.azure.com/dnceng/internal/_git/dotnet-sdk/pullrequest/47532

[vseanreesermsft]

More builds that didn't flow:
https://dev.azure.com/dnceng/internal/_build/results?buildId=2641512&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2641510&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2641695&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2641696&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2641698&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2641864&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2641869&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2641913&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2642023&view=results

[premun]

@dkurepa can you have a look?

[dkurepa]

Hi Sean, it appears that the service does have problems in some cases.
For example the internal/release/runtime build that you linked 256243 DID trigger did do what it was supposed to, and opened up PRs in all of the repos that wanted it. The logs we have in our application insights, along with the AzDo build times tell us this. However the subsequent runtime build of the same branch 256298 didn't do this. It finished around 5AM UTC yesterday, but no subscriptions got triggered until manual triggers, around 14h later.

While looking into it, we noticed that we currently don't have enough logging information to tell what's happening here, so we're adding some in the PR above.

We'll roll out these changes ASAP and go on to fixing the issues once we find out what's causing it

[vseanreesermsft]

256243 didn't trigger any PRs. It didn't open PRs in any of the repos that I listed until I triggered manually. I looked in every repo and there were no PRs there. I waited and waited, then I triggered them manually and they showed up within 30 seconds.

[vseanreesermsft]

There has not been a single PR created for dependency flow without manually triggering subscriptions in any of the following repos for at least two weeks. If your logs say they're creating PRs then your logs are incorrect.
dotnet-sdk, dotnet-installer, dotnet-aspnetcore, dotnet-efcore, dotnet-windowsdesktop, dotnet-wpf, dotnet-winforms.

[vseanreesermsft]

Here, existing PRs (triggered manually) are not being updated by completed builds.

SDK Builds:

• https://dev.azure.com/dnceng/internal/_build/results?buildId=2642508&view=results
• https://dev.azure.com/dnceng/internal/_build/results?buildId=2642511&view=results
• https://dev.azure.com/dnceng/internal/_build/results?buildId=2642512&view=results

The PRs don't have updates:
https://dev.azure.com/dnceng/internal/_git/dotnet-installer/pullrequest/47525?_a=commits

https://dev.azure.com/dnceng/internal/_git/dotnet-installer/pullrequest/47526?_a=commits

https://dev.azure.com/dnceng/internal/_git/dotnet-installer/pullrequest/47527?_a=commits

Taking screenshots before triggering subs manually to show these didn't happen automatically.

[vseanreesermsft]

Triggered subs manually and got the updates.