BICAWS7-3351: Add concurrency limits to workflows

[ian-antking]

This commit adds concurrency limits to the first job of each workflow to limit the number of concurrent workflows to 40.

By adding a concurrencyExecLimit to the first task of each workflow, we can control how many of those workflows run at a given time. Because phases 1, 2 & 3 share the same first task (lock_s3_file), applying a concurrency limit to that task means we can control the total number of those three workflows running simultaneously.

Without concurrency limit (2000 messages):

Commit latency starts lower, but spikes when messages start clearing phase 1, and the compare files workflows begin.

The time to process messages was about 16.9 minutes.

With concurrency limit (2000 messages):

Commit latency more evenly spread, with a lower maximum.

The time to process messages was about 18.5 minutes.

Adding a concurrency limit reduces database commit latency at the cost of an extra two minutes to process the incoming message queue.

Note

It's not as easy to limit the compare_files workflow, as the first task is a native conductor one, so there is no corresponding task.json to add a concurrencyExecLimit. As we can only limit the second task, this would probably explain why the throttled data still shows a spike in commit latency. This problem will solve itself as we reduce the need to compare outputs from old/new Bichard

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud