Deadlock patch 001: Addressing accumulating locks on contributors table on very large instances

[sgoggins]

Description

• Deadlock management in Augur is generally addressed using a progressive transaction shrinking and waiting policy in order to:
  • Maximize data gathering throughput
  • Minimize contention and deadlocking
• Experiments show that this strategy is working well with the occasional exception of deadlocks around the contributors table. In this case, and because contributors are so logically coupled with every other data gathering process, deadlocks can quickly cascade and accumulate.
  • This PR implements the general strategy found in more general terms throughout Augur, but centered on the unique properties of the contributors problem (The snowball of deadlocks has been observed to accelerate quickly on instances of Augur with more than 10k repositories and more than 10 simultaneous data collection processes.
  • The "storm" does not last forever and so far there is not any indication that in place retries are not ultimately working.
  • Rather, this is about lowering the load on the database engine so that we avoid generating deadlocks as much as is sometimes happening when the "large instance" thresholds are met).
  • Of course this problem like all programming and data problems is idiosyncratic to the particular repositories and contributors being selected at a point in time. Early indications are that when a repository is both new to Augur and contains more than 1,000 pull requests deadlocks occur more frequently

[github-actions]

[pylint] _{reported by reviewdog 🐶}
E1120: No value for argument 'natural_keys' in function call (no-value-for-parameter)

[sgoggins]

This does not seem to fix all of the deadlocks. Lets review together.