Jask concurrency control

[HZ89]

Description

A job executing on a large number of nodes at the same time will cause excessive concurrency to other systems on which the job itself depends, requiring a way to control the concurrency of the job.

Design

• Job object to add concurrency attributes
• The server generates a token counter according to the value of concurrency when the job can be triggered
• When an agent is ready to perform a job try to get the token, if successful do it, else to wait for a cycle of retry

[HZ89]

Implementation

• concurrency status save to etcd. In case a server is down it loses concurrency control
• server start some goroutine and assign a job status to a fixed one of the gorountine maintenance
• modify grpc service job.GetJob and job.ExecDone, add the token's distribution and recycle logic
• when an agent make a job.GetJob rpc call, server reduce concurrency first. if succeed return job obj, if not sleep a well and retry, until this is succeed.