Automatically restart a failed job from spot pre-emption events

[rojomisin]

For on-premise runners, using cloud spot instances, can we use the pre-emption events to handle the job retry in the helm chart? We can use the gke node logs and react via pubsub and cloud functions to send a restart on any recent failed jobs, but is there a way to do this easier already in the arc controller?