How to change auto-requeue hpc.ckpt path

[arijit-hub]

Hi,

I am using a slurm environment and am requeuing the job using lightning's automatic slurm handler. It works flawlessly. However, I just have one small issue. The temporary checkpoints hpc_ckpt_*.ckpt are saved in the current working directory instead of the directory I specified for model checkpoint saving. This causes a flaw in my experiments when I try to run a new job when an earlier job is in auto-queue. What I mean is this:
(1) I had an old job which has hit the wall-time, saved a temporary ckpt, and is requeued. This will use the hpc_ckpt_*.ckpt to resume training.
(2) My new experiment with the same .sh file will not start from scratch as it thinks that the hpc_ckpt_*.ckpt that is there is intended for it to use.

Is there any fix for this?

[arijit-hub]

I figured it out. One needs to specifically set the default-root_dir in trainer to save the hpc checkpoints in a specific location. I am setting it to the slurm job id and I think that makes the most sense given the job id remains the same with requeue!