Attach debugger to remote process on cluster

[tsbertalan]

I'm running VSC on a SLURM cluster, where the actual VSC process runs on a login node, but I would like to run, and attach the debugger to, GPU nodes that are accessible interactively by running either the wrapper command

interact -p gpuk80 -g 1

or its generated command

salloc -J interact -N 1-1 -n 1 --time=30:00 --gres=gpu:1 -p gpuk80 srun --pty bash

Once such an interactive session is granted, various SLURM tools can provide the local network hostname for that assigned node, e.g. gpu075, and then I can ssh gpu075, enter my password (SSH public key authentication does not work), and see both sessions as child processes of /software/centos7/bin/bash. However, while I don't think anyone cares if I SSH in directly this way to check on process hierarchy, or issue SIGINT or something, it's not the right way to start new nontrivial processes (with actual RAM and CPU/GPU usage); that's what interact and sbatch are for.

I am able to create terminal configurations in ~/.config/Code/User/settings.json that look like the following to get new terminals that start out on the compute nodes:

"terminal.integrated.profiles.linux": {
        "shared": {
            "path": "SNIPPED_PATH/interact_shared.sh",
            "icon": "thumbsup"
        },
        "k80": {
            "path": "SNIPPED_PATH/interact_k80.sh",
            "icon": "beaker"
        },
        "p100": {
            "path": "SNIPPED_PATH/interact_p100.sh",
            "icon": "zap"
        }
    },

with attendant executable bash scripts like interact_k80.sh:

#!/bin/bash
interact -p gpuk80 -g 1

However, if I set one of these to be my default terminal with Ctrl+P > "Terminal: Select Default Profile" or "terminal.integrated.defaultProfile.linux": "k80" (again in ~/.config/Code/User/settings.json), while the correct interactive session is created and debug command is dropped in, the debugger does not connect:

(VIRTUALENV) [USERNAME@compute0199 EmergentPDE]$  /usr/bin/env /home-4/USERNAME/code/kse/bin/python /home-net/home-4/USERNAME/.vscode-server/extensions/ms-python.python-2022.0.1814523869/pythonFiles/lib/python/debugpy/launcher 36301 -- /home-4/USERNAME/PROJECTDIR/PYTHONFILENAME.PY 
Traceback (most recent call last):
  File "/software/apps/python/3.8.1/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/software/apps/python/3.8.1/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home-net/home-4/USERNAME/.vscode-server/extensions/ms-python.python-2022.0.1814523869/pythonFiles/lib/python/debugpy/launcher/__main__.py", line 97, in <module>
    main()
  File "/home-net/home-4/USERNAME/.vscode-server/extensions/ms-python.python-2022.0.1814523869/pythonFiles/lib/python/debugpy/launcher/__main__.py", line 53, in main
    launcher.connect(host, port)
  File "/home-net/home-4/USERNAME/.vscode-server/extensions/ms-python.python-2022.0.1814523869/pythonFiles/lib/python/debugpy/launcher/../../debugpy/launcher/__init__.py", line 34, in connect
    sock.connect((host, port))
ConnectionRefusedError: [Errno 111] Connection refused

(Here and above, I've replaced some strings with all-caps anonymizations.)

Apparently there needs to be some additional Debug Adapter Protocol client/server setup between the login node and the compute node for mediate this connection. My question is--has anyone encountered an analogous situation and already developed a such an adapter, ideally already in the vscode-python extension, or should I give up and just hope (since I don't have time to write one myself)? However, I'm open to being told this is an XY problem, and I should use a different approach to get my debugger connected.

Thanks

[tsbertalan]

FWIW, I'm pretty sure this can be done by modifying runpy.py to create an SSH tunnel to forward the UNIX socket from the compute node to the login node, and then modifying e.g. debugpy/launcher/__init__.py that runs on the login node to use the forwarded socket.

If I had more time to devote to this, I'd mess around with changing the corresponding files in ~/.vscode-server to try this out. However, this would probably get clobbered by an update or something, and anyway there's probably a more elegant way to handle it with extensions (or existing built-in features) that I don't know of.

[karthiknadig]

@tsbertalan What does your launch.json look like? Are you able to open a listening port on the login node (i.e, are there any firewall restrictions), if not, there is a way to reverse the connection so that the login node is the listener and adapter connects back to the login node form the compute node. For this you will have to use "listen" setting in launch json with attach as the request type. then launch the debugger yourself from the compute node with --connect command line argument for the debugger and the file you want to run.