SSH Agent Signed Certificate Forwarding not working

[Soneji]

Is there an existing issue for this bug?

• I have searched the existing open issues and found none that apply.
• If I find any issue of interest that is related or closed, I will included a link to it in this issue.

Required Troubleshooting Steps

• I have read and performed the troubleshooting steps
• I have tried both values of the remote.SSH.useLocalServer setting
• My issue was not covered in the Tips and Tricks linked from the Troubleshooting Wiki.
• I will include a complete copy of my Remote - SSH logs by running Remote-SSH: Show Log in the command palette or from View > Output in the menu bar

[Optional] Diagnose with Copilot

I asked @remote-ssh but it did not help

In step 2 of the troubleshooting wiki, what was the result of running the generated SSH command verbatim outside of VS Code?

I did not try step 2 of the troubleshooting steps

Remote-SSH Log

Remote-SSH Log

[12:10:18.913] Log Level: 2
[12:10:18.915] VS Code version: 1.96.4
[12:10:18.915] Remote-SSH version: remote-ssh@0.117.1
[12:10:18.915] darwin arm64
[12:10:18.954] SSH Resolver called for "ssh-remote+dev", attempt 1
[12:10:18.956] remote.SSH.useLocalServer = true
[12:10:18.956] remote.SSH.useExecServer = true
[12:10:18.956] remote.SSH.bindHost = {}
[12:10:18.956] remote.SSH.path = undefined
[12:10:18.956] remote.SSH.configFile = undefined
[12:10:18.956] remote.SSH.useFlock = true
[12:10:18.956] remote.SSH.lockfilesInTmp = false
[12:10:18.956] remote.SSH.localServerDownload = auto
[12:10:18.956] remote.SSH.remoteServerListenOnSocket = false
[12:10:18.956] remote.SSH.defaultExtensions = []
[12:10:18.956] remote.SSH.defaultExtensionsIfInstalledLocally = []
[12:10:18.956] remote.SSH.loglevel = 2
[12:10:18.956] remote.SSH.enableDynamicForwarding = true
[12:10:18.956] remote.SSH.enableRemoteCommand = false
[12:10:18.956] remote.SSH.serverPickPortsFromRange = {}
[12:10:18.956] remote.SSH.serverInstallPath = {}
[12:10:18.956] remote.SSH.permitPtyAllocation = false
[12:10:18.957] remote.SSH.preferredLocalPortRange = undefined
[12:10:18.957] remote.SSH.useCurlAndWgetConfigurationFiles = false
[12:10:18.957] remote.SSH.experimental.chat = false
[12:10:18.957] remote.SSH.experimental.enhancedSessionLogs = false
[12:10:18.963] SSH Resolver called for host: dev
[12:10:18.963] Setting up SSH remote "dev"
[12:10:18.966] Acquiring local install lock: /var/folders/yv/25c5dqg55_l4k9p0p4mg0vnc0000gq/T/vscode-remote-ssh-d55997bc-install.lock
[12:10:18.967] Looking for existing server data file at /Users/[USERNAME]/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-d55997bc-cd4ee3b1c348a13bafd8f9ad8060705f6d4b9cba-0.117.1-es/data.json
[12:10:18.967] No existing data file
[12:10:18.967] Using commit id "cd4ee3b1c348a13bafd8f9ad8060705f6d4b9cba" and quality "stable" for server
[12:10:18.967] Extensions to install: 
[12:10:18.970] Install and start server if needed
[12:10:18.972] PATH: /Users/[USERNAME]/localbins:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin
[12:10:18.972] Checking ssh with "ssh -V"
[12:10:18.984] > OpenSSH_9.8p1, LibreSSL 3.3.6

[12:10:18.985] askpass server listening on /var/folders/yv/25c5dqg55_l4k9p0p4mg0vnc0000gq/T/vscode-ssh-askpass-0c15e0b7d1190710b8f8c14ffaeb9a04a2798303.sock
[12:10:18.985] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/yv/25c5dqg55_l4k9p0p4mg0vnc0000gq/T/vscode-ssh-askpass-bf9f97c5ca67db11afed5b8220d2528806993650.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","62088","-o","ConnectTimeout=15","dev"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/[USERNAME]/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-d55997bc-cd4ee3b1c348a13bafd8f9ad8060705f6d4b9cba-0.117.1-es/data.json"}
[12:10:18.985] Local server env: {"SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.ta9dmq8QVS/Listeners","SHELL":"/usr/local/bin/fish","DISPLAY":"/private/tmp/com.apple.launchd.eDyNTE4YsH/org.xquartz:0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/[USERNAME]/.vscode/extensions/ms-vscode-remote.remote-ssh-0.117.1/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"","VSCODE_SSH_ASKPASS_MAIN":"/Users/[USERNAME]/.vscode/extensions/ms-vscode-remote.remote-ssh-0.117.1/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/yv/25c5dqg55_l4k9p0p4mg0vnc0000gq/T/vscode-ssh-askpass-0c15e0b7d1190710b8f8c14ffaeb9a04a2798303.sock"}
[12:10:18.986] Spawned 44923
[12:10:18.986] Using connect timeout of 17 seconds
[12:10:19.051] > local-server-1> Running ssh connection command: ssh -v -T -D 62088 -o ConnectTimeout=15 dev
[12:10:19.052] > local-server-1> Spawned ssh, pid=44930
[12:10:19.059] stderr> OpenSSH_9.8p1, LibreSSL 3.3.6
[12:10:19.084] stderr> Identity added: /Users/[USERNAME]/.ssh/id_ecdsa (USERNAME@LOCALCOMPUTER)
[12:10:19.085] stderr> Certificate added: /Users/[USERNAME]/.ssh/id_ecdsa-cert.pub (signingprogram)
[12:10:19.091] stderr> /Users/[USERNAME]/.ssh/id_ecdsa_cert.pub: No such file or directory
[12:10:20.550] stderr> debug1: Server host key: ssh-ed25519 SHA256:LZA/IhgbTvNRDagZ/K9QyNQo+TKdmqPS7EV5X2mQm70
[12:10:20.697] stderr> Authenticated to SERVER (via proxy) using "publickey".
[12:10:20.714] > Agent pid 44966
[12:10:20.723] stderr> Identity added: /Users/[USERNAME]/.ssh/id_ecdsa (USERNAME@LOCALCOMPUTER)
[12:10:20.723] stderr> Certificate added: /Users/[USERNAME]/.ssh/id_ecdsa-cert.pub (signingprogram)
[12:10:20.728] stderr> /Users/[USERNAME]/.ssh/id_ecdsa_cert.pub: No such file or directory
[12:10:20.834] stderr> Identity added: /Users/[USERNAME]/.ssh/id_ecdsa (USERNAME@LOCALCOMPUTER)
[12:10:20.835] stderr> Certificate added: /Users/[USERNAME]/.ssh/id_ecdsa-cert.pub (signingprogram)
[12:10:20.841] stderr> /Users/[USERNAME]/.ssh/id_ecdsa_cert.pub: No such file or directory
[12:10:22.580] > ready: b5c2a102ad4f
[12:10:22.603] > Linux 5.10.233-204.894.amzn2int.x86_64 #1 SMP Tue Jan 28 01:56:25 UTC 2025
[12:10:22.603] Platform: linux
[12:10:22.625] > /bin/zsh
[12:10:22.625] Parent Shell: zsh
[12:10:22.625] Parent Shell pid: 44923
[12:10:22.626] Waiting for subshell to start
[12:10:22.681] Waiting for subshell to start
[12:10:22.704] > 3601
> 3601
[12:10:22.704] stdout -> '3601
3601'
[12:10:22.704] sub-process detected
[12:10:22.737] > b5c2a102ad4f: running
> Script executing under PID: 3601
[12:10:22.745] > Found existing installation at /home/USERNAME/.vscode-server...
[12:10:22.746] > Starting VS Code CLI...
[12:10:22.749] > Removing old logfile at /home/USERNAME/.vscode-server/.cli.cd4ee3b1c348a13bafd8f9ad8060705f6d4b9cba.log
[12:10:22.752] > Spawned remote CLI: 3634
[12:10:22.755] > Waiting for server log...
[12:10:22.794] > b5c2a102ad4f: start
[12:10:22.796] > listeningOn==127.0.0.1:39791==
> osReleaseId==amzn==
> arch==x86_64==
> vscodeArch==x64==
> bitness==64==
> tmpDir==/run/user/30717269==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> serverStartTime==41==
> execServerToken==1a1111a1-aa1a-11aa-a11a-aa11aa1aa111==
> platformDownloadPath==cli-alpine-x64==
> SSH_AUTH_SOCK==/tmp/ssh-YqQWIFCT77/agent.3553==
> DISPLAY====
> b5c2a102ad4f: end
[12:10:22.797] Received install output: listeningOn==127.0.0.1:39791==
osReleaseId==amzn==
arch==x86_64==
vscodeArch==x64==
bitness==64==
tmpDir==/run/user/30717269==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
serverStartTime==41==
execServerToken==1a1111a1-aa1a-11aa-a11a-aa11aa1aa111==
platformDownloadPath==cli-alpine-x64==
SSH_AUTH_SOCK==/tmp/ssh-YqQWIFCT77/agent.3553==
DISPLAY====

[12:10:22.798] Remote server is listening on port 39791
[12:10:22.798] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":39791},"osReleaseId":"amzn","arch":"x86_64","sshAuthSock":"/tmp/ssh-YqQWIFCT77/agent.3553","display":"","tmpDir":"/run/user/30717269","platform":"linux","execServerToken":"1a1111a1-aa1a-11aa-a11a-aa11aa1aa111"},"serverStartTime":41,"installUnpackCode":""}
[12:10:22.799] Persisting server connection details to /Users/[USERNAME]/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-d55997bc-cd4ee3b1c348a13bafd8f9ad8060705f6d4b9cba-0.117.1-es/data.json
[12:10:22.804] Starting forwarding server. local port 62108 -> socksPort 62088 -> remotePort 39791
[12:10:22.804] Forwarding server listening on port 62108
[12:10:22.805] Waiting for ssh tunnel to be ready
[12:10:22.807] [Forwarding server port 62108] Got connection 0
[12:10:22.808] Tunneled port 39791 to local port 62108
[12:10:22.808] Resolved "ssh-remote+dev" to "port 62108"
[12:10:22.812] Initizing new exec server for ssh-remote+dev
[12:10:22.812] Resolving exec server at port 62108
[12:10:22.813] [Forwarding server port 62108] Got connection 1
[12:10:22.964] Exec server for ssh-remote+dev created and cached
[12:10:22.964] Extensions to install: 
[12:10:22.965] Updating $SSH_AUTH_SOCK: ln -f -s "/tmp/ssh-YqQWIFCT77/agent.3553" "/run/user/30717269/vscode-ssh-auth-sock-220652328"
[12:10:22.966] Using cwd: vscode-remote://ssh-remote%2Bdev/
[12:10:22.967] Remote extension host environment: {"SSH_AUTH_SOCK":"/run/user/30717269/vscode-ssh-auth-sock-220652328"}
[12:10:22.972] ------




[12:10:22.972] No hints found in the recent session.
[12:10:23.006] [server] Checking /home/USERNAME/.vscode-server/cli/servers/Stable-cd4ee3b1c348a13bafd8f9ad8060705f6d4b9cba/log.txt and /home/USERNAME/.vscode-server/cli/servers/Stable-cd4ee3b1c348a13bafd8f9ad8060705f6d4b9cba/pid.txt for a running server...
[12:10:23.007] [server] Found running server (pid=10291)
[12:10:24.634] "Update SSH_AUTH_SOCK" terminal command done
[12:10:25.141] Opening exec server for ssh-remote+dev
[12:10:25.153] Opening exec server for ssh-remote+dev
[12:10:25.170] Verified and reusing cached exec server for ssh-remote+dev
[12:10:25.170] No hints found in the recent session.
[12:10:25.179] Verified and reusing cached exec server for ssh-remote+dev
[12:10:25.179] No hints found in the recent session.
[12:10:25.205] Opening exec server for ssh-remote+dev
[12:10:25.233] Verified and reusing cached exec server for ssh-remote+dev
[12:10:25.234] No hints found in the recent session.

Expected Behavior

I expect to be able to use my SSH signed certificate on my remote machine

Actual Behavior

❯ ssh-add -l           
error fetching identities for protocol 1: agent refused operation
256 SHA256:XXX XXX (ECDSA)

Missing (ECDSA-CERT)

Steps To Reproduce

I don't know the exact details of how my EDCSA cert and CA works in the background but this chatgpt guide explains the basics
https://chatgpt.com/share/67b47d48-198c-800b-8366-279f61fa3b01

Now we have a

• private key
• public key
• signed public key

With VSCode the private key is copied over to the machine, but the signed public key is not

When I SSH normally it is actually copied across.

❯ ssh-add -l
error fetching identities for protocol 1: agent refused operation
256 SHA256:XXX XXX (ECDSA)
256 SHA256:XXX XXX (ECDSA-CERT)

Anything else?

My fish config

eval "$(ssh-agent -c)"
ssh-add ~/.ssh/id_ecdsa

My ssh config

# Cloud Desktop
Host dev
  HostName MYHOST
  User MYUSER
  ForwardAgent yes

[Soneji]

When I run ssh dev locally the agent seems to automatically add the certificates properly:

❯ ssh dev
Identity added: /Users/USERNAME/.ssh/id_ecdsa (LOCALUSER@LOCALHOST)
Certificate added: /Users/USERNAME/.ssh/id_ecdsa-cert.pub (SIGNINGPROG)
Agent pid 60732
Identity added: /Users/USERNAME/.ssh/id_ecdsa (LOCALUSER@LOCALHOST)
Certificate added: /Users/USERNAME/.ssh/id_ecdsa-cert.pub (SIGNINGPROG)
Identity added: /Users/USERNAME/.ssh/id_ecdsa (LOCALUSER@LOCALHOST)
Certificate added: /Users/USERNAME/.ssh/id_ecdsa-cert.pub (SIGNINGPROG)

And then gives me a prompt

[joshspicer]

Thanks for the issue. Are you able to provide steps of how someone could repro your situation?

[vs-code-engineering]

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 10 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!