Propagating environmental variables from virtual environments to other extensions

[gnikit]

Hi All,

First of all thanks for the great extension.
I am one of the authors of Modern Fortran for VS Code. Currently, we have a lot of tooling that gets downloaded through pip (language server, formatters, preprocessors, etc.) and the extension expects it to be reachable in $PATH (by default) or via a user-specified path setting.

The problem is that we are completely ignoring any virtual environment the user might have setup for their project through
vscode-python. So the user's expectation for example would be, for our extension to install the language server in their venv, which should make it reachable in $PATH, and hence they don't have to edit any of the default settings.

Do you have/see a way to propagate the Python interpreter and its env, set in vscode-python to other extensions?
For the installation process we use vscode.Task and vscode.ShellExecution, while for spawning the individual tools we just use spawnSync, but we would be happy to swap them with whatever works.

I think this would be useful for a few other extensions, reStructuredText and CMake Tools come to mind that are struggling with installing and fetching, like us (reStructuredText), or just fetching (CMake tools) binaries installed via virtual environments.

This Discussion is tangentially related but not fully applicable to #18978

[brettcannon]

So if I'm understanding things correctly, you're trying to to install stuff into the user's environment via pip? And currently you're simply calling pip from the terminal, but that can do the wrong thing because you're not getting the activated terminal?

If that's correct, then I have (potential) news and a suggestion. First, @karrtikr is working on new terminal integration which should make opening a new, activated terminal work much better. We could potentially expose what environment variables are involved via our API and then you could replicate it if you didn't want to execute some command we provide.

But honestly, if you're just running pip to do installation then none of this is necessary. A simpler, less error-prone approach is to get the execution command for the interpreter¹ and then use -m pip to actually run pip from the user's environment instead of calling pip directly/blindly in the terminal (I have a whole blog post on the subject if you want deeper understanding as to why I am making this recommendation). And we are making sure our Create Environment command installs pip for users in their environment, but if you're worried about pip not being installed then you can grab https://bootstrap.pypa.io/pip/pip.pyz and pass that to Python as what you want Python to execute (e.g. python pip.pyz <args to pip>).

Footnotes

1. I would say "get the path to the interpreter", but Anaconda doesn't like that and you have to use conda run. ↩

[gnikit]

So if I'm understanding things correctly, you're trying to to install stuff into the user's environment via pip? And currently you're simply calling pip from the terminal, but that can do the wrong thing because you're not getting the activated terminal?

Yes, that is correct.

But honestly, if you're just running pip to do installation then none of this is necessary. A simpler, less error-prone approach is to get the execution command for the interpreter1 and then use -m pip to actually run pip from the user's environment instead of calling pip directly/blindly in the terminal

We are already doing that, however there are 2 problems with this approach (from our users' perspective):

1. Calling {pythonInterpreter} -m pip would not pick up the locally configured Python env i.e. the correct {pythonInterpreter}. That I have addressed by using the API you provide (which I somehow missed before starting this discussion).
2. Spawning the relevant binaries (language servers, formatters, etc.) installed in a Python env will fail if the install location is not in the user's $PATH. That can be somewhat ameliorated by running our tools like {pythonInterpreter} -m myTool. The only problem is that some of these tools don't support that (they don't have a __main__.py), so we are forced to a) either assume they are reachable via spawn or b) locate the absolute path to bin/Scripts and try and locate them that way.
   Currently, we are doing option a), which is problematic, especially for Fortran Windows users who rarely have their Python environments setup correctly.
   We will start implementing option b), in the least possibly hacky way, but it would make things so much simpler on our end if we could pass additional { env: ... } variables obtained by ms-python.python's API to our running process

const pythonApi: IExtensionApi = extension.exports as IExtensionApi;
const pythonEnv = await pythonApi.environments.resolveEnvironment(
  pythonApi.environments.getActiveEnvironmentPath(resource)
);
// No need to pass absolute path to `myTool` since it will be discoverable via the pythonEnv.env
const pid = child_process.spawn('myTool', ['arg1', 'arg2'], { env: { ...pythonEnv.env, ...process.env })

Another alternative for problem 2, would be to use pipx and bypass all of this. Do you think that might be preferable?
Have you considered it yourselves as an installing/running mechanism in ms-python.python for certain tools you install via pip like formatters and some language servers?

P.S. I am slightly ignoring Anaconda ATM

[brettcannon]

would be to use pipx and bypass all of this. Do you think that might be preferable?

It's definitely the simplest solution for what you're after since you could control where pipx did the install. But what pipx does isn't complicated either and you could replicate it that experience yourself if you wanted as well (all depends on your needs, how much control you require, and how much work you want to put into this).

Have you considered it yourselves as an installing/running mechanism in ms-python.python for certain tools you install via pip like formatters and some language servers?

We're actually moving away from doing any installs by the extension. We are moving any tools we support into separate extensions which ship with copies of the tool, avoiding the whole installation question to begin with.

[gnikit]

I think I got things to work. Thanks @brettcannon for the help!