Set default conda kernel, or choose more cleverly?

[moble]

Description

I have a lot of conda kernels that do some fairly weird stuff. For example, one automatically opens an SSH tunnel to a cluster and starts a very specific version of python there to use as the kernel. I don't want to use that kernel to build my quarto files (especially because it fails when I'm not connected to the internet), but that's actually the one that quarto selects somehow. I have a vanilla kernel that I use for most of my work that I'd prefer to use by default, but I can't see how to make that happen. I've read the docs, but don't see anything relevant — even to explain how quarto is choosing what it's choosing.

I've also seen this discussion about global defaults. The main reasoning against having such things appears to be

For us, reproducibility (especially as content moves from being rendered locally to rendered, say, on CI) is a central concern, and ambient user-wide configurations like this tend to cause problems.

But if I want other people to reproduce my document, being able to use a vanilla kernel would actually improve their chances. On the other hand, having to specify the jupyter metadata option makes it less reproducible.

Is there a good way to handle this? Or is there at least some consistent way quarto is selecting the kernel to use?

[cderv]

There is a way with virtual to make Quarto use the python version from an active env. I don't think we have that for Conda, and we should definitely have it by checking active CONDA_DEFAULT_ENV. But it should work if conda put the env version first on PATH I think 🤔

Otherwise, the way to tell Quarto to use a version of Python without looking for it is to set QUARTO_PYTHON environment variable. If not set, it will look on PATH or use Python Launcher on windows. It will then take first kernelspec that provide python language.

[cscheid]

And I think that's right; I believe kernels are supposed to be sort of at a higher level than envs

Yes, this is a Jupyter feature that I believe to be a design mistake - the list of Jupyter kernels is stored globally, and there's no way to check the consistency between Jupyter kernels and available Python setups in the environment. This is particularly salient when distributions upgrade Python installations and leave stale Jupyter kernels behind; there's nothing we can do to guard against this possible source of problems.

[moble]

Fair enough, but I'll point out that in my case the kernels and python envs are consistent, and nothing stale is causing any problems. I actively use the various kernels.

[cderv]

For example, kernels on my Mac all exist in ~/Library/Jupyter/kernels, rather than an env-specific directory.

This is probably where preferring any kernel with language python located in CONDA_DEFAULT_ENV make sense. We did that for virtual env - I'll add for Conda. This will at least solve this use case : Creating a specific conda env for a project, and get default kernel for this env unless specified otherwise.

[moble]

any kernel with language python located in CONDA_DEFAULT_ENV

I don't understand this. I don't think any kernel will ever be located inside an env, and I don't know of any way that the env will know about a particular kernel. The only connection I see is that the first element of argv in the kernel.json may contain the name of the env — depending on how it's created.

If I do

conda create -y -n test_kernel_env python ipykernel
conda activate test_kernel_env
python -m ipykernel install --user --name=my_new_kernel

it installs the kernel in /Users/moble/Library/Jupyter/kernels/my_new_kernel, and nothing I see in the env changes.

[cderv]

The only connection I see is that the first element of argv in the kernel.json may contain the name of the env — depending on how it's created.

I think this is the assumption we did for virtual env. Quarto will prefer any kernel with supported language that is found in virtual env path. If none, that default behavior is done. So this is a way to prefer the current virtual env installation when it is active.