Use uv / pixi?

[TomNicholas]

without much context, obligatory pixi endorsement comment

emerging from a rabbit hole (thanks @cisaacstern 😂) I'm floating around this decision framework for feedback:

• If you're starting a new Python package, use uv
• If you need to manage multiple Python virtual environments at a time, also use hatch
• If you need Conda-Forge in addition to PyPI distributions, also use pixi

More on topic, feel welcome to ping me if you'd like to assign any CI / packaging updates.

Originally posted by @maxrjones in #309 (comment)

[cisaacstern]

In general I think this decision framework makes sense, though (with the caveat that I've never used hatch environments) I might omit the middle bullet to make it just

• If you're starting a new Python package [and don't conda dependencies], use uv
• If you need Conda-Forge in addition to PyPI distributions, use pixi

Pixi will manage multiple environments for use in ci, and switching to pixi would mean being able to entirely remove the ci/ directory from this project, moving all of what's specified there into the pixi.toml manifest file, or put directly into the [tool.pixi] table of the pyproject.toml if you prefer.

[cisaacstern]

Pixi will manage multiple environments for use in ci, and switching to pixi would mean being able to entirely remove the ci/ directory from this project, moving all of what's specified there into the pixi.toml manifest file, or put directly into the [tool.pixi] table of the pyproject.toml if you prefer.

I've been meaning to help out with this project somehow, and if this is of interest, I could make a PR to demonstrate.

[maxrjones]

Pixi will manage multiple environments for use in ci, and switching to pixi would mean being able to entirely remove the ci/ directory from this project, moving all of what's specified there into the pixi.toml manifest file, or put directly into the [tool.pixi] table of the pyproject.toml if you prefer.

I've been meaning to help out with this project somehow, and if this is of interest, I could make a PR to demonstrate.

I expect that we'll want conda for some of the drivers, so pixi seems like a good option to me for virtualizarr. plus I'd be excited to learn from your PR 😄

[TomNicholas]

if this is of interest, I could make a PR to demonstrate.

This is definitely of interest, and I would love it if you could point us to how it should be done 😁

[cisaacstern]

Great to know! I'd love to share a PR showing how I'd approach it and get all of your feedback.

Are any other outstanding issues / PRs dependent on this, or are there any deadlines this is needed for?

I will try to get to it ASAP but want to make sure I don't accidentally miss some important context.

[TomNicholas]

Are any other outstanding issues / PRs dependent on this, or are there any deadlines this is needed for?

No none at all. It's already distributed through pip and conda, this would just be modernizing / a quality of life improvement.

[maxrjones]

The RFC PRs for pixi (#407) and uv (#479) are ready for review.

As a developer, the best way to get familiar with how this would impact your workflows is to check out the updated contributor guides for the pixi and uv options (or to try them out directly.

what would we get with uv?

The main feature that uv adds for us is the ability to specify sources. For example,

VirtualiZarr/pyproject.toml

Lines 139 to 148 in 1485b4a

	[tool.uv.sources]
	virtualizarr = { workspace = true }
	xarray = { git = "https://github.com/pydata/xarray" }
	universal-pathlib = { git = "https://github.com/fsspec/universal_pathlib" }
	numcodecs = { git = "https://github.com/zarr-developers/numcodecs" }
	ujson = { git = "https://github.com/ultrajson/ultrajson" }
	fsspec = { git = "https://github.com/fsspec/filesystem_spec" }
	s3fs = { git = "https://github.com/fsspec/s3fs" }
	kerchunk = { git = "https://github.com/fsspec/kerchunk" }
	icechunk = { git = "https://github.com/earth-mover/icechunk", subdirectory = "icechunk-python" }

specifies the upstream sources for various dependencies. The dependencies will be installed using these git sources by default when using uv run pytest. If you want to use the pypi released versions instead, it's necessary to specify --no-sources (e.g., uv run --all-extras --no-sources pytest).

The main drawback of uv over our existing setup (conda environments) is that not all compressors are available on PyPI (e.g., Unidata/netcdf4-python#1164). While users could still install additional binaries available on conda-forge, we would only be able to test against zlib compression for NetCDF files.

what would we get with pixi?

pixi gives us a few additional features on top of uv:

1. Ability to specify multiple environments (e.g., min-deps, upstream, various python version).
2. Ability to tie commands to different environments (e.g., pixi run docs to build all the docs)
3. Ability to specify both conda (E.g., NetCDF4 compression plugins) and PyPI deps in a single project.toml

pixi does not have any drawbacks relative to our current setup. It's primary drawback relative to uv is that it's less widely used, which means less familiarity for potential new contributors and less of a community to get support from.

can we get multiple environments with uv?

We could use hatch in addition to uv to gain pixi's listed benefits 1 and 2 (ala Zarr-Python's setup), but still would not have all NetCDF4 plugins available in the CI.

request for feedback

I'd like feedback on which to go forward with from people who're likely to be submitting PRs to VirtualiZarr (maybe @TomNicholas @abarciauskas-bgse @sharkinsspatial @norlandrhagen @chuckwondo @jsignell have opinions). I have a strong preference for moving from the current environment.yml based set up to a pyproject.toml consolidated setup. I have a slight preference towards pixi to be able to have a more complete test suite, but am fine with going with the groups preference. I can commit to updating our project setup again if either choice loses maintenance support in the future.

[abarciauskas-bgse]

I haven't used pixi yet but am concerned about:

The main drawback of uv over our existing setup (conda environments) is that not all compressors are available on PyPI (e.g., Unidata/netcdf4-python#1164). While users could still install additional binaries available on conda-forge, we would only be able to test against zlib compression for NetCDF files.

This is a hard choice but if the best suited tool overall is pixi, I would go with pixi as long as it is actively maintained. I'm not so concerned that it's not as widely used as uv.

[cisaacstern]

I am not (yet 😄 ) a contributor to VirtualiZarr, but as a community member who has been using pixi daily for nearly a year now, I would like to make a strong endorsement in favor of pixi.

In my assessment, pixi is perfectly suited for exactly this type of use case; namely, centralized management of test/project environments for a project that can benefit from testing against conda-forge dependencies (even if those dependencies are not required). As shown in the RFC pixi PR, the ability for pixi to manage python versions is also very useful for replicating/debugging python-version-specific issues locally.

To be more specific, without the ability to explicitly manage Unidata/netcdf4-python#1164 as part of a test environment(s), how can this project be confident that regressions are not being introduced in the compatibility with such non-required-but-commonly-used compressors?

Re: the maintenance question, in my own usage of pixi (and its surrounding ecosystem, including py-ratter, ratter-build, etc.) I have encountered the usual number of rough edges that one would expect of a relatively new and ambitious package, and my experience has been that the core devs on these projects are uniquely responsive in triaging issues, both via github and also their Discord (link in navbar here). I have had many questions answered, gotten guidance on contributing PRs (which have been promptly reviewed), and had bugs I've reported fixed, all with remarkably very fast turnarounds.

For what it's worth, and in case it's not already known to others here, the team behind pixi is the same team that has developed what will eventually become the new official package spec and build system for conda forge. I bring this up only to highlight, in case it wasn't obvious, how deeply embedded in, and trusted, this team is within the conda ecosystem.

As a final thought and at the risk of getting philosophical, VirtualiZarr is quickly becoming a very prominent and visible package within the python scientific world, attracting a lot of users and amazing developer talent. I believe pixi is doing something uniquely valuable for this ecosystem in modernizing the integration of conda-compatible packages into python package development in a highly ergonomic and reproducible manner. As such, I feel there is also tremendous value from an ecosystem perspective, in this project lending an "endorsement" to pixi, insofar as that results in more developer energy going towards pixi, which is truly an important and groundbreaking project.

[maxrjones]

Thanks so much for sharing your thoughts @cisaacstern! Apologies for sounding exclusionary in my RFC, I definitely value all opinions and appreciate everything you've offered to the project even if you haven't yet gotten a change to write code 🙇

[TomNicholas]

I'm less confident that pixi will be around for the long-term, and I do really like the developer experience uv offers, but if pixi is the best way to solve our projects' needs right now then I'm fine with using it, because it doesn't really lock us in in any significant way anyway.

[douglatornell]

I'm +1 for pixi because conda and conda-forge play major roles in the projects I work on and the research groups that I work with. It would be distressing for VirtualiZarr to fail in some way in the conda ecosystem because its CI infrastructure was not conda-aware. With pixi that conda-awareness in CI is possible. With uv (which I acknowledge is very cool!) it's not, at least presently.