Build fails with free-threaded Python

[brianhelba]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Issue description

When building polars against free-threaded Python 3.13.2 on Linux (using the nightly version of Rust), the build fails with the error:

         Compiling numpy v0.23.0
      error[E0432]: unresolved import `pyo3::sync::GILProtected`
        --> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/numpy-0.23.0/src/datetime.rs:63:12
         |
      63 | use pyo3::{sync::GILProtected, Bound, Py, Python};
         |            ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
         |
      note: found an item that was configured out
        --> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.23.5/src/sync.rs:47:12
         |
      47 | pub struct GILProtected<T> {
         |            ^^^^^^^^^^^^
      note: the item is gated here
        --> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.23.5/src/sync.rs:46:1
         |
      46 | #[cfg(not(Py_GIL_DISABLED))]
         | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      error[E0432]: unresolved import `pyo3::sync::GILProtected`
        --> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/numpy-0.23.0/src/strings.rs:15:5
         |
      15 |     sync::GILProtected,
         |     ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
         |
         = note: unresolved item `crate::datetime::units::GILProtected` exists but is inaccessible
      note: found an item that was configured out
        --> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.23.5/src/sync.rs:47:12
         |
      47 | pub struct GILProtected<T> {
         |            ^^^^^^^^^^^^
      note: the item is gated here
        --> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.23.5/src/sync.rs:46:1
         |
      46 | #[cfg(not(Py_GIL_DISABLED))]
         | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      error[E0282]: type annotations needed for `&mut _`
         --> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/numpy-0.23.0/src/datetime.rs:229:13
          |
      229 |         let dtype = match dtypes.get_or_insert_with(Default::default).entry(unit) {
          |             ^^^^^
      ...
      247 |         dtype.bind(py).to_owned()
          |               ---- type must be known at this point
          |
      help: consider giving `dtype` an explicit type, where the type for type parameter `V` is specified
          |
      229 |         let dtype: &mut V = match dtypes.get_or_insert_with(Default::default).entry(unit) {
          |                  ++++++++

      error[E0282]: type annotations needed for `&mut _`
         --> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/numpy-0.23.0/src/strings.rs:185:13
          |
      185 |         let dtype = match dtypes.get_or_insert_with(Default::default).entry(size) {
          |             ^^^^^
      ...
      198 |         dtype.bind(py).to_owned()
          |               ---- type must be known at this point
          |
      help: consider giving `dtype` an explicit type, where the type for type parameter `V` is specified
          |
      185 |         let dtype: &mut V = match dtypes.get_or_insert_with(Default::default).entry(size) {
          |                  ++++++++

      Some errors have detailed explanations: E0282, E0432.
      For more information about an error, try `rustc --explain E0282`.
      error: could not compile `numpy` (lib) due to 4 previous errors
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/.../.cache/uv/builds-v0/.tmpV8LOZL/bin/python"
      PYTHON_SYS_EXECUTABLE="/.../.cache/uv/builds-v0/.tmpV8LOZL/bin/python" "cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "/.../.cache/uv/sdists-v8/pypi/polars/1.25.2/ItmSIH_8XlFqKYzSiXKdr/src/py-polars/Cargo.toml"
      "--release" "--lib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/.../.cache/uv/builds-v0/.tmpV8LOZL/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

---

This error has already been solved upstream by PyO3/rust-numpy#471 and released in rust-numpy 0.24.

To resolve this, I believe that at least polars needs to upgrade the Rust dependency numpy to 0.24 (and likely pyo3 to 0.24 too). Other fixes may be necessary too.

This upgrade work may be related to the PRs #20111 and #20683 by @bschoenmaeckers .

[bschoenmaeckers]

It is currently not possible to upgrade to the latest PyO3 release because of a upstream bug. I will pick this up when ready. Free-threading probably requires more internal changes to prevent deadlocks.

[bschoenmaeckers]

I did some testing in #21914 and the main blockers in our dependency tree are:

• cffi
• ongoing work at https://github.com/quansight-labs/cffi
• cython
• will be released under v3.1.0
• gevent which depends on cython
• greenlet
• cryptography which depends on cffi

After this we should be able to do some testing in the CI to make sure polars is thread-safe.

[Object905]

Note that greenlet is only a dependency of gevent and it's not used directly.
Also given how they're maintained (not abandoned, but mostly catching up with upstream python) and how they work (by exploiting python internals and swapping stacks) I doubt that we will see them support free-threaded in foreseeable future.

Also they're used only for collect_async (which are marked experimental) as an alternative mode to asyncio, that I made, because I needed that for a legacy project, that since migrated to not using gevent 😃
My point is while they're default libraries for green threading - green threading in general is not a popular choice in python nowadays and my feeling that you can count users that use polars+gevent+collect_async on one hand (if there are any).
So, removing gevent support in collect_async altogether should be fine?

But seems like they shouldn't block free-threaded builds, because its an optional dependency and you couldn't build/use them in free-threaded anyway. But they're have to be excluded from requirements/tests in free-threaded in CI.

ref
python-greenlet/greenlet#423
gevent/gevent#2087