PDEP-10: Add pyarrow as a required dependency #52711
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| # PDEP-10: PyArrow as a required dependency | ||
|
|
||
| - Created: 17 April 2023 | ||
| - Status: Under discussion | ||
| - Discussion: [#52711](https://github.com/pandas-dev/pandas/pull/52711) | ||
| [#52509](https://github.com/pandas-dev/pandas/issues/52509) | ||
| - Author: [Matthew Roeschke](https://github.com/mroeschke) | ||
| - Revision: 1 | ||
|
|
||
| ## Abstract | ||
|
|
||
| This PDEP proposes that: | ||
|
|
||
| - PyArrow becomes a runtime dependency starting with pandas 2.1 | ||
| - The minimum version of PyArrow supported starting with pandas 2.1 is version 7 of PyArrow. | ||
| - The minimum version of PyArrow will be bumped every major pandas release to the highest | ||
| PyArrow version that has been released for at least 2 years, and the minimum PyArrow version will be | ||
| maintained for every minor version in the major version series. | ||
|
|
||
| ## Background | ||
|
|
||
| PyArrow is an optional dependency of pandas that provides a wide range of supplemental features to pandas: | ||
|
|
||
| - Since pandas 0.21.0, PyArrow provided I/O reading functionality for Parquet | ||
| - Since pandas 1.2.0, pandas integrated PyArrow into the `ExtensionArray` interface to provide an optional string data type backed by PyArrow | ||
| - Since pandas 1.4.0, PyArrow provided I/0 reading functionality for CSV | ||
| - Since pandas 1.5.0, pandas provided an `ArrowExtensionArray` and `ArrowDtype` to support all PyArrow data types within the `ExtensionArray` interface | ||
| - Since pandas 2.0.0, all I/O readers have the option to return PyArrow-backed data types, and many methods now utilize PyArrow compute functions to | ||
| accelerate PyArrow-backed data in pandas, notibly string and datetime types. | ||
|
|
||
| As of pandas 2.0, one can feasibly utilize PyArrow as an alternative data representation to NumPy with advantages such as: | ||
|
|
||
| 1. Consistent ``NA`` support for all data types | ||
| 2. Broader support of data types such as ``decimal``, ``date`` and nested types | ||
Dr-Irv marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ## Motivation | ||
|
|
||
| While all the functionality described in the previous paragraph is currently optional, PyArrow has significant integration into many areas | ||
| of pandas. With our roadmap noting that pandas strives for better Apache Arrow interoperability [^1] and many projects [^2], within or beyond the Python ecosystem, adopting or interacting with the Arrow format, making PyArrow a required dependency provides an additional signal of confidence in the Arrow | ||
phofl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ecosystem to pandas users. | ||
|
|
||
| Additionally, requiring PyArrow would simplify the related development within pandas and potentially improve NumPy functionality that would be better suited | ||
| by PyArrow including: | ||
|
|
||
| - Avoiding runtime checking if PyArrow is available to perform PyArrow object inference during constructor or indexing operations | ||
|
There was a problem hiding this comment. Are there any small code samples we can add to drive this point home? I think still we would make a runtime determination whether to return a pyarrow or numpy-backed object even if both are installed, no? There was a problem hiding this comment. not sure this comment by Will has been addressed (unless I missed it?) to make it easier to find: the link is here, and says:
|
||
| - Currently, there are 17 `import_optional_dependency("pyarrow")` checks throughout the pandas code base | ||
| - Avoiding NumPy object data types more by default for analogous types that have native PyArrow support such as string, decimal, binary, and nested types | ||
|
|
||
| ## Drawbacks | ||
|
|
||
| Including PyArrow would naturally increase the installation size of pandas. For example, installing pandas and PyArrow using pip from wheels, numpy and pandas | ||
| are about `70MB`, and PyArrow is around `120MB`. An increase of installation size would have negative impliciation using pandas in space-constrained development | ||
| or deployment environments such as AWS Lambda. | ||
|
|
||
| Additionally, if a user is installing pandas in an environment where wheels are not available and needs to build from source, the user will need to build Arrow C++ and related dependencies. These environments include | ||
|
There was a problem hiding this comment. I think it would be good to more explicitly say that you need to do this (installing Arrow C++) manually and this is not possible through There was a problem hiding this comment. thanks for providing details here - is this ( There was a problem hiding this comment. Whoops, sorry there was a "not" missing in "not possible through But on the actual question how much of a hurdle this is: I would say, try it out yourself :) That's the best way to get an idea of how difficult it is, otherwise you can only take (or not) my words in saying that: yes, this is a huge hurdle. Installing pyarrow and Arrow C++ from source is far from trivial. There was a problem hiding this comment. thanks - from the installation instructions, it certainly seems tricky https://arrow.apache.org/docs/developers/cpp/building.html# |
||
|
|
||
| - Alpine linux (commonly used as a base for Docker containers) | ||
| - WASM (pyodide and pyscript) | ||
| - Python development versions | ||
|
|
||
| Lastly, pandas development and releases will need to be mindful of PyArrow's development and release cadance. For example when supporting a newly released Python version, pandas will also need to be mindful of PyArrow's wheel support for that Python version before releasing a new pandas version. | ||
|
|
||
| ### PDEP-1 History | ||
phofl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - 17 April 2023: Initial version | ||
phofl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| [^1] <https://pandas.pydata.org/docs/development/roadmap.html#apache-arrow-interoperability> | ||
| [^2] <https://arrow.apache.org/powered_by/> | ||
attack68 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm.. This might be too aggressive and might also make it hard to predict what the minimum version will be.
I'd recommend following what we do for numpy, which is according to NEP 29, support
"all minor versions of NumPy released in the prior 24 months from the anticipated release date with a minimum of 3 minor versions of NumPy", for arrow as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the challenge with offering a similar support window for the two libraries is that NumPy has a very stable ABI whereas PyArrow does not
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if I'm missing something, but sounds like what @lithomas1 is proposing is pretty much the same as what's written in the proposal but phrased in a different way. Has the proposal been updated, or am I misunderstanding that supporting the releases of the last 24 months, and supporting the highest/oldest version two years old?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I read this as every major release we will bump the min required version of pyarrow to the latest version, but might be misreading here.
Note that my proposed change would be different in that we would drop Arrow versions in both major/minor versions (as opposed to every major version), just like we do with numpy (once we reach the end of the NEP support window).
I might have missed some more discussion on this, but I thought we were going to restrict current usage of pyarrow to just what's exposed through Python.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To the latest version
that has been released for at least 2 years. So, the minimum PyArrow version we support will be around 24 months old, and we should be supporting all the versions since that one, so more or less the same policy as NumPy. @mroeschke not sure if it's easy to rephrase in a way that it's more obvious what's the policy.About bumping in major or minor releases, I don't have a preference, either is fine for me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can rephrase this to make it more clear but @datapythonista has it correct. The only distinction here, compared to what we do with numpy today, is that pyarrow would be bumped only during a pandas major release.
Under this proposal, PyArrow will only be used as a runtime dependency
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you consider this to upgrading pyarrow in both major and minor versions to be consistent with numpy?
I ask because it is probably tricky for downstream to predict the length of our major release cycle (for 2.0 I think we delayed it twice. IIRC 1.4 was supposed to be 2.0).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure that would be okay with me too