GH-48473: [CI][Python] Require numpy 2.0 by alippai · Pull Request #50769 · apache/arrow

alippai · 2026-08-02T17:26:59Z

Rationale for this change

Require Numpy 2.0 to support newer features, allow newer numpy API/ABI

What changes are included in this PR?

Numpy 2.0+ requirement
Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
Pyodide 0.27.1 usage (numpy 2.0 api)

Are there any user-facing changes?

Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1

This PR includes breaking changes to public APIs.

This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.

GitHub Issue: [Python] Require numpy 2.x #48473

github-actions · 2026-08-02T17:27:24Z

⚠️ GitHub issue #48473 has been automatically assigned in GitHub to PR creator.

alippai · 2026-08-02T19:21:25Z

@tadeja @rok @raulcd @jorisvandenbossche this is ready for review. The failure seems to be an unrelated arrow flight timeout. Once this is on master, I'll file a new StringDType from/to conversion PR.

kou · 2026-08-03T05:20:54Z

@github-actions crossbow submit -g python

github-actions · 2026-08-03T05:25:25Z

Revision: d0a51cc

Submitted crossbow builds: ursacomputing/crossbow @ actions-764bbcb6f5

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hdfs-2.9.2
test-conda-python-3.11-hdfs-3.2.1
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-2.2.2-numpy-2.0.2
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-latest-numpy-2.1.0
test-conda-python-3.13-pandas-latest-numpy-latest
test-conda-python-3.14
test-conda-python-3.14-cpython-debug
test-conda-python-3.14-pandas-nightly-numpy-nightly
test-conda-python-3.14-pandas-upstream_devel-numpy-nightly
test-conda-python-emscripten
test-debian-13-python-3-amd64
test-debian-13-python-3-i386
test-fedora-42-python-3
test-ubuntu-24.04-python-3

raulcd

Thanks @alippai for working on this. Changes LGTM in general, let me lauch extra CI and get the ok from @pitrou @jorisvandenbossche @AlenkaF

raulcd · 2026-08-03T09:13:07Z

@github-actions crossbow submit wheel-cp313

github-actions · 2026-08-03T09:19:50Z

Revision: d0a51cc

Submitted crossbow builds: ursacomputing/crossbow @ actions-449ef0ee7b

Task	Status
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-musllinux-1-2-cp313-cp313-amd64
wheel-musllinux-1-2-cp313-cp313-arm64
wheel-windows-cp313-cp313-amd64

raulcd · 2026-08-03T10:11:55Z

@github-actions crossbow submit wheel-cp311

github-actions · 2026-08-03T10:14:47Z

Revision: d0a51cc

Submitted crossbow builds: ursacomputing/crossbow @ actions-012e7c747c

Task	Status
wheel-macos-monterey-cp311-cp311-amd64
wheel-macos-monterey-cp311-cp311-arm64
wheel-manylinux-2-28-cp311-cp311-amd64
wheel-manylinux-2-28-cp311-cp311-arm64
wheel-musllinux-1-2-cp311-cp311-amd64
wheel-musllinux-1-2-cp311-cp311-arm64
wheel-windows-cp311-cp311-amd64

rok

Thanks for working on this @alippai !

I found some other places where version checks occur 1, 2, 3, 4, 5. Could you take a look and see if we can simplify/remove those?

ngoldbaum

I asked an AI model to review this with an eye toward code that can be deleted to simplify interfacing with Pandas and it found several more places where you can delete or simplify code. I'm happy to take on these cleanups in a followup if you think my suggestions are scope creep.

alippai · 2026-08-03T16:26:48Z

@ngoldbaum @rok thanks, I’ll address the pandas upgrade too. It was addressing the numpy upgrade only. I believe pandas improvements are in scope for this MR. Addressing upstream fixes and even onboarding the optimal/newer APIs allowed by the new version should land together.

ngoldbaum

Just one last nit but this otherwise looks good to me!

jorisvandenbossche

Looks good to me!

…d docs.

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

alippai · 2026-08-12T14:46:13Z

@rok @raulcd @kou this is now ready again, two approvals added.

pandas also requires numpy 2.0 in the upcoming pandas 3.1: pandas-dev/pandas#63403 (comment)

kou · 2026-08-13T21:58:26Z

I'll merge this in the next week if nobody objects it.

raulcd · 2026-08-14T10:38:29Z

@github-actions crossbow submit -g python -g wheel

github-actions · 2026-08-14T10:43:18Z

Revision: 55efae1

Submitted crossbow builds: ursacomputing/crossbow @ actions-1ef568d46d

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
python-sdist
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hdfs-2.9.2
test-conda-python-3.11-hdfs-3.2.1
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-2.2.2-numpy-2.0.2
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-latest-numpy-2.1.3
test-conda-python-3.13-pandas-latest-numpy-latest
test-conda-python-3.14
test-conda-python-3.14-cpython-debug
test-conda-python-3.14-pandas-nightly-numpy-nightly
test-conda-python-3.14-pandas-upstream_devel-numpy-nightly
test-conda-python-emscripten
test-debian-13-python-3-amd64
test-debian-13-python-3-i386
test-fedora-42-python-3
test-ubuntu-24.04-python-3
wheel-macos-monterey-cp311-cp311-amd64
wheel-macos-monterey-cp311-cp311-arm64
wheel-macos-monterey-cp312-cp312-amd64
wheel-macos-monterey-cp312-cp312-arm64
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-macos-monterey-cp314-cp314-amd64
wheel-macos-monterey-cp314-cp314-arm64
wheel-macos-monterey-cp314-cp314t-amd64
wheel-macos-monterey-cp314-cp314t-arm64
wheel-manylinux-2-28-cp311-cp311-amd64
wheel-manylinux-2-28-cp311-cp311-arm64
wheel-manylinux-2-28-cp312-cp312-amd64
wheel-manylinux-2-28-cp312-cp312-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp314-cp314-amd64
wheel-manylinux-2-28-cp314-cp314-arm64
wheel-manylinux-2-28-cp314-cp314t-amd64
wheel-manylinux-2-28-cp314-cp314t-arm64
wheel-musllinux-1-2-cp311-cp311-amd64
wheel-musllinux-1-2-cp311-cp311-arm64
wheel-musllinux-1-2-cp312-cp312-amd64
wheel-musllinux-1-2-cp312-cp312-arm64
wheel-musllinux-1-2-cp313-cp313-amd64
wheel-musllinux-1-2-cp313-cp313-arm64
wheel-musllinux-1-2-cp314-cp314-amd64
wheel-musllinux-1-2-cp314-cp314-arm64
wheel-musllinux-1-2-cp314-cp314t-amd64
wheel-musllinux-1-2-cp314-cp314t-arm64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp314-cp314-amd64
wheel-windows-cp314-cp314t-amd64

raulcd

There seems to be a single test failure with upstream and nightly pandas but the rest LGTM. Thanks for the great job @alippai

raulcd · 2026-08-14T13:08:48Z

@@ -430,16 +429,14 @@ def test_backwards_compatible_index_multi_level_some_named(datadir):

 @pytest.mark.pandas
 def test_backwards_compatible_column_metadata_handling(datadir):


The failures seem related to the changes on test_backwards_compatible_column_metadata_handling. They didn't fail for the last nightly run:
https://github.com/ursacomputing/crossbow/actions/runs/31758047403/job/94638094168
Last nightly for test-conda-python-3.14-pandas-upstream_devel-numpy-nightly used: numpy 2.6.0.dev0+git20260812.d712d42 and pandas 3.1.0.dev0+1566.gf040cfd389
on this job: numpy 2.6.0.dev0+git20260812.d712d42 and pandas 3.1.0.dev0+1567.g7b0bc4da28

So there's a single pandas commit difference which seems unrelated:
pandas-dev/pandas@7b0bc4d

I think this is happening because of pandas-dev/pandas#65194. This test is skipped on main so this is the first time it's been hit. One way to paper over this is to pass check_freq=False to assert_frame_equal below.

@jorisvandenbossche should this maybe have been a warning instead of an error on the Pandas side?

For a normal (non-multi) index, pandas already raised before in assert_frame_equal (and so there the referenced PR is not changing anything), but for a MultiIndex I indeed see a change in behaviour, and ideally that should go through a deprecation warning as well.

But indeed, for here, passing check_freq=False should resolve it. Or alternatively add a ._with_freq(None) to the chain when creating the dates variable.

(reported as pandas-dev/pandas#66761)

alippai requested review from AlenkaF, assignUser, jonkeane, kou, pitrou, raulcd and rok as code owners August 2, 2026 17:27

github-actions Bot added Component: C++ Component: Python Component: Documentation awaiting review Awaiting review labels Aug 2, 2026

alippai force-pushed the alippai-numpy-20 branch 2 times, most recently from 459c7f0 to d0a51cc Compare August 2, 2026 18:42

raulcd reviewed Aug 3, 2026

View reviewed changes

Comment thread .github/workflows/python.yml

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 3, 2026

rok reviewed Aug 3, 2026

View reviewed changes

Comment thread python/pyarrow/tests/test_extension_type.py Outdated

ngoldbaum mentioned this pull request Aug 3, 2026

[Python] Conversion to/from numpy 2.0+ new StringDType #42018

Open

ngoldbaum reviewed Aug 3, 2026

View reviewed changes

alippai force-pushed the alippai-numpy-20 branch from cb5085f to d0a51cc Compare August 3, 2026 15:59

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 3, 2026

alippai requested review from ngoldbaum and rok August 3, 2026 23:54

alippai force-pushed the alippai-numpy-20 branch from 5c4831f to dc6ef99 Compare August 4, 2026 00:20

ngoldbaum approved these changes Aug 4, 2026

View reviewed changes

Comment thread python/pyarrow/types.pxi Outdated

alippai force-pushed the alippai-numpy-20 branch from dc6ef99 to b2ceece Compare August 4, 2026 23:41

alippai requested a review from raulcd August 4, 2026 23:44

jorisvandenbossche approved these changes Aug 6, 2026

View reviewed changes

Comment thread .github/workflows/python.yml

Comment thread dev/tasks/tasks.yml Outdated

github-actions Bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Aug 6, 2026

alippai added 7 commits August 11, 2026 19:46

apacheGH-48473: [CI][Python] Require numpy 2.0

a11c0cc

apacheGH-48473: Update Pyodide to 0.27.1

8ee2a31

Pandas 2.2.2 and numpy 2.0 upgrade followups

d22f524

Update pyodide to 0.27.1 in docker. Update pandas related comments an…

6780493

…d docs.

apacheGH-48473: [Python] Preserve pandas 2.2 string dtype guard

e26770e

apacheGH-48473: [Python] Pin legacy pandas test to nanoseconds

ad13bfd

apacheGH-48473: [Python] Update pandas type map comment

1a74543

alippai force-pushed the alippai-numpy-20 branch from 8ea5dc3 to 1a74543 Compare August 11, 2026 23:47

Use latest patch

55efae1

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

raulcd reviewed Aug 14, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Aug 14, 2026

		@@ -430,16 +429,14 @@ def test_backwards_compatible_index_multi_level_some_named(datadir):

		@pytest.mark.pandas
		def test_backwards_compatible_column_metadata_handling(datadir):

Uh oh!

Conversation

alippai commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Aug 2, 2026

Uh oh!

alippai commented Aug 2, 2026

Uh oh!

kou commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raulcd commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

raulcd commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

rok left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alippai commented Aug 3, 2026

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alippai commented Aug 12, 2026

Uh oh!

kou commented Aug 13, 2026

Uh oh!

raulcd commented Aug 14, 2026

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

raulcd Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

ngoldbaum Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

alippai commented Aug 2, 2026 •

edited

Loading