BUILD: Pandas 1.2.5 build no longer works

[adrianonobre]

Installation check

• I have read the installation guide.

Platform

macOS-14.5-arm64-arm-64bit

Installation Method

pip install

pandas Version

1.2.5

Python Version

Python 3.11.7

Installation Logs

pip install --no-cache six==1.16.0
pip install --no-cache pytz==2024.1
pip install --no-cache python-dateutil==2.9.0.post0
pip install --no-cache numpy==1.26.4
pip install --no-cache cython==0.29.21
# blows up
pip install --no-cache pandas==1.2.5 

  clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/ujson/python -Ipandas/_libs/src/ujson/lib -Ipandas/_libs/src/datetime -I/private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include -I/Users/adriano.nobre/dev/test/sample-venv/include -I/Users/adriano.nobre/.pyenv/versions/3.11.7/include/python3.11 -c pandas/_libs/src/ujson/lib/ultrajsonenc.c -o build/temp.macosx-13.6-arm64-cpython-311/pandas/_libs/src/ujson/lib/ultrajsonenc.o -D_GNU_SOURCE -Wno-error=unreachable-code
  clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/ujson/python -Ipandas/_libs/src/ujson/lib -Ipandas/_libs/src/datetime -I/private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include -I/Users/adriano.nobre/dev/test/sample-venv/include -I/Users/adriano.nobre/.pyenv/versions/3.11.7/include/python3.11 -c pandas/_libs/src/ujson/python/JSONtoObj.c -o build/temp.macosx-13.6-arm64-cpython-311/pandas/_libs/src/ujson/python/JSONtoObj.o -D_GNU_SOURCE -Wno-error=unreachable-code
  pandas/_libs/src/ujson/python/JSONtoObj.c:195:49: warning: incompatible pointer types passing 'PyObject *' (aka 'struct _object *') to parameter of type 'const PyArrayObject *' (aka 'const struct tagPyArrayObject_fields *') [-Wincompatible-pointer-types]
          new_data = PyDataMem_RENEW(PyArray_DATA(ret), i * npyarr->elsize);
                                                  ^~~
  /private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1508:35: note: passing argument to parameter 'arr' here
  PyArray_DATA(const PyArrayObject *arr)
                                    ^
  pandas/_libs/src/ujson/python/JSONtoObj.c:260:33: error: no member named 'elsize' in 'struct _PyArray_Descr'
          npyarr->elsize = dtype->elsize;
                           ~~~~~  ^
  pandas/_libs/src/ujson/python/JSONtoObj.c:305:53: warning: incompatible pointer types passing 'PyObject *' (aka 'struct _object *') to parameter of type 'const PyArrayObject *' (aka 'const struct tagPyArrayObject_fields *') [-Wincompatible-pointer-types]
              new_data = PyDataMem_RENEW(PyArray_DATA(npyarr->ret),
                                                      ^~~~~~~~~~~
  /private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1508:35: note: passing argument to parameter 'arr' here
  PyArray_DATA(const PyArrayObject *arr)
                                    ^
  pandas/_libs/src/ujson/python/JSONtoObj.c:316:18: warning: incompatible pointer types passing 'PyObject *' (aka 'struct _object *') to parameter of type 'const PyArrayObject *' (aka 'const struct tagPyArrayObject_fields *') [-Wincompatible-pointer-types]
      PyArray_DIMS(npyarr->ret)[0] = i + 1;
                   ^~~~~~~~~~~
  /private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1520:35: note: passing argument to parameter 'arr' here
  PyArray_DIMS(const PyArrayObject *arr)
                                    ^
  pandas/_libs/src/ujson/python/JSONtoObj.c:318:33: warning: incompatible pointer types passing 'PyObject *' (aka 'struct _object *') to parameter of type 'const PyArrayObject *' (aka 'const struct tagPyArrayObject_fields *') [-Wincompatible-pointer-types]
      if ((item = PyArray_GETPTR1(npyarr->ret, i)) == NULL ||
                                  ^~~~~~~~~~~
  /private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarrayobject.h:138:57: note: expanded from macro 'PyArray_GETPTR1'
  #define PyArray_GETPTR1(obj, i) ((void *)(PyArray_BYTES(obj) + \
                                                          ^~~
  /private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1514:36: note: passing argument to parameter 'arr' here
  PyArray_BYTES(const PyArrayObject *arr)
                                     ^
  pandas/_libs/src/ujson/python/JSONtoObj.c:318:33: warning: incompatible pointer types passing 'PyObject *' (aka 'struct _object *') to parameter of type 'const PyArrayObject *' (aka 'const struct tagPyArrayObject_fields *') [-Wincompatible-pointer-types]
      if ((item = PyArray_GETPTR1(npyarr->ret, i)) == NULL ||
                                  ^~~~~~~~~~~
  /private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarrayobject.h:139:62: note: expanded from macro 'PyArray_GETPTR1'
                                           (i)*PyArray_STRIDES(obj)[0]))
                                                               ^~~
  /private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1526:38: note: passing argument to parameter 'arr' here
  PyArray_STRIDES(const PyArrayObject *arr)
                                       ^
  pandas/_libs/src/ujson/python/JSONtoObj.c:319:25: warning: incompatible pointer types passing 'PyObject *' (aka 'struct _object *') to parameter of type 'PyArrayObject *' (aka 'struct tagPyArrayObject_fields *') [-Wincompatible-pointer-types]
          PyArray_SETITEM(npyarr->ret, item, value) == -1) {
                          ^~~~~~~~~~~
  /private/var/folders/gx/t9hqn79x4wdgbqy8n09y0bww0000gp/T/pip-build-env-74z_mupr/overlay/lib/python3.11/site-packages/numpy/_core/include/numpy/ndarrayobject.h:292:32: note: passing argument to parameter 'arr' here
  PyArray_SETITEM(PyArrayObject *arr, char *itemptr, PyObject *v)
                                 ^
  6 warnings and 1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pandas
Failed to build pandas
ERROR: Could not build wheels for pandas, which is required to install pyproject.toml-based projects

[adrianonobre]

potentially caused by this:
numpy/numpy@b3f9fc0

[myriky]

same issue

[lithomas1]

We are not planning on adding numpy 2 support to pandas 1.2.5 since numpy 2 compat is a pretty involved process and pandas 1.2.5 is already 3 years old at this point.

Please upgrade to pandas 2.2.2 as that is the first pandas version to support numpy 2.

[adrianonobre]

Hi @lithomas1, I think there's a misunderstanding (I'll take numpy 2.0 mention from the title)
I'm not trying to use numpy 2.
The issue is rather that pandas 1.2.5 build no longer works (with numpy 1.26, see notes in the "installation logs" section).

I was just pointing out that a change done for numpy 2 seems to have bled into numpy 1.26 as a breaking change, potentially affecting projects that depend on numpy 1.26

Thanks for looking into this

[lithomas1]

Sorry for the misunderstanding.

Did you mention that this started failing after numpy 2.0 was released?
(If so, this might because the build of pandas 2.0 is pulling the newest numpy, not the one you have installed)

[adrianonobre]

No worries! (and thanks again for your time, @lithomas1 )

Correct.
We noticed the pandas build started failing yesterday "out of the blue".
We've got dependencies versions pinned in our requirements file. We didn't make any changes to these versions. Here's a couple of them:
numpy==1.26.4
pandas==1.2.5

Investigating a bit we noticed that the NumPy project did a release release this week (2.0) and we found this change which seems to align with the error message we're getting in the pandas build (i.e. a missing struct attribute elsize):

      pandas/_libs/src/ujson/python/JSONtoObj.c:260:33: error: no member named 'elsize' in 'struct _PyArray_Descr'
              npyarr->elsize = dtype->elsize;
                               ~~~~~  ^

The easiest way a colleague found to repro this is as follows:

# make virtual env (make sure python is 3.11)
python -m venv ./sample-venv
# activate virtual env
. ./sample-venv/bin/activate
# install same packages we do in the project (relevant to pandas/numpy)
pip install --no-cache six==1.16.0
pip install --no-cache pytz==2024.1
pip install --no-cache python-dateutil==2.9.0.post0
pip install --no-cache numpy==1.26.4
pip install --no-cache cython==0.29.21
# blows up
pip install --no-cache pandas==1.2.5 

[lithomas1]

Can you try installing pandas with --no-build-isolation?

(You'll need all dependencies pre-installed, but this should force pip to use your numpy, and not pull its own numpy)

[mttr]

For what it's worth, this problem seems to be due to this line in pyproject.toml and can be fixed by changing it to "numpy<2; python_version>='3.9'"