BUG: "Field type has incompatible types" in loading GCS parquet with read_parquet

[legiondean]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import gcsfs
fs = gcsfs.GCSFileSystem(project="***")
path = 'gs://***/2025-09-01T00_15_30_events.parquet'
df = pd.read_parquet(path, filesystem=fs)

Issue Description

When parquet files contain logical type like map, read_parquet behaviors differently on local parquet file and gcs file.
The same parquet file can be loaded as pandas df from local, but got error when load from gcs:
ArrowTypeError: Unable to merge: Field type has incompatible types: binary vs dictionary<values=string, indices=int32, ordered=0>

load from local file:

import pandas as pd
path = '2025-09-01T00_15_30_events.parquet'
df = pd.read_parquet(path)
print("Row count:", len(df))
---------------------------------------------------------------------------
Row count: 327827

load from gcs:

import pandas as pd
import gcsfs
fs = gcsfs.GCSFileSystem(project="***")
path = 'gs://***/2025-09-01T00_15_30_events.parquet'
df = pd.read_parquet(path, filesystem=fs)
---------------------------------------------------------------------------
ArrowTypeError                            Traceback (most recent call last)
Cell In[6], line 5
      3 fs = gcsfs.GCSFileSystem(project="***")
      4 path = 'gs://****/2025-09-01T00_15_30_events.parquet'
----> 5 df = pd.read_parquet(path, filesystem=fs)

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pandas/io/parquet.py:669, in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, dtype_backend, filesystem, filters, **kwargs)
    666     use_nullable_dtypes = False
    667 check_dtype_backend(dtype_backend)
--> 669 return impl.read(
    670     path,
    671     columns=columns,
    672     filters=filters,
    673     storage_options=storage_options,
    674     use_nullable_dtypes=use_nullable_dtypes,
    675     dtype_backend=dtype_backend,
    676     filesystem=filesystem,
    677     **kwargs,
    678 )

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pandas/io/parquet.py:265, in PyArrowImpl.read(self, path, columns, filters, use_nullable_dtypes, dtype_backend, storage_options, filesystem, **kwargs)
    258 path_or_handle, handles, filesystem = _get_path_or_handle(
    259     path,
    260     filesystem,
...
File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()

ArrowTypeError: Unable to merge: Field type has incompatible types: binary vs dictionary<values=string, indices=int32, ordered=0>

Expected Behavior

The read_parquet should be able to load the same GCS parquet file same as loading from local file.

Installed Versions

INSTALLED VERSIONS ------------------ commit : c888af6 python : 3.13.3 python-bits : 64 OS : Darwin OS-release : 24.6.0 Version : Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:40 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6041 machine : arm64 processor : arm byteorder : little LC_ALL : None LANG : None LOCALE : en_US.UTF-8

pandas : 2.3.1
numpy : 2.3.2
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 25.2
Cython : None
sphinx : None
IPython : 9.5.0
adbc-driver-postgresql: None
...
zstandard : 0.23.0
tzdata : 2025.2
qtpy : None
pyqt5 : None

[Alvaro-Kothe]

Does it work if you use

df = pd.read_parquet(path, storage_options=fs.storage_options)

[legiondean]

Does it work if you use

df = pd.read_parquet(path, storage_options=fs.storage_options)

Still the same error:

---------------------------------------------------------------------------
ArrowTypeError                            Traceback (most recent call last)
Cell In[2], line 5
      3 fs = gcsfs.GCSFileSystem(project="***")
      4 path = 'gs:/***/events.parquet'
----> 5 df = pd.read_parquet(path, storage_options=fs.storage_options)

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pandas/io/parquet.py:669, in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, dtype_backend, filesystem, filters, **kwargs)
    666     use_nullable_dtypes = False
    667 check_dtype_backend(dtype_backend)
--> 669 return impl.read(
    670     path,
    671     columns=columns,
    672     filters=filters,
    673     storage_options=storage_options,
    674     use_nullable_dtypes=use_nullable_dtypes,
    675     dtype_backend=dtype_backend,
    676     filesystem=filesystem,
    677     **kwargs,
    678 )

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pandas/io/parquet.py:265, in PyArrowImpl.read(self, path, columns, filters, use_nullable_dtypes, dtype_backend, storage_options, filesystem, **kwargs)
    258 path_or_handle, handles, filesystem = _get_path_or_handle(
    259     path,
    260     filesystem,
...
File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()

ArrowTypeError: Unable to merge: Field type has incompatible types: binary vs dictionary<values=string, indices=int32, ordered=0>

[Alvaro-Kothe]

I can't reproduce on main, nor on 2.3.1. I uploaded a parquet file to gcs and tested this script:

import pandas as pd
import gcsfs

bucket = "my-bucket"
file_name = "test.parquet"
path = f"gcs://{bucket}/{file_name}"

fs = gcsfs.GCSFileSystem()
print(fs.ls(bucket))
# ['my-bucket/test.parquet']

df = pd.read_parquet(path, engine="pyarrow", filesystem=fs)
print(df)
#    a   b
# 0  1   2
# 1  2  22

You are probably missing some dependency. The most important one is fsspec. Here is my versions:

INSTALLED VERSIONS

commit : f538741
python : 3.11.13
python-bits : 64
OS : Linux
OS-release : 6.16.8-200.fc42.x86_64
Version : #1 SMP PREEMPT_DYNAMIC Fri Sep 19 17:47:18 UTC 2025
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : pt_BR.UTF-8
LOCALE : pt_BR.UTF-8

pandas : 2.2.0
numpy : 2.3.3
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 25.2
Cython : 3.1.4
sphinx : 8.2.3
IPython : 9.5.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.14.0
blosc : None
bottleneck : 1.6.0
dataframe-api-compat : None
fastparquet : 2024.11.0
fsspec : 2025.9.0
html5lib : 1.1
hypothesis : 6.140.2
gcsfs : 2025.9.0
jinja2 : 3.1.6
lxml.etree : 6.0.2
matplotlib : 3.10.6
numba : 0.62.0
numexpr : 2.13.0
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : 2.9.10
pymysql : 1.4.6
pyarrow : 21.0.0
pyreadstat : 1.3.1
pytest : 8.4.2
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2025.9.0
scipy : 1.16.2
sqlalchemy : 2.0.43
tables : 3.10.2
tabulate : 0.9.0
xarray : 2025.9.0
xlrd : 2.0.2
xlsxwriter : 3.2.9
zstandard : 0.25.0
tzdata : 2025.2
qtpy : None
pyqt5 : None

[Alvaro-Kothe]

If you still can't read using the file uri, you can also get a file handler from gcsfs and use pandas to read it:

with fs.open(f"{bucket}/{file_name}") as fh:
    df = pd.read_parquet(fh, engine="pyarrow")

[legiondean]

If you still can't read using the file uri, you can also get a file handler from gcsfs and use pandas to read it:

with fs.open(f"{bucket}/{file_name}") as fh:
df = pd.read_parquet(fh, engine="pyarrow")

Thanks @Alvaro-Kothe , the with fs.open(f"{bucket}/{file_name}") as fh: does work.
From what I see, under the hood of pd.read_parquet(, it's also with fs.open that streaming the file. But pd.read_parquet( does throw the above error with the same gcs parquet. This confused me. Maybe something is off in there.

On another hand, other frameworks like Dask also use pd.read_parquet( to load parquets. That means I can't use Dask to handle large files until this issue is resolved.

[legiondean]

I can't reproduce on main, nor on 2.3.1. I uploaded a parquet file to gcs and tested this script:

import pandas as pd
import gcsfs

bucket = "my-bucket"
file_name = "test.parquet"
path = f"gcs://{bucket}/{file_name}"

fs = gcsfs.GCSFileSystem()
print(fs.ls(bucket))

['my-bucket/test.parquet']

df = pd.read_parquet(path, engine="pyarrow", filesystem=fs)
print(df)

a b

0 1 2

1 2 22

You are probably missing some dependency. The most important one is fsspec. Here is my versions:

I also tested against parquets which only have simple columns, not able to produce the error either.
The issue occur only with partitioned parquets that contain complex column types like

identifier ROW(
    wrt VARCHAR,
    cuid VARCHAR,
    ipv4 INTEGER,
    user_agent VARCHAR,
    account_info ROW(
      hashed_username VARCHAR,
      previous_order_count INTEGER,
      signup_method VARCHAR
    ),
    operating_system VARCHAR,

We generated those parquet files via spark or trino (probably doesn't matter).

[legiondean]

The fsspec is available in my env. Also as mentioned above, the same parquet file has no error if it's downloaded to local to process.

import pandas as pd

pd.show_versions()

INSTALLED VERSIONS
------------------
commit                : c888af6d0bb674932007623c0867e1fbd4bdc2c6
python                : 3.13.3
python-bits           : 64
OS                    : Darwin
OS-release            : 24.6.0
Version               : Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:40 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6041
machine               : arm64
processor             : arm
byteorder             : little
LC_ALL                : None
LANG                  : None
LOCALE                : en_US.UTF-8

pandas                : 2.3.1
numpy                 : 2.3.2
pytz                  : 2025.2
dateutil              : 2.9.0.post0
pip                   : 25.2
Cython                : None
sphinx                : None
IPython               : 9.5.0
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.13.4
blosc                 : None
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : None
fsspec                : 2025.9.0
html5lib              : None
hypothesis            : None
gcsfs                 : 2025.9.0
jinja2                : 3.1.6
lxml.etree            : None
matplotlib            : 3.10.6
numba                 : None
numexpr               : None
odfpy                 : None
openpyxl              : None
pandas_gbq            : None
psycopg2              : 2.9.10
pymysql               : None
pyarrow               : 19.0.1
pyreadstat            : None
pytest                : None
python-calamine       : None
pyxlsb                : None
s3fs                  : None
scipy                 : None
sqlalchemy            : 2.0.43
tables                : None
tabulate              : None
xarray                : None
xlrd                  : None
xlsxwriter            : None
zstandard             : 0.23.0
tzdata                : 2025.2
qtpy                  : None
pyqt5                 : None

[jorisvandenbossche]

@legiondean this is a bizarre issue, and it is definitely a bug if reading a set of Parquet files locally vs from remote cloud gives a different result (I assume the issue might rather be in pyarrow though).

To try to reproduce it, would it be possible to share the data files? I don't know if they contain private data, but otherwise you could try to generate some random data but that has similar characteristics (with the more complex types) that still reproduces the issue.

Also, could you show the entire error traceback? (the one you posted in the top comment has a part trimmed out, which might be interesting to see where the error is coming from)

[legiondean]

@legiondean this is a bizarre issue, and it is definitely a bug if reading a set of Parquet files locally vs from remote cloud gives a different result (I assume the issue might rather be in pyarrow though).

To try to reproduce it, would it be possible to share the data files? I don't know if they contain private data, but otherwise you could try to generate some random data but that has similar characteristics (with the more complex types) that still reproduces the issue.

Also, could you show the entire error traceback? (the one you posted in the top comment has a part trimmed out, which might be interesting to see where the error is coming from)

Sorry, can't reproduce the issue yet with manually created dum parquets. But I found something very interesting: the error is related to both py env and also the gcs url. This makes deps gcsfs more suspicious.
the read_parquet works totally fine with this env:

import pandas as pd
path = 'gs://bucket/test/70eaf1b0db4a.parquet'
path = 'gs://bucket/test/hdfsloader/queues/sdk/error_events/completed/report_date=2025-09-29/build=2620/type=WEB_VIEW_ERROR/prod_00990248-03d0-4b37-a87e-70eaf1b0db4a.parquet'

df_output = pd.read_parquet(path)
pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : 965ceca9fd796940050d6fc817707bba1c4f9bff
python           : 3.11.13.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 24.6.0
Version          : Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:40 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6041
machine          : arm64
processor        : arm
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : en_US.UTF-8

pandas           : 2.0.2
numpy            : 1.25.0
pytz             : 2022.6
dateutil         : 2.8.2
setuptools       : 67.8.0
pip              : 25.2
Cython           : None
pytest           : 7.2.0
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : 3.0.3
lxml.etree       : None
html5lib         : None
pymysql          : 1.0.2
psycopg2         : None
jinja2           : 3.1.6
IPython          : 8.18.0
pandas_datareader: None
bs4              : None
bottleneck       : None
brotli           : None
fastparquet      : None
fsspec           : 2025.5.1
gcsfs            : 2025.5.1
matplotlib       : 3.10.6
numba            : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 13.0.0
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : 1.10.1
snappy           : None
sqlalchemy       : 1.4.44
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
zstandard        : 0.23.0
tzdata           : 2025.2
qtpy             : None
pyqt5            : None

But with following env, the gcs url matters. The same gcs parquet file with a short url works fine, but show errors when the path get longer:
env:

pd.show_versions()


INSTALLED VERSIONS
------------------
commit                : c888af6d0bb674932007623c0867e1fbd4bdc2c6
python                : 3.13.3
python-bits           : 64
OS                    : Darwin
OS-release            : 24.6.0
Version               : Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:40 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6041
machine               : arm64
processor             : arm
byteorder             : little
LC_ALL                : None
LANG                  : None
LOCALE                : en_US.UTF-8

pandas                : 2.3.1
numpy                 : 2.3.2
pytz                  : 2025.2
dateutil              : 2.9.0.post0
pip                   : 25.2
Cython                : None
sphinx                : None
IPython               : 9.5.0
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.13.4
blosc                 : None
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : None
fsspec                : 2025.9.0
html5lib              : None
hypothesis            : None
gcsfs                 : 2025.9.0
jinja2                : 3.1.6
lxml.etree            : None
matplotlib            : 3.10.6
numba                 : None
numexpr               : None
odfpy                 : None
openpyxl              : None
pandas_gbq            : None
psycopg2              : 2.9.10
pymysql               : None
pyarrow               : 19.0.1
pyreadstat            : None
pytest                : None
python-calamine       : None
pyxlsb                : None
s3fs                  : None
scipy                 : None
sqlalchemy            : 2.0.43
tables                : None
tabulate              : None
xarray                : None
xlrd                  : None
xlsxwriter            : None
zstandard             : 0.23.0
tzdata                : 2025.2
qtpy                  : None
pyqt5                 : None

error

import pandas as pd

# works fine
path = 'gs://bucket/test/70eaf1b0db4a.parquet'

# throw errors
path = 'gs://bucket/test/hdfsloader/queues/sdk/error_events/completed/report_date=2025-09-29/build=2620/type=WEB_VIEW_ERROR/prod_00990248-03d0-4b37-a87e-70eaf1b0db4a.parquet'

df = pd.read_parquet(path, engine="pyarrow")

---------------------------------------------------------------------------
ArrowTypeError                            Traceback (most recent call last)
Cell In[45], line 18
     16 # path = 'gs://bucket/test/hdfsloader/queues/sdk/error_events/completed/report_date=2025-09-29/build=2620/type=WEB_VIEW_ERROR/prod_00990248-03d0-4b37-a87e-70eaf1b0db4a.parquet'
     17 path = 'gs://bucket/test/hdfsloader/queues/sdk/error_events/completed/report_date=2025-09-29/build=2620/type=WEB_VIEW_ERROR/prod_00990248-03d0-4b37-a87e-70eaf1b0db4a.parquet'
---> 18 df = pd.read_parquet(path, engine="pyarrow")
     19 # df.info(verbose=True)

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pandas/io/parquet.py:669, in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, dtype_backend, filesystem, filters, **kwargs)
    666     use_nullable_dtypes = False
    667 check_dtype_backend(dtype_backend)
--> 669 return impl.read(
    670     path,
    671     columns=columns,
    672     filters=filters,
    673     storage_options=storage_options,
    674     use_nullable_dtypes=use_nullable_dtypes,
    675     dtype_backend=dtype_backend,
    676     filesystem=filesystem,
    677     **kwargs,
    678 )

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pandas/io/parquet.py:265, in PyArrowImpl.read(self, path, columns, filters, use_nullable_dtypes, dtype_backend, storage_options, filesystem, **kwargs)
    258 path_or_handle, handles, filesystem = _get_path_or_handle(
    259     path,
    260     filesystem,
    261     storage_options=storage_options,
    262     mode="rb",
    263 )
    264 try:
--> 265     pa_table = self.api.parquet.read_table(
    266         path_or_handle,
    267         columns=columns,
    268         filesystem=filesystem,
    269         filters=filters,
    270         **kwargs,
    271     )
    273     with catch_warnings():
    274         filterwarnings(
    275             "ignore",
    276             "make_block is deprecated",
    277             DeprecationWarning,
    278         )

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/parquet/core.py:1793, in read_table(source, columns, use_threads, schema, use_pandas_metadata, read_dictionary, memory_map, buffer_size, partitioning, filesystem, filters, use_legacy_dataset, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, decryption_properties, thrift_string_size_limit, thrift_container_size_limit, page_checksum_verification)
   1787     warnings.warn(
   1788         "Passing 'use_legacy_dataset' is deprecated as of pyarrow 15.0.0 "
   1789         "and will be removed in a future version.",
   1790         FutureWarning, stacklevel=2)
   1792 try:
-> 1793     dataset = ParquetDataset(
   1794         source,
   1795         schema=schema,
   1796         filesystem=filesystem,
   1797         partitioning=partitioning,
   1798         memory_map=memory_map,
   1799         read_dictionary=read_dictionary,
   1800         buffer_size=buffer_size,
   1801         filters=filters,
   1802         ignore_prefixes=ignore_prefixes,
   1803         pre_buffer=pre_buffer,
   1804         coerce_int96_timestamp_unit=coerce_int96_timestamp_unit,
   1805         decryption_properties=decryption_properties,
   1806         thrift_string_size_limit=thrift_string_size_limit,
   1807         thrift_container_size_limit=thrift_container_size_limit,
   1808         page_checksum_verification=page_checksum_verification,
   1809     )
   1810 except ImportError:
   1811     # fall back on ParquetFile for simple cases when pyarrow.dataset
   1812     # module is not available
   1813     if filters is not None:

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/parquet/core.py:1371, in ParquetDataset.__init__(self, path_or_paths, filesystem, schema, filters, read_dictionary, memory_map, buffer_size, partitioning, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, decryption_properties, thrift_string_size_limit, thrift_container_size_limit, page_checksum_verification, use_legacy_dataset)
   1367 if partitioning == "hive":
   1368     partitioning = ds.HivePartitioning.discover(
   1369         infer_dictionary=True)
-> 1371 self._dataset = ds.dataset(path_or_paths, filesystem=filesystem,
   1372                            schema=schema, format=parquet_format,
   1373                            partitioning=partitioning,
   1374                            ignore_prefixes=ignore_prefixes)

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/dataset.py:794, in dataset(source, schema, format, filesystem, partitioning, partition_base_dir, exclude_invalid_files, ignore_prefixes)
    783 kwargs = dict(
    784     schema=schema,
    785     filesystem=filesystem,
   (...)    790     selector_ignore_prefixes=ignore_prefixes
    791 )
    793 if _is_path_like(source):
--> 794     return _filesystem_dataset(source, **kwargs)
    795 elif isinstance(source, (tuple, list)):
    796     if all(_is_path_like(elem) or isinstance(elem, FileInfo) for elem in source):

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/dataset.py:486, in _filesystem_dataset(source, schema, filesystem, partitioning, format, partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
    478 options = FileSystemFactoryOptions(
    479     partitioning=partitioning,
    480     partition_base_dir=partition_base_dir,
    481     exclude_invalid_files=exclude_invalid_files,
    482     selector_ignore_prefixes=selector_ignore_prefixes
    483 )
    484 factory = FileSystemDatasetFactory(fs, paths_or_selector, format, options)
--> 486 return factory.finish(schema)

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/_dataset.pyx:3138, in pyarrow._dataset.DatasetFactory.finish()

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File ~/Apps/python_virtual_envs/cursor/lib/python3.13/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()

ArrowTypeError: Unable to merge: Field type has incompatible types: binary vs dictionary<values=string, indices=int32, ordered=0>