Skip to content

BUG: incorrect string parsing leads to segfault #62617

@leolavaur

Description

@leolavaur

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import io
import pandas as pd
pd.read_csv(
    io.StringIO("data1,81e3104049863b72,data3"),
    names=["header1", "header2", "header3"]
)

Issue Description

I tracked down a segfault to the provided snippet when parsing big raw CSVs. One of the values (here under "header2") is a hexadecimal 16-character string, and under some conditions, it cannot be parsed by the C engine, yielding a segfault. Pyarrow fails as well, but the Python engine succeeds.

I tested multiple variations of the issue (as illustrated by the examples below). My hypothesis is that the C engine tries to cast it as an integer written in scientific notation, but fails. I have not been able to pinpoint exactly the conditions in the time I spent debugging, and this is the only line that failed in the 35M lines that I parsed for this project.

The issue appears in the latest pandas version, but not in the main branch.

import io

import pandas as pd

print("works (only 0-9 after e)")
pd.read_csv(
    io.StringIO("data1,81e310404986372,data3"),
    names=[
        "header1",
        "header2",
        "header3",
    ],
)

print("works (only 9 chars between e and b)")
pd.read_csv(
    io.StringIO("data1,81e310404986b72,data3"),
    names=[
        "header1",
        "header2",
        "header3",
    ],
)

print("works (casted to string)")
pd.read_csv(
    io.StringIO("data1,81e3104049863b72,data3"),
    names=[
        "header1",
        "header2",
        "header3",
    ],
    dtype={"header2": str},
)

print("segfault")
pd.read_csv(
    io.StringIO("data1,81e3104049863b72,data3"),
    names=[
        "header1",
        "header2",
        "header3",
    ],
)
works (only 0-9 after e)
works (only 9 chars between e and b)
works (casted to string)
segfault
zsh: segmentation fault (core dumped)  python tests/test_segfault.py

Expected Behavior

No segfault :)

Installed Versions

INSTALLED VERSIONS

commit : 9c8bc3e
python : 3.12.10
python-bits : 64
OS : Linux
OS-release : 6.14.0-33-generic
Version : #33~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 17:02:30 UTC 2
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.3.3
numpy : 1.26.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : None
Cython : None
sphinx : None
IPython : 9.6.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2025.9.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : None
matplotlib : 3.10.6
numba : 0.62.1
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 21.0.0
pyreadstat : None
pytest : 8.4.2
python-calamine : None
pyxlsb : None
s3fs : 2025.9.0
scipy : 1.16.2
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : 2025.9.1
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2025.2
qtpy : None
pyqt5 : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugNeeds TriageIssue that has not been reviewed by a pandas team member

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions