-
-
Notifications
You must be signed in to change notification settings - Fork 19.1k
Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import io
import pandas as pd
pd.read_csv(
io.StringIO("data1,81e3104049863b72,data3"),
names=["header1", "header2", "header3"]
)
Issue Description
I tracked down a segfault to the provided snippet when parsing big raw CSVs. One of the values (here under "header2") is a hexadecimal 16-character string, and under some conditions, it cannot be parsed by the C engine, yielding a segfault. Pyarrow fails as well, but the Python engine succeeds.
I tested multiple variations of the issue (as illustrated by the examples below). My hypothesis is that the C engine tries to cast it as an integer written in scientific notation, but fails. I have not been able to pinpoint exactly the conditions in the time I spent debugging, and this is the only line that failed in the 35M lines that I parsed for this project.
The issue appears in the latest pandas version, but not in the main branch.
import io
import pandas as pd
print("works (only 0-9 after e)")
pd.read_csv(
io.StringIO("data1,81e310404986372,data3"),
names=[
"header1",
"header2",
"header3",
],
)
print("works (only 9 chars between e and b)")
pd.read_csv(
io.StringIO("data1,81e310404986b72,data3"),
names=[
"header1",
"header2",
"header3",
],
)
print("works (casted to string)")
pd.read_csv(
io.StringIO("data1,81e3104049863b72,data3"),
names=[
"header1",
"header2",
"header3",
],
dtype={"header2": str},
)
print("segfault")
pd.read_csv(
io.StringIO("data1,81e3104049863b72,data3"),
names=[
"header1",
"header2",
"header3",
],
)
works (only 0-9 after e)
works (only 9 chars between e and b)
works (casted to string)
segfault
zsh: segmentation fault (core dumped) python tests/test_segfault.py
Expected Behavior
No segfault :)
Installed Versions
INSTALLED VERSIONS
commit : 9c8bc3e
python : 3.12.10
python-bits : 64
OS : Linux
OS-release : 6.14.0-33-generic
Version : #33~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 17:02:30 UTC 2
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.3.3
numpy : 1.26.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : None
Cython : None
sphinx : None
IPython : 9.6.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2025.9.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : None
matplotlib : 3.10.6
numba : 0.62.1
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 21.0.0
pyreadstat : None
pytest : 8.4.2
python-calamine : None
pyxlsb : None
s3fs : 2025.9.0
scipy : 1.16.2
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : 2025.9.1
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2025.2
qtpy : None
pyqt5 : None