Skip to content

BUG: fix memory leak in JSON datetime serialization #62217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Sep 3, 2025
Merged

BUG: fix memory leak in JSON datetime serialization #62217

Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
791bb41
fix: memory leak in JSON datetime serialization
Alvaro-Kothe Aug 29, 2025
f7b2f5b
fix: flag to free `cStr` in `Object_getBigNumStringValue`
Alvaro-Kothe Aug 29, 2025
f324bd8
fix: use dynamic allocated memory to store strings
Alvaro-Kothe Sep 2, 2025
15c19c0
fix: replace PyObject_Malloc with PyMem_Malloc
Alvaro-Kothe Sep 2, 2025
1ea0283
fix: drop `const` qualifier in `cStr`
Alvaro-Kothe Sep 2, 2025
0925e6b
fix: remove void pointer case in PyMem_Free
Alvaro-Kothe Sep 2, 2025
62b0e26
fix: create const for default `cStr` size
Alvaro-Kothe Sep 2, 2025
0f20b7d
try to fix ci error
Alvaro-Kothe Sep 2, 2025
05f4926
hack: go back to use `PyObject_*` because of CI errors
Alvaro-Kothe Sep 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion pandas/_libs/src/vendored/ujson/python/objToJSON.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ typedef struct __TypeContext {
JSINT64 longValue;

const char *cStr;
int freeCStr;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally the problem with the JSON code is that over time we have incrementally added many different state management operations, and it is not always clear how they should work in tandem. So I am a little hesitant to add more state to solve the current issue. Is it possible to solve the issue without changing this struct?

Copy link
Contributor

@swt2c swt2c Sep 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any suggestions on how to do that? Revert the changes in fb6c4e3 that make some of the uses of cStr stack allocated? It doesn't seem possible to make all uses of cStr stack allocated, at least at first glance.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure that might be a reasonable approach

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the freeCStr flag and stored all the strings assigned in *IterNext in dynamic memory.

NpyArrContext *npyarr;
PdBlockContext *pdblock;
int transpose;
Expand Down Expand Up @@ -162,6 +163,7 @@ static TypeContext *createTypeContext(void) {
pc->longValue = 0;
pc->doubleValue = 0.0;
pc->cStr = NULL;
pc->freeCStr = 0;
pc->npyarr = NULL;
pc->pdblock = NULL;
pc->rowLabels = NULL;
Expand Down Expand Up @@ -327,13 +329,15 @@ static const char *NpyDateTimeToIsoCallback(JSOBJ Py_UNUSED(unused),
NPY_DATETIMEUNIT base = ((PyObjectEncoder *)tc->encoder)->datetimeUnit;
NPY_DATETIMEUNIT valueUnit = ((PyObjectEncoder *)tc->encoder)->valueUnit;
GET_TC(tc)->cStr = int64ToIso(GET_TC(tc)->longValue, valueUnit, base, len);
GET_TC(tc)->freeCStr = 1;
return GET_TC(tc)->cStr;
}

/* JSON callback. returns a char* and mutates the pointer to *len */
static const char *NpyTimeDeltaToIsoCallback(JSOBJ Py_UNUSED(unused),
JSONTypeContext *tc, size_t *len) {
GET_TC(tc)->cStr = int64ToIsoDuration(GET_TC(tc)->longValue, len);
GET_TC(tc)->freeCStr = 1;
return GET_TC(tc)->cStr;
}

Expand All @@ -347,7 +351,9 @@ static const char *PyDateTimeToIsoCallback(JSOBJ obj, JSONTypeContext *tc,
}

NPY_DATETIMEUNIT base = ((PyObjectEncoder *)tc->encoder)->datetimeUnit;
return PyDateTimeToIso(obj, base, len);
GET_TC(tc)->cStr = PyDateTimeToIso(obj, base, len);
GET_TC(tc)->freeCStr = 1;
return GET_TC(tc)->cStr;
}

static const char *PyTimeToJSON(JSOBJ _obj, JSONTypeContext *tc,
Expand Down Expand Up @@ -1880,6 +1886,9 @@ static void Object_endTypeContext(JSOBJ Py_UNUSED(obj), JSONTypeContext *tc) {
GET_TC(tc)->rowLabels = NULL;
NpyArr_freeLabels(GET_TC(tc)->columnLabels, GET_TC(tc)->columnLabelsLen);
GET_TC(tc)->columnLabels = NULL;
if (GET_TC(tc)->freeCStr) {
PyObject_Free((void *)GET_TC(tc)->cStr);
}
GET_TC(tc)->cStr = NULL;
PyObject_Free(tc->prv);
tc->prv = NULL;
Expand All @@ -1906,6 +1915,7 @@ static const char *Object_getBigNumStringValue(JSOBJ obj, JSONTypeContext *tc,
char *bytes = PyObject_Malloc(*_outLen + 1);
memcpy(bytes, str, *_outLen + 1);
GET_TC(tc)->cStr = bytes;
GET_TC(tc)->freeCStr = 1;

Py_DECREF(repr);

Expand Down
Loading