ENH: Migrate from ujson to orjson

[rhshadrach]

Ref: #62072

pandas has a vendored copy of ujson that has not been kept up to date. ujson itself recommends users should migrate to orjson, it seems to me pandas should as well. While this would mean users need another dependency to utilize JSON, it seems better from a maintenance perspective.

[mroeschke]

Could we just use pyarrow's json parser?

[jbrockmendel]

Is the idea to vendor orjson and adapt it like we have ujson? Or can we just use orjson directly?

[jorisvandenbossche]

Could we just use pyarrow's json parser?

FWIW pyarrow has no JSON writer, only reader, and also the reader only supports newline-delimited json. So at this point it could never be a full replacement (and also for the reader I don't know how it would compare in terms of flexibility handling all kinds of json structures. For example, we have the different orient options, which I suppose pyarrow won't support, given its focus on line-delimited json)

[rhshadrach]

@jbrockmendel it seems better to me to have it as an optional dependency. I haven't found any discussion on why we vendored ujson in the first place.

[jbrockmendel]

IIUC we modified the vendored code pretty extensively

[mroeschke]

Could we just use pyarrow's json parser?

Gotcha sounds like just using pyarrow for json parsing is a non-starter.

IIUC we modified the vendored code pretty extensively

Yeah I think the ujson parser is modified IIRC for NaN missing value handling. But overall, not philosophically opposed to using orjson (or any non vendored implementation) if behaviors say largely the same

[WillAyd]

I haven't found any discussion on why we vendored ujson in the first place.

ujson far pre-exists having Meson in place, so vendoring was the only sane option at that time

IIUC we modified the vendored code pretty extensively

Pretty much everything in pandas/_libs/src/vendored/ujson/python/objToJSON.c is custom. The rest of the base is mostly stock; there are changes compared to whenever we forked ujson 10+ years ago, but the divergence isn't largely material

orjson looks interesting. Most of the customizations we did to ujson have to do with serializing the object dtype and handling indices, so if there's a way to partially replace that with orjson that would be great

[rhshadrach]

The sense I'm getting is that we are open to switching to orjson and not vendoring if we can do so in a (mostly?) backwards-compatible way that is maintainable. Removing the Needs Discussion tag, this issue is open for investigation.

[WillAyd]

One other thing to keep note of, and likely the largest challenge, is the number of orients that our JSON formatter handles. I don't know much about orjson specifically, but this same conversation with other tools in the past ultimately stumbles there.

There might be a way to get rid of some of the formats (ex: values doesn't seem useful in a broad sense) which would help any transition