Why is `JSON.rawJSON` limited to primitives only?

[airhorns]

Forgive me if this is the wrong spot to put this.

I think JSON.rawJSON is a really powerful API for performance-optimizing JSON serialization. But, because it is limited to only producing valid primitive JSON, it can't be used for "inline"-ing existing JSON.

I've got a couple use cases I want to use it for that requires feeding pre-serialized objects and arrays into the serialization of outer object trees. For example, in a typical REST API, you might retrieve 10 records from the database, and reply with one big JSON array of all of them. Each record might have a big JSON value on it, and if they are large, it performs poorly to de-serialize each record's JSON object to then just serialize it again to produce the REST API response holding all 10 records. Instead, it'd be great to leave the data as a string when fetching from the database, and then just insert it into the final JSON string produced by JSON.stringify using JSON.rawJSON to wrap each of these strings.

Without this capability, one has to resort to manually clobbering together JSON strings which is far less performant and correct than using the engine's built-in capabilities, or always deserializing just to serialize again. Userland implementations like json-stream-stringify are far, far slower, and at least in my case, the JSON objects are really big, so deserializing and reserializing is a major performance issue.

I presume there is a justification for limiting what can be go through a .rawJSON, but what is it? And, could there ever be a trusted mode, or some sort of escape hatch where for very performance sensitive use cases, any ole string could be sent along?

Also one other note: it seems that this low level API could really assist with performance optimization around avoiding re-serializing values you already have the source JSON string for, but as currently specified it can't because it does the safety check by parsing the string anyways. That seems correct but inefficient, again suggesting that it'd be great to have some sort of escape hatch for the brave. Notably, [[IsRawJSON]] being an internal slot means that userland can't create their own raw JSON objects and pay the complexity / reliability price.

[airhorns]

@gibson042 apologies for the direct ping but it'd be super helpful to understand this and/or collaborate on widening the applicability!

I ended up open sourcing the thing I would want to use rawJSON for here: https://github.com/gadget-inc/deferredjson

[gibson042]

Thanks for the ping. The reason for limiting to primitive values is cutting off what would otherwise be a bigger opportunity for surreptitious communication by varying representation details within JSON text representing the same data. See #12 (comment) , #19 (comment) , and also the extensive discussion at the October 2021 plenary that ultimately resulting in global availability with primitive-only constraints as a balance of convenience vs. integrity (the latter being a concern about the ability for an untrusted data-only input object to encode itself as arbitrary JSON text, originally raised in July 2020).

[SamB]

So the worry is that people would exfiltrate information through whitespace and/or repeated keys? Because as far as I can tell, toJSON already allows an object to replace itself with something completely different (or, more likely, something with some extra fields), and if that were a problem for some reason, rawJSON already gives us quite a few places to squeeze some information:

• For strings:

  • which characters to escape (only control characters, \, and " must be escaped)
  • in some cases, which escape to use

• For numbers:

  • what exponent to use
  • for integers, whether to use an exponent at all
  • whether to precede an exponent with e or E
  • whether or not to place a + before the digits of a positive exponent
  • trailing zeros in the fractional part
  • leading zeros in the exponent

Or, getting back to things possible even without this proposal, you could just reorder the fields in objects.

[gibson042]

You aren't wrong, and the door remains open for a later proposal to add non-primitive rawJSON.