Expose Turbo Stream Plugins Options

[zwhitchcox]

It would be really great if we could do something like:

import { Prisma } from "prisma-client";
import type { DecodePlugin, EncodePlugin } from "turbo-stream";

const decimalEncodePlugin: EncodePlugin = (value) => {
  if (value instanceof Prisma.Decimal) {
    return ["Decimal", value.toString()];
  }
};

const decimalDecodePlugin: DecodePlugin = (type, ...values) => {
  if (type === "Decimal") {
    return { value: new Prisma.Decimal(values[0] as string) };
  }
};

export function handleRequest(request, responseStatusCode, responseHeaders, routerContext) {
  return handleRequest(request, responseStatusCode, responseHeaders, routerContext, {
    encodePlugins: [decimalEncodePlugin],
    decodePlugins: [decimalDecodePlugin],
  });
}

and pass our own decoders/encoders to turbo-stream.

There was a previous discussion that I cannot find where the answer was that loaders are going away because of RSC which would handle types like Decimal, but the docs say that there are only plans to support the types supported by RSC, so it doesn't look like RSC will have an answer for this.

Eventually, decimal might become a javascript primitive, but until then, we have to either:

1. patch react-router (current approach):

diff --git a/dist/development/chunk-PMGK554W.mjs b/dist/development/chunk-PMGK554W.mjs
index 3f978cb2c5defdc4bd2dabc2abd0b6b2c7d7a6c6..93ad52d1ccda994a9e521552bee138fb2b587975 100644
--- a/dist/development/chunk-PMGK554W.mjs
+++ b/dist/development/chunk-PMGK554W.mjs
@@ -966,6 +966,7 @@ function encodeViaTurboStream(data2, requestSignal, streamTimeout, serverMode) {
   return encode(data2, {
     signal: controller.signal,
     plugins: [
+      ...(global.TURBO_STREAM_EXTRA_ENCODE_PLUGINS ?? []),
       (value) => {
         if (value instanceof Error) {
           let { name, message, stack } = serverMode === "production" /* Production */ ? sanitizeError(value, serverMode) : value;
diff --git a/dist/development/chunk-RBZI3ZHD.js b/dist/development/chunk-RBZI3ZHD.js
index 5616d55dab48271d93eef8f6868ad630677def0f..e04ce501eb9af301c0e79531cc7cb16ed74f0de1 100644
--- a/dist/development/chunk-RBZI3ZHD.js
+++ b/dist/development/chunk-RBZI3ZHD.js
@@ -6408,6 +6408,7 @@ async function fetchAndDecodeViaTurboStream(args, basename, targetRoutes) {
 function decodeViaTurboStream(body, global) {
   return decode(body, {
     plugins: [
+      ...(window.TURBO_STREAM_EXTRA_DECODE_PLUGINS ?? []),
       (type, ...rest) => {
         if (type === "SanitizedError") {
           let [name, message, stack] = rest;
diff --git a/dist/development/chunk-WWGJGFF6.mjs b/dist/development/chunk-WWGJGFF6.mjs
index b0a48d4ac32c74f80db3f32522e29d7e8960d8dc..341e80cfe6af94d9af3fa51a55a420d8eec80afa 100644
--- a/dist/development/chunk-WWGJGFF6.mjs
+++ b/dist/development/chunk-WWGJGFF6.mjs
@@ -7899,6 +7899,7 @@ async function fetchAndDecodeViaTurboStream(args, basename, targetRoutes) {
 function decodeViaTurboStream(body, global) {
   return decode(body, {
     plugins: [
+      ...(window.TURBO_STREAM_EXTRA_DECODE_PLUGINS ?? []),
       (type, ...rest) => {
         if (type === "SanitizedError") {
           let [name, message, stack] = rest;
diff --git a/dist/production/chunk-ANIOYBQK.mjs b/dist/production/chunk-ANIOYBQK.mjs
index 765e7091a9d5662846c77f14be1534c1c10f9791..4666b6fd44f6182b2b050335869d6866b8de25e8 100644
--- a/dist/production/chunk-ANIOYBQK.mjs
+++ b/dist/production/chunk-ANIOYBQK.mjs
@@ -7899,6 +7899,7 @@ async function fetchAndDecodeViaTurboStream(args, basename, targetRoutes) {
 function decodeViaTurboStream(body, global) {
   return decode(body, {
     plugins: [
+      ...(window.TURBO_STREAM_EXTRA_DECODE_PLUGINS ?? []),
       (type, ...rest) => {
         if (type === "SanitizedError") {
           let [name, message, stack] = rest;
diff --git a/dist/production/chunk-TDCOAFPJ.mjs b/dist/production/chunk-TDCOAFPJ.mjs
index 340f4db030e7a4e50e497a827dd62eaef03e5bbc..283efe2f7e2867ab12870a73730ef23be6565d9b 100644
--- a/dist/production/chunk-TDCOAFPJ.mjs
+++ b/dist/production/chunk-TDCOAFPJ.mjs
@@ -966,6 +966,7 @@ function encodeViaTurboStream(data2, requestSignal, streamTimeout, serverMode) {
   return encode(data2, {
     signal: controller.signal,
     plugins: [
+      ...(global.TURBO_STREAM_EXTRA_ENCODE_PLUGINS ?? []),
       (value) => {
         if (value instanceof Error) {
           let { name, message, stack } = serverMode === "production" /* Production */ ? sanitizeError(value, serverMode) : value;
diff --git a/dist/production/chunk-UQPBOMFP.js b/dist/production/chunk-UQPBOMFP.js
index 31c3e8d526e5664f3815758ebb0a12809c8fba02..5a36eac16e82c7fd5571d6e1052e7c2cb03f4205 100644
--- a/dist/production/chunk-UQPBOMFP.js
+++ b/dist/production/chunk-UQPBOMFP.js
@@ -6408,6 +6408,7 @@ async function fetchAndDecodeViaTurboStream(args, basename, targetRoutes) {
 function decodeViaTurboStream(body, global) {
   return decode(body, {
     plugins: [
+      ...(window.TURBO_STREAM_EXTRA_DECODE_PLUGINS ?? []),
       (type, ...rest) => {
         if (type === "SanitizedError") {
           let [name, message, stack] = rest;

Along with the SerializesTo type:

import "prisma-client";

// This makes it an augmentation

declare module "prisma-client" {
	namespace Prisma {
		interface Decimal {
			unstable__ReactRouter_SerializesTo: [Decimal];
		}
	}
}

This is great, but we have to maintain the patch every time a new react router version comes out.

or

2. add serialization/deserialization helpers for all of our loader/useLoaderData functions. This adds noise to the codebase and is inefficient and provides no assurance that what you're decoding is actually a Decimal.

I've tried lots of workarounds, but there's really no way to achieve this without maintaining a fork or a patch.

Is there any chance the React Router team might reconsider exposing turbostream plugins?

If the problem is opening yourself up to any plugin being added, maybe the tc39-compatible Decimal polyfill would be an option: https://github.com/yukinotech/JSBD

Because it is just exposing what could eventually be part of the JS spec and could be enabled with an unstable feature flag?

[sergiodxa]

This change will make turbo-stream not be an internal detail of RR anymore.

In the future once RSC is stable and enabled always its streaming format should replace turbo-stream.

If the plugin system is made open it will make it harder for apps to upgrade later.

[immerrr]

I respect the decision, but would like to add two points for consideration.

This change will make turbo-stream not be an internal detail of RR anymore.

If a user is only sending native types around, they can be blissfully unaware of the way RR serializes/deserializes data, and they should be able to keep being equally unaware of the potential serialization/deserialization plugins.

If on the other hand someone hits the issue with decimals or any other unsupported datatypes, turbo-stream becomes a very evident implementation detail. And what adds to the frustration, is that adding custom datatypes is super easy with turbo-stream, RR itself is using those serialization/deserialization plugins, it is just that there is no way to configure it from the outside.

If the plugin system is made open it will make it harder for apps to upgrade later.

I believe RR has done a great job with exposing parts of its API as unstable_. It recognized that certain features were useful to at least some of the users before becoming fully fleshed out, while also conveying the fact the exported types or variables could be changed or removed at any moment. So anyone using those knew they were doing it at their own risk.

It sounds like the same logic could be applied to turbo-stream serialization/deserialization plugins.

[sergiodxa]

If on the other hand someone hits the issue with decimals or any other unsupported datatypes, turbo-stream becomes a very evident implementation detail.

If RR still used JSON.stringify/JSON.parse, you would hit the same serialization issues, so it's not like turbo-stream is limiting you in anyway, if any is actually allowing more return values from the server.

You shouldn't need to know RR uses turbo-stream, you should only need to know only standard JS values can be sent over the network, not any custom object.

I believe RR has done a great job with exposing parts of its API as unstable_.

Unstable features are features intended to be stable in the future, but the team is not ready yet to mark them as stable. Opening the turbo-stream as unstable just to then deprecate it in favor of RSC, or nothing, will not be a good usage of the unstable prefix.

And developers using this unstable feature to find out they either need to write their plugins or remove them and update the rest of their code (there's no guarantee that RSC will have plugins) will end up on people never upgrading RR, and a lot of complaints on how the RR team keeps doing breaking changes.

---

I don't see a reason to expose configuration for an internal lib that's intended to be replaced in the future. I would say to wait until RSC is stable, and then ask if it's possible to extend what RSC format serialize.

And meanwhile, keep Decimal and any other JS object that can't be serialized as server-side only, use the non-serializable value to generate other serializable values like formatted strings and send that from the loader.

If you can't do that, and really want to send Decimal to the browser, you can manually serialize it and parse it again on the component, or a clientLoader.

export async function loader() {
  return { value: serialize(new Decimal(0)) } // convert Decimal to a serializable value
}

export async function clientLoader({ serverLoader }) {
  let loaderData = await serverLoader()
  return { value: serialize(loaderData.value) } // convert serializable value back to Decimal
}

export async function Component({ loaderData }) {
  loaderData.value // this will be a Decimal instance
}

[immerrr]

First off, I appreciate you answering so fast and in such detail, thank you.

If RR still used JSON.stringify/JSON.parse, you would hit the same serialization issues, so it's not like turbo-stream is limiting you in anyway, if any is actually allowing more return values from the server.

You shouldn't need to know RR uses turbo-stream, you should only need to know only standard JS values can be sent over the network, not any custom object.

Agreed, turbo-stream itself is an addition over JSON, and is not the limiting factor.

Re: standard JS values: that's fair, but in the stack I'm working with right now, decimals to represent monetary values are standard: Postgres supports them, Prisma supports them, the backend and frontend code supports them (via Decimal.js), and there's no problem with decimals coming in with form data as strings, because we can spend some extra cycles parsing/deserializing every action cleanly with zod.

It's getting the data to the client that needs work every time. There's about a dozen of models, some with multiple decimal attributes, each with multiple routes, decimals being sent in arrays and in objects, at different nesting levels or even in a recursive tree-like structure. You can hand-code serialization/deserialization manually each time, but it starts to feel like you need to fight against the framework a little bit.

We did consider an automatic serializer, but then the deserialization has to be a heuristic, "is this an object that has this magic key/combination of keys -> it must be a decimal". It works, but it doesn't seem right to have a compact and reliable solution using a properly extensible serialization protocol so tantalisingly close, yet still hidden behind this little piece of red tape.

Unstable features are features intended to be stable in the future, but the team is not ready yet to mark them as stable. Opening the turbo-stream as unstable just to then deprecate it in favor of RSC, or nothing, will not be a good usage of the unstable prefix.

Out of curiosity, would you say the same about unstable_usePrompt, given that there is no intention of stabilizing it in the future?

In any case, like I said previously, I respect the decision, and appreciate the replies. You probably have seen the most of the arguments for and against the thing, just wanted to make sure this particular use case was not a blind spot.

[zvin]

This is not convenient when you have deeply nested structures with Decimals (or anything else you'd like to serialize).
You can always write wrappers like above but they are complex for no reason, even more if you have circular references in your data (which turbo-stream supports).

The frustrating part is that it is really easy to add support for additional serializable types using turbo-stream.

[zwhitchcox]

Do you have a suggestion for handling Decimals that are used all throughout our code base and much of the time deeply nested? Or would you consider supporting the tc39 Decimal proposal instead of the exposing the plugin options?

also the problem of serializing/deserializing decimals will still exist, correct? Or are you planning on exposing a plugin system for RSC’s like Next.js does?

Basically will there be some way of dealing with this in RSC that will be better, or will this always be a problem.

[sergiodxa]

Do you have a suggestion for handling Decimals that are used all throughout our code base and much of the time deeply nested?

I recommend you to send from loader the final values the UI will need, if you need the decimal to format a value as a money amount, do that formatting on the loader and send a string to the UI. This way you don't need to send the Decimal object itself.

Or would you consider supporting the tc39 Decimal proposal instead of the exposing the plugin options?

I'm sure once Decimal is supported by JS runtimes it will be supported, until then the proposal can be rejected and never become a thing.

also the problem of serializing/deserializing decimals will still exist, correct?

The problem will still exists, the difference is that whatever RSC plugins may look like may be different than turbo-stream, exposing turbo-stream plugins now makes the change harder later.

Or are you planning on exposing a plugin system for RSC’s like Next.js does?

There's no plan AFAIK, at least for now.

Basically will there be some way of dealing with this in RSC that will be better, or will this always be a problem.

Can't confirm until RSC is stable and the default.