node-fetch v3.1 has a own formData decoder now

[jimmywarting]

Hi, i saw remix in a weekly js newsletter and read on your page that you can kind of do:

export async function action({ request }) {
  let body = await request.formData();
  let invoice = await createInvoice(body);
  return redirect(`/invoices/${invoice.id}`);
}

This spark my interest so i investigated how it was being done and found this:

remix/packages/remix-node/fetch.ts

Lines 19 to 22 in a69a631

	async formData() {
	let body = await this.clone().text();
	return new RemixFormData(body);
	}

Saw that you do actually use node-fetch 👍 🙂

We have finally implemented a own 100% spec compatible FormData decoder ourself into node-fetch v3.1 that handles both url encoded and multipart/formdata payload. so you can deal with things such as File uploads also

export async function action({ request }) {
  const fd = await request.formData();
  const pdf = fd.get('attachment') // (returns a `File` class)
  
  const ab = await pdf.arrayBuffer()
  const text = await pdf.text()
  const stream = pdf.stream()
}

tip if you want to use the FormData, Blob and File class node-fetch uses then you could do:

import fetch, {Response, Request} from 'node-fetch'

const FormData = (await new Response(new URLSearchParams()).formData()).constructor
const Blob = (await new Response().blob()).constructor

// (or just import formdata-polyfill and fetch-blob)

const fd = new FormData()
const blob = new Blob([new ArrayBuffer(1024)])
fd.append('attachment', blob, 'xyz.pdf')

// Now you got the `File` class as well... (according to spec, appending a Blob should convert it to a File)
const File = fd.get('attachment').constructor

const file = new File([Buffer.from('xyz'), 'dummy.txt', {type: 'text/plain'}])
fd.append('upload', file)

new Request(url, {method: 'POST', body: fd}).formData().then(...)

[QuentinWidlocher]

Hi !
I'm not sure how I'm supposed to use this with the current version of Remix ?

  const fd = await request.formData();
  const pdf = fd.get('attachment') // This actually returns a string

Could this be used to send files with form actions in Remix ?

[jimmywarting]

The FormData version node-fetch uses is a 100% spec compatible FormData implementation... With support for 100% spec compatible Blob/Files attachments as well.

Remixs FormData impl only accepts strings

remix/packages/remix-node/form-data.ts

Lines 7 to 12 in a69a631

	append(name: string, value: string | Blob, fileName?: string): void {
	if (typeof value !== "string") {
	throw new Error("formData.append can only accept a string");
	}
	this._params.append(name, value);
	}

And the constructor shouldn't be able to accept a string as a body

remix/packages/remix-node/form-data.ts

Lines 4 to 6 in a69a631

	constructor(body?: string) {
	this._params = new URLSearchParams(body);
	}

Remixs version only supports url-encodeded payloads so you won't be able to upload files with that...

---

if you where to update to node-fetch v3.1 then you could remove your formdata implementation and stop extending Request with a own .formData() method. node-fetch have now implemented request.formData() and response.formData()

if you use node-fetch own method instead then you will be able to upload files with multipart/form-data in remixs action

[matmilbury]

Maybe we could update remix source code to use that new node-fetch decoder?

[mjackson]

@jimmywarting thank you for letting us know about the FormData implementation in node-fetch. I wasn’t aware it existed.

how do you handle streaming file uploads to external storage, like S3?

[jimmywarting]

The FormData implementation came quite recently with the newest version 3.1.0, so i wouldn't expect so many to know about it...

how do you handle streaming file uploads to external storage, like S3?

Right now, everything will be decoded into the memory and resolved once it has decoded everything.
Even the File will be a references to some memory allocation.

So streaming isn't perfect but it gets the job done and works as it's suppose to be according to the spec.

export async function action({ request }) {
  const fd = await request.formData()
  const pdf = fd.get('attachment') // (returns a File - all data is held in memory at this point)
  
  // to read the File (Blob) you could use the `stream()` method to read it chunk by chunk
  const whatwgStream = pdf.stream()
  
  // and pipe it to some destination
  whatwgStream.pipeTo(dest)
}

[jimmywarting]

A potentially better option would just be to send the file as raw binary data without the FormData if you want to better stream the data...

One thing we might look into later is for us to offload huge files to a temporary file on the disk and create a File class backed up by the filesystem instead. using: https://github.com/node-fetch/fetch-blob#blob-part-backed-up-by-filesystem

[mjackson]

I just looked into the FormData support in node-fetch and it looks like they don't support streaming to external storage, which is the primary use case for anyone handling file uploads in Remix. So we will stick with our own multipart handling for now.

[jimmywarting]

How exactly do you handle streaming files in Remix while also using FormData?

From the locks of your FormData impl it only supports strings and no files?
Your constructor is using the URLSearchParams after all to decode the payload. So it can't support files?