How do you upload a file to S3?

[n1ghtmare]

Hi there, I looked at: this similar question that was asked at a previous date. From what I see however the API has changed and there isn't a stream anymore, instead we now get data and it's of type ArrayIterable<Uint8Array>. I'm not entirely sure what I can do with it. Is there a way to convert this to a Buffer or something of the sort?

Has anyone done this by any chance? I've spent way too much time on this.

Any help would be much appreciated.

[gdonoso94]

Hi!

I've been struggling with this for a while too. It'd be great to have more documentation on this...

[tylermmorton]

Hey, check out my show and tell on GraphQL file uploads. The code I posted shows how to convert the file parts into a File or Blob. From there you can call Blob.arrayBuffer()

[awaseem]

Just a heads up, I was able to get this to work for the node environment. You can create a Readable from async iterables by calling Readable.from(data). Here is my code (I'm using S3 api w/ backblaze):

import { Readable } from 'stream'
import type { UploadHandler } from '@remix-run/node'
import { S3Client } from '@aws-sdk/client-s3'
import { Upload } from '@aws-sdk/lib-storage'

const {
  S3_BUCKET,
  S3_REGION,
  S3_ENDPOINT,
  S3_ACCESS_KEY_ID,
  S3_SECRET_ACCESS_KEY,
} = process.env

if (!S3_BUCKET) {
  throw new Error(`Storage is missing required configuration.`)
}

if (!S3_REGION) {
  throw new Error(`Storage is missing required configuration.`)
}

if (!S3_ACCESS_KEY_ID) {
  throw new Error(`Storage is missing required configuration.`)
}

if (!S3_ENDPOINT) {
  throw new Error(`Storage is missing required configuration.`)
}

if (!S3_SECRET_ACCESS_KEY) {
  throw new Error(`Storage is missing required configuration.`)
}

const storage = new S3Client({
  endpoint: S3_ENDPOINT,
  credentials: {
    accessKeyId: S3_ACCESS_KEY_ID,
    secretAccessKey: S3_SECRET_ACCESS_KEY,
  },
  region: S3_REGION,
})

export async function uploadToS3(stream: Readable, filename: string) {
  return new Upload({
    client: storage,
    leavePartsOnError: false,
    params: {
      Bucket: S3_BUCKET,
      Key: filename,
      Body: stream,
    },
  }).done()
}

export const uploadHandler: UploadHandler = async ({ data, filename }) => {
  if (!filename) {
    return ''
  }

  const stream = Readable.from(data)
  const upload = await uploadToS3(stream, filename)

  if (upload.$metadata.httpStatusCode === 200) {
    return filename
  }

  return ''
}

[spencerjsmall]

Just a heads up, I was able to get this to work for the node environment. You can create a Readable from async iterables by calling Readable.from(data). Here is my code (I'm using S3 api w/ backblaze):

import { Readable } from 'stream'
import type { UploadHandler } from '@remix-run/node'
import { S3Client } from '@aws-sdk/client-s3'
import { Upload } from '@aws-sdk/lib-storage'

const {
  S3_BUCKET,
  S3_REGION,
  S3_ENDPOINT,
  S3_ACCESS_KEY_ID,
  S3_SECRET_ACCESS_KEY,
} = process.env

if (!S3_BUCKET) {
  throw new Error(`Storage is missing required configuration.`)
}

if (!S3_REGION) {
  throw new Error(`Storage is missing required configuration.`)
}

if (!S3_ACCESS_KEY_ID) {
  throw new Error(`Storage is missing required configuration.`)
}

if (!S3_ENDPOINT) {
  throw new Error(`Storage is missing required configuration.`)
}

if (!S3_SECRET_ACCESS_KEY) {
  throw new Error(`Storage is missing required configuration.`)
}

const storage = new S3Client({
  endpoint: S3_ENDPOINT,
  credentials: {
    accessKeyId: S3_ACCESS_KEY_ID,
    secretAccessKey: S3_SECRET_ACCESS_KEY,
  },
  region: S3_REGION,
})

export async function uploadToS3(stream: Readable, filename: string) {
  return new Upload({
    client: storage,
    leavePartsOnError: false,
    params: {
      Bucket: S3_BUCKET,
      Key: filename,
      Body: stream,
    },
  }).done()
}

export const uploadHandler: UploadHandler = async ({ data, filename }) => {
  if (!filename) {
    return ''
  }

  const stream = Readable.from(data)
  const upload = await uploadToS3(stream, filename)

  if (upload.$metadata.httpStatusCode === 200) {
    return filename
  }

  return ''
}

Lifesaver, thank you! Not sure why the switch from stream to asynciterable for UploadHandler hasn't been better documented....

[AlessandroVol23]

Hey 👋🏽
thanks for that! I've implemented it like that as well but still get an error with larger files. Does it work for you with larger files as well? >3 MB