Validating file streaming causes weird behavior, like server hanging

[D-Marc1]

I'm using file streaming, which works perfectly, other than when I try to add validations and a validation fails. I followed the docs and here's the code I'm using:

const files: Buffer[] = []

request.multipart.onFile('files', { extnames: ['png', 'gif'] }, async (file, reportChunk) => {
  file.pause()
  file.on('data', reportChunk)

  const fileContent = await getStream.buffer(file)

  files.push(fileContent)
})

await request.multipart.process()

There's no issues with this this if I upload a .png for instance, as it obviously won't break the validation. However, if I do something specifically to break the validation, like upload a .jpg, which isn't on the extnames, then it starts causing issues, like the server just hanging.

I do something like following for the error message:

const filesInput = ctx.request.files('files')

if (filesInput[0].errors.length > 0) {
  return ctx.response.status(422).send({ errors: filesInput[0].errors })
}

What's weird is that when I add breakpoints, I can clearly see the message is there. However, I experience two types of issues with related behaviors. With some images, it successfully returns the response on the first time I upload a file, but then just hangs on every subsequent upload. With other images, it hangs immediately on the first upload. This behavior is consistent with the exact photo I use. Like I said, I clearly see the error message when I debug my code, but ultimately, I get the hanging issue if a file validation fails.

[thetutlage]

Hello @D-Marc1

Can you please share this code as a repo since I am not able to reproduce it by directly copy/pasting the code

[thetutlage]

Just checked the repo and all looks fine to me. There is one thing to handle, ie not throwing the validation errors from the stream handler. So basically, you need to wrap the get-stream method inside a try/catch block and then ignore the self validation errors.

Something like this

    try {
      const fileContent = await getStream.buffer(file)
      files.push(fileContent)
    } catch (error) {
      if (error.code === 'E_STREAM_VALIDATION_FAILURE') {
        return
      }
      throw error
    }

[thetutlage]

Lemme if this works and then we can update the docs as well

[D-Marc1]

I tried out this code, but it still doesn't resolve my issue. In fact, it's technically worse with the try/catch, than without it, as it consistently hangs with every image I use. Whereas, without the try/catch, some images do correctly work on the first upload attempt with the right validation message, but then it hangs after every subsequent attempt.

[thetutlage]

I don't think using try/catch anywhere is technically worse. Maybe you meant, in your case it turned out not to be doing what you want.

But anyways I am not able to reproduce it at all. I have used your repo with the following modifications to the route handler. Following the complete routes.ts file

import getStream from 'get-stream'
import Route from '@ioc:Adonis/Core/Route'

Route.post('/', async ({ request, response }) => {
  const files: Buffer[] = []

  request.multipart.onFile('files', { extnames: ['png', 'gif'] }, async (file, reportChunk) => {
    file.pause()
    file.on('data', reportChunk)

    try {
      const fileContent = await getStream.buffer(file)
      files.push(fileContent)
    } catch (error) {
      if (error.code === 'E_STREAM_VALIDATION_FAILURE') {
        return
      }
      throw error
    }
  })

  await request.multipart.process()
  const filesInput = request.files('files')

  if (filesInput[0].errors.length > 0) {
    return response.status(422).send({ errors: filesInput[0].errors })
  }

  return filesInput
})

And here's a screen recording to the behavior I get https://res.cloudinary.com/adonis-js/video/upload/v1636787747/Assets/file-uploads_totsww.mp4

[D-Marc1]

So I did some thorough testing to give my full analysis on the matter.

1. The code with the try/catch always hangs for me consistently. I'm honestly not sure you were able to get it to work in your video with that.
2. Without the try/catch, like how I had it initially, it works perfectly for me in Postman and I consistently see the 422 response.
3. The weird issues I described in my initial comment only happen in the following scenario. On my frontend, I upload a Blob, which is what seems to be the cause of the issue. As I said previously, this has an odd behavior of most images working on the first upload attempt to give a 422 when the validation fails, but then it hangs on subsequent attempts to upload. It should be noted that I didn't experience this behavior when I used custom and third party validators, even when I uploaded Blobs. So for whatever reason, uploading with a Blob, while using Adonis' built in validator for streams is what causes the issue for me.

Not sure if it's important or not, but here's a version of me using a custom and third party validator.

const files: Buffer[] = []

const fileValidationError = { message: '', status: 0 }

const imageExtTypesAllowed = ['jpg', 'png', 'gif']

request.multipart?.onFile('files', {}, async (file, reportChunk) => {
  file.pause()
  file.on('data', reportChunk)

  const fileContent = await getStream.buffer(file)

  const fileSizeInMB = Number((fileContent.byteLength / (1024 * 1024)).toFixed(2))

  if (fileSizeInMB > 20) {
    fileValidationError.message = `Max image file size is 20 megabytes`
    fileValidationError.status = 422
  }

  const { ext } = (await FileType.fromBuffer(fileContent)) ?? {}

  if (!imageExtTypesAllowed.includes(ext ?? '')) {
    const allowedExtTypesString = imageExtTypesAllowed.join(', ')

    fileValidationError.message = `Image files must be ${allowedExtTypesString}`
    fileValidationError.status = 422
  }

  files.push(fileContent)
})

await request.multipart?.process()

if (fileValidationError.message && fileValidationError.status) {
  const { message, status } = fileValidationError

  return response.status(status).send({ errors: [{ message }] })
}

As I said, this works perfectly for me when I pass in Blob images.

[MANTENN]

I'm using file streaming, which works perfectly, other than when I try to add validations and a validation fails. I followed the docs and here's the code I'm using:

const files: Buffer[] = []

request.multipart.onFile('files', { extnames: ['png', 'gif'] }, async (file, reportChunk) => {
  file.pause()
  file.on('data', reportChunk)

  const fileContent = await getStream.buffer(file)

  files.push(fileContent)
})

await request.multipart.process()

There's no issues with this this if I upload a .png for instance, as it obviously won't break the validation. However, if I do something specifically to break the validation, like upload a .jpg, which isn't on the extnames, then it starts causing issues, like the server just hanging.

I do something like following for the error message:

const filesInput = ctx.request.files('files')

if (filesInput[0].errors.length > 0) {
  return ctx.response.status(422).send({ errors: filesInput[0].errors })
}

What's weird is that when I add breakpoints, I can clearly see the message is there. However, I experience two types of issues with related behaviors. With some images, it successfully returns the response on the first time I upload a file, but then just hangs on every subsequent upload. With other images, it hangs immediately on the first upload. This behavior is consistent with the exact photo I use. Like I said, I clearly see the error message when I debug my code, but ultimately, I get the hanging issue if a file validation fails.

It hangs for me when I throw an error in the partHandler callback. Im debugging the node_modules execution and the code gets through the reportError function in PartHandler.

  public async upload({ request, params, response }: HttpContextContract) {
    try {
      const { proposalId } = params
      // // extra overhead writing directly to tmp storage; however this validates file types
      // const file = request.file('file', { size: '20mb', extnames: ['pdf', 'doc', 'docx'] })
      // console.log('file', file.tmpPath)
      // if (file.hasErrors) {
      //   return { state: 'error', errors: file.errors }
      // }
      // const tmpFileStream = await Drive.use('local').getStream(file.tmpPath)
      // console.log('tmpFileStream', tmpFileStream)
      // const s3Results = await Drive.use('s3').putStream(`${Env.get('S3_FILEPREFIX')}/${proposalId}/${uuidv4()}/${file.fileName}`, tmpFileStream)

      // console.log('file', file.hasErrors)

      let s3Results

      request.multipart
        .onFile(
          'file',
          {
            size: '20mb',
            extnames: ['pdf', 'doc', 'docx'],
          },
          async (part, reportChunk) => {
            throw new Error('blow up the stream')
            })
                  await request.multipart.process()
                  // hangs on process
                  }

It also hangs with pause the stream bit as well:

           await part.pause()
            await part.on('data', reportChunk)
            throw new Error('blow up the stream')