Handling multipart/form-data body

[treble-snake]

Hi, I'm working on a multipart body parser and wondering if you could reveal the chunking logic for that data type.

It looks like it's kinda supposed to pass one field per chunk for text fields, separate headers and body for file fields (starting chunk with headers, then N chunks with data), and the last chunk would be the closing boundary.

But sometimes the server concatenates all or some of the text fields, it also can concatenate (into one chunk) part of the file with field headers and/or closing boundary. With a file it's especially inconvenient since it makes steaming to the disk a complicated task.

I've confirmed it with the very basic example described in examples/Upload.js (v15.8.0).
Can't see any pattern yet, it just does it sometimes (maybe more ofter with smaller files).

I'm wondering if it's the intended behaviour or something went wrong (or maybe I'm doing something wrong ofc).

Thanks.

[treble-snake]

Well, I wasn't looking for help, more like reporting presumably unexpected behaviour.
Just to clarify things, this is the code (almost the same as the upload example):

const uWS = require('uWebSockets.js');
const port = 3000;

uWS.App().post('/*', (res, req) => {
  let q = 0;
  res.onData((chunk, isLast) => {
    /* Buffer this anywhere you want to */
    console.log('Chunk #', q, ', size:', chunk.byteLength);

    if (q === 0 || isLast) {
      if (chunk.byteLength <= 180) {
        console.log('Handling chunk', Buffer.from(chunk).toString());
      } else {
        console.error('Boundary expected!');
        process.exit(1);
      }
    }

    q++;

    /* We respond when we are done */
    if (isLast) {
      res.end('Thanks for the data!');
    }
  });

  res.onAborted(() => {
    /* Request was prematurely aborted, stop reading */
    console.log('Eh, okay. Thanks for nothing!');
  });
}).listen(port, (token) => {
  if (token) {
    console.log('Listening to port ' + port);
  } else {
    console.log('Failed to listen to port ' + port);
  }
});

Then I sent the same simple form with just 1 field (a file), and here is the output of 2 tries:

➜ node example/pure.js
Listening to port 3000
Chunk # 0 , size: 180
Handling chunk ----------------------------499162952261603574929250
Content-Disposition: form-data; name="the_file"; filename="Screenshot 2019-03-18 at 13.45.02.png"
Content-Type: image/png


Chunk # 1 , size: 131072
Chunk # 2 , size: 48936
Chunk # 3 , size: 16600
Chunk # 4 , size: 65536
Chunk # 5 , size: 65536
Chunk # 6 , size: 65536
Chunk # 7 , size: 51394
Chunk # 8 , size: 58
Handling chunk
----------------------------499162952261603574929250--

➜ node example/pure.js
Listening to port 3000
Chunk # 0 , size: 16492
Boundary expected!

So, as you can see, 1st time chunks came as expected, and 2nd time field headers got merged with a part of the file data in the first chunk, and that looks wrong.
But if you say it's OK, then OK.

[ghost]

There is a very fast multipart parser built-in now: uWS.getParts(body)

[mtsewrs]

@Wyzix33 Like so

const header = req.getHeader('content-type');
const fields = getParts(buffer, header)

parse incoming data into a buffer from res.onData

[Wyzix33]

@mattiasewers
what is buffer?
I'm trying something like this, and i still get undefined:

(res, req) => {
 let writeStream, fileName;
 const header = req.getHeader('content-type');
 console.log('Posted to', req.getUrl());
 res.onData((chunk, isLast) => {
  let buffer = Buffer.from(chunk);
  const parts = uWS.getParts(buffer, header);
  console.log(parts); 
  /* Buffer this anywhere you want to */
  console.log('Got chunk of data with length', chunk.byteLength, 'isLast:', isLast);
  /* We respond when we are done */
  if (isLast) {
   if (writeStream) writeStream.close();
   res.end('Thanks for the data!');
  }
 });

 res.onAborted(() => {
  /* Request was prematurely aborted, stop reading */
  if (writeStream) writeStream.close();
  console.log('Eh, okay. Thanks for nothing!');
 });
};

and this is what i see in browser

POST /upload HTTP/1.1
Host: analitic.go.ro
Connection: keep-alive
Content-Length: 919304
Pragma: no-cache
Cache-Control: no-cache
sec-ch-ua: "Chromium";v="88", "Google Chrome";v="88", ";Not A Brand";v="99"
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36
Content-type: application/x-www-form-urlencoded
Accept: */*
Origin: https://analitic.go.ro
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

and form data:

------WebKitFormBoundaryj411aOxd9ET0TRvX
Content-Disposition: form-data; name="name"

1614172919662ipibgj0.jpg
------WebKitFormBoundaryj411aOxd9ET0TRvX
Content-Disposition: form-data; name="file"; filename="landscape-1984964_1920.jpg"
Content-Type: image/jpeg


------WebKitFormBoundaryj411aOxd9ET0TRvX--

I get undefined parts...
Any help?

[mtsewrs]

You need to wait for for all data to be done so full example what I am doing

const header = req.getHeader('content-type');
function readBody(res){
  return new Promise((resolve) => {
    let buffer = Buffer.from('');
    res.onData((ab, isLast) => {
      const chunk = Buffer.from(ab);
      buffer = Buffer.concat([buffer, chunk]);
      if (isLast) {
        resolve(buffer);
      }
    });
  });
}
const buffer = await readBody(res)
const data = getParts(buffer, header);

or

 const header = req.getHeader('content-type');
 res.onData((chunk, isLast) => {
   let buffer = Buffer.from('');
   const chunk = Buffer.from(ab);
   buffer = Buffer.concat([buffer, chunk]);
   if (isLast) {
     const data = getParts(buffer, header);
     // do something with data
   }
 });

[Wyzix33]

Thanks, it's working now :)

[arupadhy]

Hey @alexhultman , I am trying to use this library (specifically getParts) and hitting a road block for some multi-part requests (legacy clients that can't really be changed)

Here is a sample of multi-part requests that works fine:

----------------------------656819903371688537225955
Content-Disposition: form-data; name="encrypted"

textDataHere
----------------------------656819903371688537225955
Content-Disposition: form-data; name="id"

uniqueId
----------------------------656819903371688537225955--

Here is an example that does not

 --boundary_.oOo._TquwsEdXMcoGdGzxm8Z064t0YaZ6SslT
Content-Type: text/plain	
Content-Disposition: form-data; name="encrypted"

textDataHere
--boundary_.oOo._TquwsEdXMcoGdGzxm8Z064t0YaZ6SslT
Content-Type: text/plain
Content-Disposition: form-data; name="id"

uniqueId
--boundary_.oOo._TquwsEdXMcoGdGzxm8Z064t0YaZ6SslT--

One more example that works (content-type is after content-Disposition)

----------------------------037395247820009249958973
Content-Disposition: form-data; name="id"
Content-Type: text/plain

uniqueId
----------------------------037395247820009249958973
Content-Disposition: form-data; name="encrypted"
Content-Type: text/plain

textDataHere
----------------------------037395247820009249958973--

final update:

I found the problem...While calling getParts(<>, header), if the header passed by the client have double quotes for the boundary, the call fails. After removing the quotes in node layer, this worked fine.
call that was failing had this as header

multipart/form-data; boundary="boundary_.oOo._5vcxtKweWUEry9uBwYXgKRR9gy8Fl38b"

As per https://tools.ietf.org/html/rfc2388, 'Content-Type' is optional for all MIME types.
PS: trying to move away from multer/bus-boy and express and both reqs worked fine before

[crotel]

sorry for my stupid question.

const header = req.getHeader('content-type'); 
 res.onData((ab, isLast) => {
   let buffer = Buffer.from('');
   const chunk = Buffer.from(ab);
   buffer = Buffer.concat([buffer, chunk]);
   if (isLast) {
     const data = getParts(buffer, header);
     // do something with data
   }
 });

how about reuse data and fetch to another dist url via formdata either? simply I need a typeof File here.

and I found the document had these :

export interface MultipartField {
    data: ArrayBuffer;
    name: string;
    type?: string;
    filename?: string;
}

/** Takes a POSTed body and contentType, and returns an array of parts if the request is a multipart request */
export function getParts(body: RecognizedString, contentType: RecognizedString): MultipartField[] | undefined;

what is this the "MultipartField" use for?

[joshxyzhimself]

In your first code block, you should aggregate the chunks outside the onData callback.

MultipartField[] is the return value of uws.getParts.

const request = {
  url: req.getUrl(),
  method: req.getMethod(),
  headers: new Map(),
  buffer: null,
  json: null,
  parts: null,
  error: null,
};
req.forEach((key, value) => {
  request.headers.set(key, value);
});
request.buffer = Buffer.from([]);
res.onData((chunk_arraybuffer, is_last) => {
  const chunk_buffer = Buffer.from(chunk_arraybuffer.slice(0));
  request.buffer = Buffer.concat([request.buffer, chunk_buffer]);
  if (is_last === true) {
    if (request.buffer.length > 0) {
      try {
        if (request.headers.get('Content-Type').includes('application/json') === true) {
          request.json = JSON.parse(request.buffer.toString());
        }
        if (request.headers.get('Content-Type').includes('multipart/form-data') === true) {
          request.parts = uws.getParts(request.buffer, request.headers.get('Content-Type'));
        }
      } catch (e) {
        request.error = e;
        console.error(e);
      }
    }

    // do your fetch here, use https://www.npmjs.com/package/node-fetch

    // craft your response here
    res.writeStatus('200');
    res.end();

  }
});
res.onAborted(() => {
  // this handler is required
});