Skip to content

Commit 2edfa93

Browse files
witemple-msftwillmtemple
witemple-msft
and
willmtemple
authored
[http-server-javascript] Support new-style multipart requests (and some other things) (microsoft#5514)
This PR implements handling for new-style multipart requests and heavily refactors the multipart core to use web streams. While this doesn't enable streaming multipart yet (i.e. the implementation layer still buffers the parts before providing them to the business logic), it's a step towards streaming multipart. - Add special handling for representation of well-known model `TypeSpec.Http.HttpPart` (link microsoft#5275). - Added encode/decode support for `bytes` through `NodeJS.Buffer` as `Uint8Array`. - Fixed an issue where serialization propagation did not flow through array types, leading to cases where models were being detected as not requiring serialization if they contained properties with array types, where the array value type did require serialization. - Implemented and pervasively used `gensym`, which generates unique symbols containing a monotonically incrementing counter. This helps me prevent collisions between names of variables I generate and names of variables generated from user typespec code without requiring a full-blown scope solution. It costs readability of the generated code, but isn't so bad. - Implemented rfc5987 header parameter parsing as a shared helper module, used to extract boundary and other information while processing multipart requests. - Implemented streaming multipart chunking via web streams in a shared helper module. When a multipart/form-data content-type is detected, a parser is generated that first converts the `http.IncomingMessage` into a ReadableStream, then the bytes read from the multipart body are split into parts and streamed individually. Finally, each individual part stream is consumed by a transform that reads enough of the stream to parse the headers and then passes the rest of the body through to a final "body stream." - Exposed error handlers (onInvalidRequest, onRequestNotFound, onInternalError) to application logic through `HttpContext`. The application logic can use these handlers to manually respond with validation/internal errors in whichever way the router would ordinarily respond to them. --------- Co-authored-by: Will Temple <[email protected]>
1 parent fda51ff commit 2edfa93

File tree

23 files changed

+1525
-242
lines changed

23 files changed

+1525
-242
lines changed

‎.chronus/changes/witemple-msft-hsj-visibility-2025-0-7-11-43-52.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
changeKind: feature
3+
packages:
4+
- "@typespec/http-server-javascript"
5+
---
6+
7+
- Implemented new-style multipart request handling.
8+
- Fixed JSON serialization/deserialization in some cases where models that required serialization occurred within arrays.

‎packages/http-server-javascript/generated-defs/helpers/header.ts‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
// Copyright (c) Microsoft Corporation
2+
// Licensed under the MIT license.
3+
4+
import { Module } from "../../src/ctx.js";
5+
6+
export let module: Module = undefined as any;
7+
8+
// prettier-ignore
9+
const lines = [
10+
"// Copyright (c) Microsoft Corporation",
11+
"// Licensed under the MIT license.",
12+
"",
13+
"export interface HeaderValueParameters {",
14+
" value: string;",
15+
" verbatim: string;",
16+
" params: { [k: string]: string };",
17+
"}",
18+
"",
19+
"/**",
20+
" * Parses a header value that may contain additional parameters (e.g. `text/html; charset=utf-8`).",
21+
" * @param headerValueText - the text of the header value to parse",
22+
" * @returns an object containing the value and a map of parameters",
23+
" */",
24+
"export function parseHeaderValueParameters<Header extends string | undefined>(",
25+
" headerValueText: Header,",
26+
"): undefined extends Header ? HeaderValueParameters | undefined : HeaderValueParameters {",
27+
" if (headerValueText === undefined) {",
28+
" return undefined as any;",
29+
" }",
30+
"",
31+
" const idx = headerValueText.indexOf(\";\");",
32+
" const [value, _paramsText] =",
33+
" idx === -1",
34+
" ? [headerValueText, \"\"]",
35+
" : [headerValueText.slice(0, idx), headerValueText.slice(idx + 1)];",
36+
"",
37+
" let paramsText = _paramsText;",
38+
"",
39+
" // Parameters are a sequence of key=value pairs separated by semicolons, but the value may be quoted in which case it",
40+
" // may contain semicolons. We use a regular expression to iteratively split the parameters into key=value pairs.",
41+
" const params: { [k: string]: string } = {};",
42+
"",
43+
" let match;",
44+
"",
45+
" // TODO: may need to support ext-parameter (e.g. \"filename*=UTF-8''%e2%82%ac%20rates\" => { filename: \"€ rates\" }).",
46+
" // By default we decoded everything as UTF-8, and non-UTF-8 agents are a dying breed, but we may need to support",
47+
" // this for completeness. If we do support it, we'll prefer an ext-parameter over a regular parameter. Currently, we'll",
48+
" // just treat them as separate keys and put the raw value in the parameter.",
49+
" //",
50+
" // https://datatracker.ietf.org/doc/html/rfc5987#section-3.2.1",
51+
" while ((match = paramsText.match(/\\s*([^=]+)=(?:\"([^\"]+)\"|([^;]+));?/))) {",
52+
" const [, key, quotedValue, unquotedValue] = match;",
53+
"",
54+
" params[key.trim()] = quotedValue ?? unquotedValue;",
55+
"",
56+
" paramsText = paramsText.slice(match[0].length);",
57+
" }",
58+
"",
59+
" return {",
60+
" value: value.trim(),",
61+
" verbatim: headerValueText,",
62+
" params,",
63+
" };",
64+
"}",
65+
"",
66+
];
67+
68+
export async function createModule(parent: Module): Promise<Module> {
69+
if (module) return module;
70+
71+
module = {
72+
name: "header",
73+
cursor: parent.cursor.enter("header"),
74+
imports: [],
75+
declarations: [],
76+
};
77+
78+
module.declarations.push(lines);
79+
80+
parent.declarations.push(module);
81+
82+
return module;
83+
}

‎packages/http-server-javascript/generated-defs/helpers/index.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ export async function createModule(parent: Module): Promise<Module> {
1616
};
1717

1818
// Child modules
19+
await import("./header.js").then((m) => m.createModule(module));
20+
await import("./multipart.js").then((m) => m.createModule(module));
1921
await import("./router.js").then((m) => m.createModule(module));
2022

2123
parent.declarations.push(module);

‎packages/http-server-javascript/generated-defs/helpers/multipart.ts‎

Lines changed: 256 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,256 @@
1+
// Copyright (c) Microsoft Corporation
2+
// Licensed under the MIT license.
3+
4+
import { Module } from "../../src/ctx.js";
5+
6+
export let module: Module = undefined as any;
7+
8+
// prettier-ignore
9+
const lines = [
10+
"// Copyright (c) Microsoft Corporation",
11+
"// Licensed under the MIT license.",
12+
"",
13+
"import type * as http from \"node:http\";",
14+
"",
15+
"export interface HttpPart {",
16+
" headers: { [k: string]: string | undefined };",
17+
" body: ReadableStream<Buffer>;",
18+
"}",
19+
"",
20+
"/**",
21+
" * Consumes a stream of incoming data and splits it into individual streams for each part of a multipart request, using",
22+
" * the provided `boundary` value.",
23+
" */",
24+
"function MultipartBoundaryTransformStream(",
25+
" boundary: string,",
26+
"): ReadableWritablePair<ReadableStream<Buffer>, Buffer> {",
27+
" let buffer: Buffer = Buffer.alloc(0);",
28+
" // Initialize subcontroller to an object that does nothing. Multipart bodies may contain a preamble before the first",
29+
" // boundary, so this dummy controller will discard it.",
30+
" let subController: { enqueue(chunk: Buffer): void; close(): void } | null = {",
31+
" enqueue() {},",
32+
" close() {},",
33+
" };",
34+
"",
35+
" let boundarySplit = Buffer.from(`--${boundary}`);",
36+
" let initialized = false;",
37+
"",
38+
" // We need to keep at least the length of the boundary split plus room for CRLFCRLF in the buffer to detect the boundaries.",
39+
" // We subtract one from this length because if the whole thing were in the buffer, we would detect it and move past it.",
40+
" const bufferKeepLength = boundarySplit.length + BUF_CRLFCRLF.length - 1;",
41+
" let _readableController: ReadableStreamDefaultController<ReadableStream<Buffer>> = null as any;",
42+
"",
43+
" const readable = new ReadableStream<ReadableStream<Buffer>>({",
44+
" start(controller) {",
45+
" _readableController = controller;",
46+
" },",
47+
" });",
48+
"",
49+
" const readableController = _readableController;",
50+
"",
51+
" const writable = new WritableStream<Buffer>({",
52+
" write: async (chunk) => {",
53+
" buffer = Buffer.concat([buffer, chunk]);",
54+
"",
55+
" let index: number;",
56+
"",
57+
" while ((index = buffer.indexOf(boundarySplit)) !== -1) {",
58+
" // We found a boundary, emit everything before it and initialize a new stream for the next part.",
59+
"",
60+
" // We are initialized if we have found the boundary at least once.",
61+
" //",
62+
" // Cases",
63+
" // 1. If the index is zero and we aren't initialized, there was no preamble.",
64+
" // 2. If the index is zero and we are initialized, then we had to have found \\r\\n--boundary, nothing special to do.",
65+
" // 3. If the index is not zero, and we are initialized, then we found \\r\\n--boundary somewhere in the middle,",
66+
" // nothing special to do.",
67+
" // 4. If the index is not zero and we aren't initialized, then we need to check that boundarySplit was preceded",
68+
" // by \\r\\n for validity, because the preamble must end with \\r\\n.",
69+
"",
70+
" if (index > 0) {",
71+
" if (!initialized) {",
72+
" if (!buffer.subarray(index - 2, index).equals(Buffer.from(\"\\r\\n\"))) {",
73+
" readableController.error(new Error(\"Invalid preamble in multipart body.\"));",
74+
" } else {",
75+
" await enqueueSub(buffer.subarray(0, index - 2));",
76+
" }",
77+
" } else {",
78+
" await enqueueSub(buffer.subarray(0, index));",
79+
" }",
80+
" }",
81+
"",
82+
" // We enqueued everything before the boundary, so we clear the buffer past the boundary",
83+
" buffer = buffer.subarray(index + boundarySplit.length);",
84+
"",
85+
" // We're done with the current part, so close the stream. If this is the opening boundary, there won't be a",
86+
" // subcontroller yet.",
87+
" subController?.close();",
88+
" subController = null;",
89+
"",
90+
" if (!initialized) {",
91+
" initialized = true;",
92+
" boundarySplit = Buffer.from(`\\r\\n${boundarySplit}`);",
93+
" }",
94+
" }",
95+
"",
96+
" if (buffer.length > bufferKeepLength) {",
97+
" await enqueueSub(buffer.subarray(0, -bufferKeepLength));",
98+
" buffer = buffer.subarray(-bufferKeepLength);",
99+
" }",
100+
" },",
101+
" close() {",
102+
" if (!/--(\\r\\n)?/.test(buffer.toString(\"utf-8\"))) {",
103+
" readableController.error(new Error(\"Unexpected characters after final boundary.\"));",
104+
" }",
105+
"",
106+
" subController?.close();",
107+
"",
108+
" readableController.close();",
109+
" },",
110+
" });",
111+
"",
112+
" async function enqueueSub(s: Buffer) {",
113+
" subController ??= await new Promise<ReadableStreamDefaultController>((resolve) => {",
114+
" readableController.enqueue(",
115+
" new ReadableStream<Buffer>({",
116+
" start: (controller) => resolve(controller),",
117+
" }),",
118+
" );",
119+
" });",
120+
"",
121+
" subController.enqueue(s);",
122+
" }",
123+
"",
124+
" return { readable, writable };",
125+
"}",
126+
"",
127+
"const BUF_CRLFCRLF = Buffer.from(\"\\r\\n\\r\\n\");",
128+
"",
129+
"/**",
130+
" * Consumes a stream of the contents of a single part of a multipart request and emits an `HttpPart` object for each part.",
131+
" * This consumes just enough of the stream to read the headers, and then forwards the rest of the stream as the body.",
132+
" */",
133+
"class HttpPartTransform extends TransformStream<ReadableStream<Buffer>, HttpPart> {",
134+
" constructor() {",
135+
" super({",
136+
" transform: async (partRaw, controller) => {",
137+
" const reader = partRaw.getReader();",
138+
"",
139+
" let buf = Buffer.alloc(0);",
140+
" let idx;",
141+
"",
142+
" while ((idx = buf.indexOf(BUF_CRLFCRLF)) === -1) {",
143+
" const { done, value } = await reader.read();",
144+
" if (done) {",
145+
" throw new Error(\"Unexpected end of part.\");",
146+
" }",
147+
" buf = Buffer.concat([buf, value]);",
148+
" }",
149+
"",
150+
" const headerText = buf.subarray(0, idx).toString(\"utf-8\").trim();",
151+
"",
152+
" const headers = Object.fromEntries(",
153+
" headerText.split(\"\\r\\n\").map((line) => {",
154+
" const [name, value] = line.split(\": \", 2);",
155+
"",
156+
" return [name.toLowerCase(), value];",
157+
" }),",
158+
" ) as { [k: string]: string };",
159+
"",
160+
" const body = new ReadableStream<Buffer>({",
161+
" start(controller) {",
162+
" controller.enqueue(buf.subarray(idx + BUF_CRLFCRLF.length));",
163+
" },",
164+
" async pull(controller) {",
165+
" const { done, value } = await reader.read();",
166+
"",
167+
" if (done) {",
168+
" controller.close();",
169+
" } else {",
170+
" controller.enqueue(value);",
171+
" }",
172+
" },",
173+
" });",
174+
"",
175+
" controller.enqueue({ headers, body });",
176+
" },",
177+
" });",
178+
" }",
179+
"}",
180+
"",
181+
"/**",
182+
" * Processes a request as a multipart request, returning a stream of `HttpPart` objects, each representing an individual",
183+
" * part in the multipart request.",
184+
" *",
185+
" * Only call this function if you have already validated the content type of the request and confirmed that it is a",
186+
" * multipart request.",
187+
" *",
188+
" * @throws Error if the content-type header is missing or does not contain a boundary field.",
189+
" *",
190+
" * @param request - the incoming request to parse as multipart",
191+
" * @returns a stream of HttpPart objects, each representing an individual part in the multipart request",
192+
" */",
193+
"export function createMultipartReadable(request: http.IncomingMessage): ReadableStream<HttpPart> {",
194+
" const boundary = request.headers[\"content-type\"]",
195+
" ?.split(\";\")",
196+
" .find((s) => s.includes(\"boundary=\"))",
197+
" ?.split(\"=\", 2)[1];",
198+
" if (!boundary) {",
199+
" throw new Error(\"Invalid request: missing boundary in content-type.\");",
200+
" }",
201+
"",
202+
" const bodyStream = new ReadableStream<Uint8Array>({",
203+
" start(controller) {",
204+
" request.on(\"data\", (chunk: Buffer) => {",
205+
" controller.enqueue(chunk);",
206+
" });",
207+
" request.on(\"end\", () => controller.close());",
208+
" },",
209+
" });",
210+
"",
211+
" return bodyStream",
212+
" .pipeThrough(MultipartBoundaryTransformStream(boundary))",
213+
" .pipeThrough(new HttpPartTransform());",
214+
"}",
215+
"",
216+
"// Gross polyfill because Safari doesn't support this yet.",
217+
"//",
218+
"// https://bugs.webkit.org/show_bug.cgi?id=194379",
219+
"// https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream#browser_compatibility",
220+
"(ReadableStream.prototype as any)[Symbol.asyncIterator] ??= async function* () {",
221+
" const reader = this.getReader();",
222+
" try {",
223+
" while (true) {",
224+
" const { done, value } = await reader.read();",
225+
" if (done) return value;",
226+
" yield value;",
227+
" }",
228+
" } finally {",
229+
" reader.releaseLock();",
230+
" }",
231+
"};",
232+
"",
233+
"declare global {",
234+
" interface ReadableStream<R> {",
235+
" [Symbol.asyncIterator](): AsyncIterableIterator<R>;",
236+
" }",
237+
"}",
238+
"",
239+
];
240+
241+
export async function createModule(parent: Module): Promise<Module> {
242+
if (module) return module;
243+
244+
module = {
245+
name: "multipart",
246+
cursor: parent.cursor.enter("multipart"),
247+
imports: [],
248+
declarations: [],
249+
};
250+
251+
module.declarations.push(lines);
252+
253+
parent.declarations.push(module);
254+
255+
return module;
256+
}

0 commit comments

Comments
 (0)