Skip to content

Commit 76e8ffb

Browse files
authored
Merge pull request #5 from ivancorrea/main
feat: Add `.reconstructBuffers()` Fluent API method
2 parents 25a603e + dba8484 commit 76e8ffb

File tree

9 files changed

+489
-149
lines changed

9 files changed

+489
-149
lines changed

β€ŽDOCS.mdβ€Ž

Lines changed: 134 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,8 @@
1717
7. [Data Flow](#data-flow)
1818
8. [Error Handling](#error-handling)
1919
9. [Memory Management](#memory-management)
20-
10. [Runtime & Bundler Compatibility](#runtime--bundler-compatibility)
21-
11. [Contributing Guide](#contributing-guide)
20+
10. [Runtime & Bundler Compatibility](#runtime--bundler-compatibility)
21+
11. [Contributing Guide](#contributing-guide)
2222

2323
---
2424

@@ -308,16 +308,17 @@ Implements the immutable builder pattern for task execution.
308308

309309
**Chainable methods:**
310310

311-
| Method | Description |
312-
| -------------------------- | ------------------------------------------ |
313-
| `.usingParams(...args)` | Pass arguments to the function |
314-
| `.setContext(obj)` | Inject external variables (closures) |
315-
| `.signal(AbortSignal)` | Enable cancellation |
316-
| `.retry(options)` | Auto-retry on failure |
317-
| `.priority(level)` | Set queue priority |
318-
| `.transfer([ArrayBuffer])` | Zero-copy for large binary data |
319-
| `.noCoalesce()` | Skip request coalescing for this execution |
320-
| `.execute()` | Run the function |
311+
| Method | Description |
312+
| -------------------------- | -------------------------------------------- |
313+
| `.usingParams(...args)` | Pass arguments to the function |
314+
| `.setContext(obj)` | Inject external variables (closures) |
315+
| `.signal(AbortSignal)` | Enable cancellation |
316+
| `.retry(options)` | Auto-retry on failure |
317+
| `.priority(level)` | Set queue priority |
318+
| `.transfer([ArrayBuffer])` | Zero-copy for large binary data |
319+
| `.noCoalesce()` | Skip request coalescing for this execution |
320+
| `.reconstructBuffers()` | Convert Uint8Array back to Buffer in results |
321+
| `.execute()` | Run the function |
321322

322323
**Example:**
323324

@@ -362,6 +363,16 @@ Enables streaming results from generator functions.
362363
- Captures return value for access after completion
363364
- Handles cleanup on cancel
364365

366+
**Chainable methods:**
367+
368+
| Method | Description |
369+
| ----------------------- | -------------------------------------------- |
370+
| `.usingParams(...args)` | Pass arguments to the generator |
371+
| `.setContext(obj)` | Inject external variables (closures) |
372+
| `.transfer([...]))` | Zero-copy for large binary data |
373+
| `.reconstructBuffers()` | Convert Uint8Array back to Buffer in results |
374+
| `.execute()` | Start streaming |
375+
365376
**Example:**
366377

367378
```js
@@ -382,6 +393,22 @@ for await (const value of stream) {
382393
console.log(stream.returnValue) // 'done'
383394
```
384395

396+
**With Buffer reconstruction:**
397+
398+
```js
399+
const stream = beeThreads
400+
.stream(function* () {
401+
yield require('fs').readFileSync('chunk1.bin')
402+
yield require('fs').readFileSync('chunk2.bin')
403+
})
404+
.reconstructBuffers()
405+
.execute()
406+
407+
for await (const chunk of stream) {
408+
console.log(Buffer.isBuffer(chunk)) // true βœ…
409+
}
410+
```
411+
385412
---
386413

387414
### `src/cache.ts` - LRU Function Cache
@@ -562,32 +589,32 @@ executeTurboReduce(fn, data, initial, options) // Parallel tree reduction
562589

563590
**TurboExecutor methods:**
564591

565-
| Method | Description |
566-
| ----------------- | -------------------------------------- |
567-
| `.map(data)` | Transform each item in parallel |
568-
| `.mapWithStats()` | Map with execution statistics |
569-
| `.filter(data)` | Filter items in parallel |
592+
| Method | Description |
593+
| --------------------- | ------------------------------------ |
594+
| `.map(data)` | Transform each item in parallel |
595+
| `.mapWithStats()` | Map with execution statistics |
596+
| `.filter(data)` | Filter items in parallel |
570597
| `.reduce(data, init)` | Reduce using parallel tree reduction |
571598

572599
**Example:**
573600

574601
```js
575602
// Map - transform each item
576-
const squares = await beeThreads.turbo((x) => x * x).map(numbers);
603+
const squares = await beeThreads.turbo(x => x * x).map(numbers)
577604

578605
// With TypedArray (SharedArrayBuffer - zero-copy)
579-
const data = new Float64Array(1_000_000);
580-
const result = await beeThreads.turbo((x) => Math.sqrt(x)).map(data);
606+
const data = new Float64Array(1_000_000)
607+
const result = await beeThreads.turbo(x => Math.sqrt(x)).map(data)
581608

582609
// Filter
583-
const evens = await beeThreads.turbo((x) => x % 2 === 0).filter(numbers);
610+
const evens = await beeThreads.turbo(x => x % 2 === 0).filter(numbers)
584611

585612
// Reduce
586-
const sum = await beeThreads.turbo((a, b) => a + b).reduce(numbers, 0);
613+
const sum = await beeThreads.turbo((a, b) => a + b).reduce(numbers, 0)
587614

588615
// With stats
589-
const { data, stats } = await beeThreads.turbo((x) => heavyMath(x)).mapWithStats(arr);
590-
console.log(stats.speedupRatio); // "7.2x"
616+
const { data, stats } = await beeThreads.turbo(x => heavyMath(x)).mapWithStats(arr)
617+
console.log(stats.speedupRatio) // "7.2x"
591618
```
592619

593620
---
@@ -783,6 +810,24 @@ await sleep(1000)
783810

784811
// Exponential backoff with jitter
785812
calculateBackoff(attempt, baseDelay, maxDelay, factor)
813+
814+
// Reconstruct Buffer from Uint8Array after postMessage serialization
815+
reconstructBuffers(value)
816+
```
817+
818+
**Why reconstructBuffers()?**
819+
820+
The Structured Clone Algorithm used by `postMessage` converts `Buffer` to `Uint8Array`.
821+
This function recursively converts them back to `Buffer` for compatibility with
822+
libraries like Sharp that expect `Buffer` returns.
823+
824+
```typescript
825+
function reconstructBuffers(value: unknown): unknown {
826+
if (value instanceof Uint8Array && !(value instanceof Buffer)) {
827+
return Buffer.from(value.buffer, value.byteOffset, value.byteLength)
828+
}
829+
// Recursively handle arrays and plain objects...
830+
}
786831
```
787832

788833
---
@@ -891,7 +936,7 @@ calculateBackoff(attempt, baseDelay, maxDelay, factor)
891936
**Performance characteristics:**
892937

893938
| Array Size | Single Worker | Turbo (8 cores) | Speedup |
894-
|------------|---------------|-----------------|---------|
939+
| ---------- | ------------- | --------------- | ------- |
895940
| 10K items | 45ms | 20ms | 2.2x |
896941
| 100K items | 450ms | 120ms | 3.7x |
897942
| 1M items | 4.2s | 580ms | 7.2x |
@@ -943,23 +988,24 @@ calculateBackoff(attempt, baseDelay, maxDelay, factor)
943988
**Decision:** Enable security protections by default with opt-out config.
944989

945990
**Rationale:**
946-
- Security should be the default state
947-
- Transparent protections don't affect normal use cases
948-
- Users who need to disable can do so explicitly
949-
- Follows principle of least surprise
991+
992+
- Security should be the default state
993+
- Transparent protections don't affect normal use cases
994+
- Users who need to disable can do so explicitly
995+
- Follows principle of least surprise
950996

951997
---
952998

953999
## Security Architecture
9541000

9551001
### Built-in Protections
9561002

957-
| Protection | Type | Default | Configurable |
958-
|------------|------|---------|--------------|
959-
| **Function size limit** | DoS prevention | 1MB | `security.maxFunctionSize` |
960-
| **Prototype pollution block** | Injection prevention | Enabled | `security.blockPrototypePollution` |
961-
| **vm.Script sandboxing** | Isolation | Always | No |
962-
| **data: URL workers** | CSP-friendly | Auto-detected | No |
1003+
| Protection | Type | Default | Configurable |
1004+
| ----------------------------- | -------------------- | ------------- | ---------------------------------- |
1005+
| **Function size limit** | DoS prevention | 1MB | `security.maxFunctionSize` |
1006+
| **Prototype pollution block** | Injection prevention | Enabled | `security.blockPrototypePollution` |
1007+
| **vm.Script sandboxing** | Isolation | Always | No |
1008+
| **data: URL workers** | CSP-friendly | Auto-detected | No |
9631009

9641010
### Function Size Limit
9651011

@@ -968,12 +1014,10 @@ Prevents DoS attacks via extremely large function strings:
9681014
```typescript
9691015
// src/validation.ts
9701016
export function validateFunctionSize(fnString: string, maxSize: number): void {
971-
const size = Buffer.byteLength(fnString, 'utf8');
972-
if (size > maxSize) {
973-
throw new RangeError(
974-
`Function source exceeds maximum size (${size} bytes > ${maxSize} bytes limit)`
975-
);
976-
}
1017+
const size = Buffer.byteLength(fnString, 'utf8')
1018+
if (size > maxSize) {
1019+
throw new RangeError(`Function source exceeds maximum size (${size} bytes > ${maxSize} bytes limit)`)
1020+
}
9771021
}
9781022
```
9791023

@@ -984,26 +1028,24 @@ Blocks dangerous keys in context objects:
9841028
```typescript
9851029
// src/validation.ts
9861030
export function validateContextSecurity(context: Record<string, unknown>): void {
987-
const keys = Object.keys(context);
988-
for (const key of keys) {
989-
if (key === 'constructor' || key === 'prototype') {
990-
throw new TypeError(
991-
`Context key "${key}" is not allowed (potential prototype pollution)`
992-
);
993-
}
994-
}
1031+
const keys = Object.keys(context)
1032+
for (const key of keys) {
1033+
if (key === 'constructor' || key === 'prototype') {
1034+
throw new TypeError(`Context key "${key}" is not allowed (potential prototype pollution)`)
1035+
}
1036+
}
9951037
}
9961038
```
9971039

9981040
### Configuration
9991041

10001042
```js
10011043
beeThreads.configure({
1002-
security: {
1003-
maxFunctionSize: 2 * 1024 * 1024, // 2MB (default: 1MB)
1004-
blockPrototypePollution: false // Disable if you know what you're doing
1005-
}
1006-
});
1044+
security: {
1045+
maxFunctionSize: 2 * 1024 * 1024, // 2MB (default: 1MB)
1046+
blockPrototypePollution: false, // Disable if you know what you're doing
1047+
},
1048+
})
10071049
```
10081050

10091051
---
@@ -1178,37 +1220,37 @@ await beeThreads
11781220

11791221
bee-threads supports multiple JavaScript runtimes:
11801222

1181-
| Runtime | Status | Notes |
1182-
|---------|--------|-------|
1183-
| **Node.js** | βœ… Full support | v16+ recommended, uses native `worker_threads` |
1184-
| **Bun** | βœ… Full support | Uses Bun's `worker_threads` compatibility layer |
1185-
| **Deno** | ⚠️ Experimental | Requires `--allow-read` flag, limited testing |
1223+
| Runtime | Status | Notes |
1224+
| ----------- | --------------- | ----------------------------------------------- |
1225+
| **Node.js** | βœ… Full support | v16+ recommended, uses native `worker_threads` |
1226+
| **Bun** | βœ… Full support | Uses Bun's `worker_threads` compatibility layer |
1227+
| **Deno** | ⚠️ Experimental | Requires `--allow-read` flag, limited testing |
11861228

11871229
**Runtime detection:**
11881230

11891231
```typescript
11901232
// src/config.ts
11911233
export function detectRuntime(): Runtime {
1192-
if (typeof globalThis.Bun !== 'undefined') return 'bun';
1193-
if (typeof globalThis.Deno !== 'undefined') return 'deno';
1194-
return 'node';
1234+
if (typeof globalThis.Bun !== 'undefined') return 'bun'
1235+
if (typeof globalThis.Deno !== 'undefined') return 'deno'
1236+
return 'node'
11951237
}
11961238

1197-
export const RUNTIME = detectRuntime();
1198-
export const IS_BUN = RUNTIME === 'bun';
1239+
export const RUNTIME = detectRuntime()
1240+
export const IS_BUN = RUNTIME === 'bun'
11991241
```
12001242

12011243
### Bundler Compatibility
12021244

12031245
bee-threads works with **all major bundlers** without any configuration:
12041246

1205-
| Bundler | Status | Notes |
1206-
|---------|--------|-------|
1207-
| **Webpack** | βœ… Works | No config needed |
1208-
| **Vite** | βœ… Works | No config needed |
1209-
| **Rspack** | βœ… Works | No config needed |
1210-
| **esbuild** | βœ… Works | No config needed |
1211-
| **Rollup** | βœ… Works | No config needed |
1247+
| Bundler | Status | Notes |
1248+
| ------------- | -------- | ---------------- |
1249+
| **Webpack** | βœ… Works | No config needed |
1250+
| **Vite** | βœ… Works | No config needed |
1251+
| **Rspack** | βœ… Works | No config needed |
1252+
| **esbuild** | βœ… Works | No config needed |
1253+
| **Rollup** | βœ… Works | No config needed |
12121254
| **Turbopack** | βœ… Works | No config needed |
12131255

12141256
### How Bundler Compatibility Works
@@ -1219,7 +1261,7 @@ Traditional `worker_threads` require external `.js` files:
12191261

12201262
```js
12211263
// ❌ This breaks with bundlers - worker.js won't be included
1222-
const worker = new Worker(path.join(__dirname, 'worker.js'));
1264+
const worker = new Worker(path.join(__dirname, 'worker.js'))
12231265
```
12241266

12251267
Bundlers (Webpack, Vite, etc.) don't automatically include worker files in the bundle.
@@ -1234,11 +1276,11 @@ export const INLINE_WORKER_CODE = `
12341276
'use strict';
12351277
const { parentPort, workerData } = require('worker_threads');
12361278
// ... complete worker code as string ...
1237-
`;
1279+
`
12381280

12391281
export function createWorkerDataUrl(code: string): string {
1240-
const base64 = Buffer.from(code, 'utf-8').toString('base64');
1241-
return `data:text/javascript;base64,${base64}`;
1282+
const base64 = Buffer.from(code, 'utf-8').toString('base64')
1283+
return `data:text/javascript;base64,${base64}`
12421284
}
12431285
```
12441286

@@ -1247,39 +1289,33 @@ export function createWorkerDataUrl(code: string): string {
12471289
```typescript
12481290
// src/config.ts
12491291
function detectBundlerMode(): boolean {
1250-
// Check 1: Worker file doesn't exist (bundled scenario)
1251-
const workerPath = path.join(__dirname, 'worker.js');
1252-
if (!fs.existsSync(workerPath)) return true;
1253-
1254-
// Check 2: Known bundler globals
1255-
if (
1256-
typeof __webpack_require__ !== 'undefined' ||
1257-
typeof __vite_ssr_import__ !== 'undefined' ||
1258-
typeof __rspack_require__ !== 'undefined'
1259-
) return true;
1260-
1261-
return false;
1292+
// Check 1: Worker file doesn't exist (bundled scenario)
1293+
const workerPath = path.join(__dirname, 'worker.js')
1294+
if (!fs.existsSync(workerPath)) return true
1295+
1296+
// Check 2: Known bundler globals
1297+
if (typeof __webpack_require__ !== 'undefined' || typeof __vite_ssr_import__ !== 'undefined' || typeof __rspack_require__ !== 'undefined') return true
1298+
1299+
return false
12621300
}
12631301

12641302
export function getWorkerScript(type: PoolType): string {
1265-
if (USE_INLINE_WORKERS) {
1266-
const code = type === 'generator'
1267-
? INLINE_GENERATOR_WORKER_CODE
1268-
: INLINE_WORKER_CODE;
1269-
return createWorkerDataUrl(code);
1270-
}
1271-
return path.join(__dirname, 'worker.js');
1303+
if (USE_INLINE_WORKERS) {
1304+
const code = type === 'generator' ? INLINE_GENERATOR_WORKER_CODE : INLINE_WORKER_CODE
1305+
return createWorkerDataUrl(code)
1306+
}
1307+
return path.join(__dirname, 'worker.js')
12721308
}
12731309
```
12741310

12751311
### Security Considerations
12761312

12771313
**Why `data:` URLs instead of `eval: true`?**
12781314

1279-
| Approach | Security | CSP Compatible | Performance |
1280-
|----------|----------|----------------|-------------|
1281-
| `eval: true` | ⚠️ Risky | ❌ Blocked by CSP | βœ… Fast |
1282-
| `data:` URL | βœ… Safe | βœ… Works | βœ… Fast |
1315+
| Approach | Security | CSP Compatible | Performance |
1316+
| ------------ | -------- | ----------------- | ----------- |
1317+
| `eval: true` | ⚠️ Risky | ❌ Blocked by CSP | βœ… Fast |
1318+
| `data:` URL | βœ… Safe | βœ… Works | βœ… Fast |
12831319

12841320
The worker code is **static** (not user input), so `data:` URLs are safe and work with Content Security Policy.
12851321

0 commit comments

Comments
Β (0)