1 build a typescriptnode worker for batch processing hls stream files into pcm wav

[imanisima]

Summary of Changes

This PR enables both CLI and REST API workflows for generating audio/text pairs from HLS streams, with full support for WAV, FLAC, and PSD outputs, and is ready for local or containerized deployment. Changes include:

• REST API Layer: Added Express-based local API with /clip, /batch, and /pairs endpoints for programmatic audio clip extraction and batch processing.

• FLAC Support: All interfaces, CLI, and documentation now support flac as a valid output format alongside wav and psd.

• Output & Metadata: Output files (.wav, .flac, .psd.json, .meta.json) use deterministic, hashed paths. Metadata includes all relevant job and processing details.

• Batch Processing: Improved batch job handling, summary output (pairs.json), and error reporting for both CLI and API.

• Documentation: README updated with new API usage, CLI examples, and dependency instructions. Added example API responses and clarified all supported formats.

• Docker & Dependencies: Dockerfile, updated to include all required dependencies for both CLI and API usage.

• Code Quality: Improved error handling and logging throughout the codebase.

[paulcretu]

Thanks for the contribution @imanisima! I tried running it in the docker container, but got some errors—not sure if they repro for you, I didn't really investigate any of them but curious if you have any ideas.

Overall the structure and general idea behind this PR is great, just want to make sure I know how to run it!

Also I think a docker compose and/or devcontainers configuration would be helpful. Though probably better as a future PR, can open a new issue for that if you like.

[paulcretu]

I tried to run things inside the provided docker container, and this is what I got:

>>> docker build -t ambient-sound-api .
[+] Building 1.2s (11/11) FINISHED                                                                                     docker:orbstack
 => [internal] load build definition from Dockerfile                                                                              0.0s
 => => transferring dockerfile: 336B                                                                                              0.0s
 => [internal] load metadata for docker.io/library/node:18                                                                        0.7s
 => [internal] load .dockerignore                                                                                                 0.0s
 => => transferring context: 2B                                                                                                   0.0s
 => [1/6] FROM docker.io/library/node:18@sha256:c6ae79e38498325db67193d391e6ec1d224d96c693a8a4d943498556716d3783                  0.0s
 => [internal] load build context                                                                                                 0.0s
 => => transferring context: 42.26kB                                                                                              0.0s
 => CACHED [2/6] WORKDIR /app                                                                                                     0.0s
 => CACHED [3/6] COPY package*.json ./                                                                                            0.0s
 => CACHED [4/6] RUN npm install                                                                                                  0.0s
 => [5/6] COPY . .                                                                                                                0.1s
 => [6/6] RUN mkdir -p /app/output/clips /app/tmp                                                                                 0.2s
 => exporting to image                                                                                                            0.1s
 => => exporting layers                                                                                                           0.1s
 => => writing image sha256:8f48149b27a2fdc5ee21c4d8ed198df8d047c6886359093b85bebbf35720c36c                                      0.0s
 => => naming to docker.io/library/ambient-sound-api                                                                              0.0s

>>> docker run -it -p 3000:3000 -v $(pwd)/output:/app/output ambient-sound-api bash

root@a78cbd886e96:/app# npx ts-node src/api.ts
Need to install the following packages:
ts-node@10.9.2
Ok to proceed? (y) y

(node:21) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `node --trace-warnings ...` to show where the warning was created)
/app/src/api.ts:17
import express from 'express';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at internalCompileFunction (node:internal/vm:76:18)
    at wrapSafe (node:internal/modules/cjs/loader:1283:20)
    at Module._compile (node:internal/modules/cjs/loader:1328:27)
    at Module.m._compile (/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/index.ts:1618:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1422:10)
    at Object.require.extensions.<computed> [as .ts] (/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1203:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1019:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12)
    at phase4 (/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/bin.ts:649:14)

When I use tsx it runs fine though:

root@a78cbd886e96:/app# npx tsx src/api.ts
Need to install the following packages:
tsx@4.20.5
Ok to proceed? (y) y

REST API listening on port 3000

[paulcretu]

When I try to run this with the API running (npx tsx src/api.ts) I get:

>>> curl -X POST http://localhost:3000/clip -H 'Content-Type: application/json' -d '{
    "feedSlug": "orcasound_lab",
    "start": "2025-09-17T12:00:00Z",
    "end": "2025-09-17T12:05:00Z",
    "formats": ["wav", "flac", "psd"],
    "out": "local"
  }'
{"error":"Failed to fetch playlist for orcasound_lab: Error: m3u8Url is required"}%

and in the container:

root@6763d1ae1f8f:/app# npx tsx src/api.ts
REST API listening on port 3000
[orcasound_lab] ERROR: Failed to fetch playlist: undefined

[paulcretu]

When I try to run this inside the docker container:

>>> docker run -it -p 3000:3000 -v $(pwd)/output:/app/output ambient-sound-api bash
root@e34dbb1f2cd0:/app# npx ts-node src/cli.ts \
  --feedSlug orcasound_lab \
  --start 2025-09-17T12:00:00Z \
  --end 2025-09-17T12:05:00Z \
  --formats wav flac psd \
  --out local
Need to install the following packages:
ts-node@10.9.2
Ok to proceed? (y) y

/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/index.ts:859
    return new TSError(diagnosticText, diagnosticCodes, diagnostics);
           ^
TSError: ⨯ Unable to compile TypeScript:
src/cli.ts:4:19 - error TS7016: Could not find a declaration file for module 'yargs'. '/app/node_modules/yargs/index.mjs' implicitly has an 'any' type.
  Try `npm i --save-dev @types/yargs` if it exists or add a new declaration (.d.ts) file containing `declare module 'yargs';`

4 import yargs from 'yargs';
                    ~~~~~~~
src/cli.ts:33:19 - error TS7006: Parameter 'argv' implicitly has an 'any' type.

33     .demandOption(argv => argv.manifest ? [] : ['feedSlug','start','end'])
                     ~~~~

    at createTSError (/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/index.ts:859:12)
    at reportTSError (/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/index.ts:863:19)
    at getOutput (/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/index.ts:1077:36)
    at Object.compile (/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/index.ts:1433:41)
    at Module.m._compile (/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/index.ts:1617:30)
    at Module._extensions..js (node:internal/modules/cjs/loader:1422:10)
    at Object.require.extensions.<computed> [as .ts] (/root/.npm/_npx/1bf7c3c15bf47d04/node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1203:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1019:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12) {
  diagnosticCodes: [ 7016, 7006 ]
}

and if I use tsx instead of ts-node:

root@e34dbb1f2cd0:/app# npx tsx src/cli.ts \
  --feedSlug orcasound_lab \
  --start 2025-09-17T12:00:00Z \
  --end 2025-09-17T12:05:00Z \
  --formats wav flac psd \
  --out local
Need to install the following packages:
tsx@4.20.5
Ok to proceed? (y) y

/app/node_modules/yargs-parser/build/lib/index.js:25
        throw Error(`yargs parser supports a minimum Node.js version of ${minNodeVersion}. Read our version support policy: https://github.com/yargs/yargs-parser#supported-nodejs-versions`);
              ^


Error: yargs parser supports a minimum Node.js version of 20. Read our version support policy: https://github.com/yargs/yargs-parser#supported-nodejs-versions
    at require (/app/node_modules/yargs-parser/build/lib/index.js:25:15)
    at Object.<anonymous> (/app/node_modules/yargs-parser/build/lib/index.js:62:16)
    at Module._compile (node:internal/modules/cjs/loader:1364:14)
    at Object.transformer (/root/.npm/_npx/fd45a72a545557e9/node_modules/tsx/dist/register-D46fvsV_.cjs:3:1104)
    at Module.load (node:internal/modules/cjs/loader:1203:32)
    at Module._load (node:internal/modules/cjs/loader:1019:12)
    at Module.require (node:internal/modules/cjs/loader:1231:19)
    at require (node:internal/modules/helpers:177:18)
    at y18n (/app/node_modules/yargs/lib/platform-shims/esm.mjs:8:20)
    at Object.<anonymous> (/app/node_modules/yargs/lib/platform-shims/esm.mjs:67:1)

Node.js v18.20.8

Seems like node needs a version bump in the dockerfile?

[paulcretu]

Though when I bump node (to v22) I still get this:

root@9d115ee5151a:/app# npx tsx src/cli.ts \
  --feedSlug orcasound_lab \
  --start 2025-09-17T12:00:00Z \
  --end 2025-09-17T12:05:00Z \
  --formats wav flac psd \
  --out local
Need to install the following packages:
tsx@4.20.5
Ok to proceed? (y) y

TypeError: import_yargs.default.option is not a function
    at main (/app/src/cli.ts:24:6)
    at fs (/app/src/cli.ts:108:1)
    at Object.<anonymous> (/app/src/cli.ts:111:2)
    at Module._compile (node:internal/modules/cjs/loader:1706:14)
    at Object.transformer (/root/.npm/_npx/fd45a72a545557e9/node_modules/tsx/dist/register-D46fvsV_.cjs:3:1104)
    at Module.load (node:internal/modules/cjs/loader:1441:32)
    at Function._load (node:internal/modules/cjs/loader:1263:12)
    at TracingChannel.traceSync (node:diagnostics_channel:322:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:237:24)
    at cjsLoader (node:internal/modules/esm/translators:268:5)

[imanisima]

@paulcretu Thanks for reviewing this PR and for giving details on the issues you're encountering. I will troubleshoot and send a fix!