Commit cf56d22
authored
feat: add client.parse() for the Data Extraction API (/extraction/parse) (#12)
* chore(spec): vendor Data Extraction API OpenAPI spec (2026-05-25)
The Data Extraction API (`POST /extraction/parse`) ships on a separate
OpenAPI document from the existing DWS Processor API. Vendor the public spec
so the new typed client surface is anchored to a checked-in source of truth.
The Processor API spec stays at `dws-api-spec.yml`; the Data Extraction spec
lives alongside it at `dws-data-extraction-spec.yml`.
* feat(types): add Data Extraction API types for /extraction/parse
Introduce hand-written types mirroring the public Data Extraction OpenAPI 3.1
contract (version 2026-05-25):
- ParseMode (text | structure | understand | agentic)
- ParseOutputFormat (spatial | markdown), ParseOutputOptions
- ParseInstructions and ParseOptions request shapes
- ParseResponseSpatial / ParseResponseMarkdown discriminated by output payload
- Per-element types: ParagraphElement, FormulaElement, PictureElement,
TableElement (with ParseTableCell), KeyValueRegionElement (with
KeyValuePair / KeyValueEntity), HandwritingElement, and shared
ParseElementBase / ParseBounds / ParsePageRef / ParseWord
- ParseErrorResponse with structured failingPaths
- ParseMetrics, ParseUsage (carrying data_extraction_credits), ParseConfiguration
The Data Extraction API bills against a separate extraction-credits bucket
from the processor API; type JSDoc makes the distinction explicit so client
code does not conflate the two billing buckets.
Wires the new endpoint into RequestTypeMap / ResponseTypeMap so the existing
HTTP layer stays type-safe end-to-end.
* feat(client): support /extraction/parse with parse() and convenience wrappers
Adds first-class client methods for the Data Extraction API:
- parse(input, options?) — full-fidelity call against POST /extraction/parse,
supporting local files, buffers, streams, and URL inputs. Handles multipart
upload for binary inputs and JSON body for URL-only requests.
- parseToMarkdown(input, mode?) — convenience wrapper returning the whole-
document Markdown string directly. Defaults to mode='text' (cheapest).
- parseElements(input, mode?, includeWords?) — convenience wrapper returning
the typed spatial-elements array. Defaults to mode='structure'.
Threads x-nutrient-api-version through the HTTP layer when the caller pins
a specific API version.
JSDoc on every new method makes the billing distinction explicit: the
Data Extraction API bills against extraction credits, a separate bucket
from the processor API credits used by the rest of NutrientClient.
The full set of new types is re-exported from the package root.
* test(parse): cover request shape, modes, output formats, and error paths
Adds 19 unit tests around the new /extraction/parse surface:
- Request shape: multipart vs JSON, apiVersion header forwarding, option
serialisation (language, output, includeWords), default behaviour.
- Mode coverage: all four modes (text, structure, understand, agentic)
round-trip through the instructions payload.
- Output coverage: spatial elements and whole-document Markdown variants
validated end-to-end, including extraction-credit accounting on the
response (data_extraction_credits, not processor credits).
- Error paths: HTTP-layer ValidationError propagation, file-input
preflight failures surfaced before the request leaves the process.
- Convenience wrappers: parseToMarkdown and parseElements default modes
and includeWords forwarding, plus defensive output-mismatch errors.
Adds examples/src/parse_smoke.ts — a live operator-runnable smoke test
that prints a parsed summary plus extraction-credit usage. Documents
the build/pack/install/run recipe in the file header.
* docs: document /extraction/parse surface and extraction-credit billing
- README: new "Data Extraction (/extraction/parse)" section with mode/
credit table, request examples for spatial + Markdown outputs, URL
input, convenience wrappers, and a pointer to the smoke example.
- docs/METHODS.md: new entries for parse, parseToMarkdown, parseElements
inserted alongside the existing extract* convenience methods.
- LLM_DOC.md: inject the same three method signatures so coding agents
steered by this rule file know about parse and the extraction-credits
bucket.
- CHANGELOG.md: Unreleased entry covering the new client surface, the
newly-exported public types, the live smoke script, and an explicit
call-out that /extraction/parse bills against extraction credits
(separate from processor API credits).
Every doc surface that mentions cost says "extraction credits" explicitly
so downstream readers cannot conflate the two billing buckets.
* docs: fix smoke script path and parseElements doc fragment
- CHANGELOG: correct path to live smoke script
- METHODS.md: fix dangling sentence on parseElements compile-time guard
* refactor(types): extract ExtractionCredits to dedicated module
Factor the inline extraction-credit billing shape out of ParseUsage into a
standalone ExtractionCredits interface in src/types/extraction_credits.ts,
mirroring the Python client's type-factoring approach.
ParseUsage.data_extraction_credits now references ExtractionCredits instead
of an anonymous inline type, making the billing object reusable if future
endpoints surface the same shape.
ExtractionCredits is re-exported from the package root alongside the other
parse types.
* docs(client): rewrite parse() JSDoc with use-case-first framing
Lead with the "Designed for" preamble naming the three canonical workflows
(RAG/search indexing, form/invoice extraction, layout-aware understanding)
before describing modes and output formats.
Broaden the @PARAM input description to explicitly mention non-PDF inputs
(Office documents, images), matching the actual endpoint capability rather
than implying PDF-only like sign().
Update the @example block to show a form/invoice extraction recipe alongside
the RAG recipe, and replace the generic paragraph-walk with a keyValueRegion
traversal that a form-extraction caller can copy directly.
* docs: rewrite Data Extraction section with use-case-first framing
Restructure the README's /extraction/parse section to lead with use cases
(RAG ingestion, form/invoice extraction, layout-aware understanding) before
the mode table and code, matching the Python client's documentation approach.
Add:
- "Choosing an output format" table (markdown vs spatial, with shape and
best-for columns).
- "Modes — when to use which" table with credit costs and decision guidance.
- Two worked recipes: RAG ingestion (PDF → Markdown → embed) and
form/invoice extraction (PDF → spatial elements → structured object),
each with the convenience-wrapper alternative shown alongside.
- Explicit note that the endpoint accepts PDFs, Office documents, and
images — not PDFs only.
- Mention of the new ExtractionCredits type in the exported-types list.
Update METHODS.md parse/parseToMarkdown/parseElements entries to match:
lead with use-case positioning, add a parameters table, align examples
with the recipe pattern from the README.
* feat(client): route parse() via DWS Extract key
DWS Extract is a separate product from DWS Processor with its own API key
and credit pool. Calling /extraction/parse with the Processor key returns
403. Add an optional `extractApiKey` constructor option (string or async
getter) that parse() prefers over apiKey when set; every non-parse method
keeps using apiKey. Falls back to apiKey when extractApiKey is omitted,
so tenants with a single global DWS key still work.
The routing happens via a per-call options copy that swaps apiKey to the
extract key — leaves this.options untouched and covers both the multipart
file-input path and the JSON url-input path.
Drop the bundled parse smoke script — its dual-key dance and pack/install
recipe were superseded by the unit-test coverage of the request shape,
response handling, and routing. Live verification against a real account
belongs to ad-hoc developer sessions, not committed scaffolding.
Mirrors PR #47 on the Python sibling client.
* refactor(types): derive parse types from generated OpenAPI spec
Add `npm run generate:types:extract` that runs openapi-typescript against
the vendored dws-data-extraction-spec.yml into src/generated/extract-types.ts,
peer to the existing `generate:types` flow for the Processor spec.
Rewrite src/types/parse.ts so the schema primitives derive from the
generated `components['schemas']` rather than being hand-rolled:
- ParseMode, ParseOutputFormat
- ParseElement and the six element subtypes (ParagraphElement,
FormulaElement, PictureElement, TableElement, KeyValueRegionElement,
HandwritingElement)
- ParseElementBase, ParseBounds, ParsePageRef, ParseWord
- ParseTableCell, KeyValuePair, KeyValueEntity
- ParseMetrics, ParseUsage, ParseConfiguration
- ParseErrorResponse, ParseErrorDetails, ParseErrorFailingPath
- ParagraphRole (now `NonNullable<ParagraphElement['role']>`)
Keep four types hand-composed where they add something the spec doesn't
express:
- ParseOutputOptions / ParseInstructions — the spec marks
`OutputOptions.includeWords` as required, but the server has a default
and clients shouldn't be forced to pass it.
- ParseResponseSpatial / ParseResponseMarkdown — cross-field discriminated
narrowing (`elements?: undefined` / `markdown?: undefined`) the spec's
ParseOutput doesn't model, letting callers write
`if (output.markdown !== undefined)` without per-call `?.` access.
- ParseOptions — adds the client-only `apiVersion` header concern that
isn't a body field in the spec.
Net: ~210 lines of hand-rolled type definitions deleted, replaced with
one-line aliases that re-route through the generated schema. The public
surface (every exported name) is unchanged.
* refactor(types): collapse parse types into http.ts and namespace the spec re-export
Most APIs in this client (sign, ocr, watermark, redact, etc.) don't have a
dedicated `src/types/<api>.ts` file — they reach types via
`components['schemas']['X']` from `src/generated/api-types.ts`. The
`src/types/parse.ts` and `src/types/extraction_credits.ts` files added on
this branch were an outlier: most of their content was thin one-line
aliases over the generated extract spec.
Collapse to the rest-of-codebase pattern:
- Delete `src/types/parse.ts` (was 254 lines, mostly aliases).
- Delete `src/types/extraction_credits.ts` (single hand-rolled interface that
duplicated the generated `Usage.data_extraction_credits` shape).
- Move the 5 hand-composed types into `src/types/http.ts` (it already
imports `ParseInstructions` / `ParseResponse` to type the endpoint maps):
`ParseOutputOptions`, `ParseInstructions`, `ParseOptions`,
`ParseResponseSpatial`, `ParseResponseMarkdown`, plus the derived
`ExtractionCredits` alias. Each carries the JSDoc explaining why it's
hand-composed instead of derived.
- Drop the 23 cosmetic spec-alias exports from the package root. Consumers
who need element-subtype types reach them via the new
`extractComponents['schemas']['ParagraphElement']` namespace re-export,
mirroring how Processor types are exposed via the existing `components`
namespace.
The package's public surface still exports the 7 hand-composed types
(`ParseOutputOptions`, `ParseInstructions`, `ParseOptions`, `ParseResponse`,
`ParseResponseSpatial`, `ParseResponseMarkdown`, `ExtractionCredits`) by
name. Internal consumers (`src/client.ts`, the parse unit tests) shift to
`extractComponents['schemas']['X']` for spec-derived types.
Net: -290 lines on the type-definition surface, no behaviour change.
* fix: address code-review findings on the Data Extraction surface
Five findings from review:
1. Empty-string `extractApiKey` bypassed constructor validation.
`apiKey` uses `!options.apiKey` (falsy, catches `''`); the new
`extractApiKey` validator only checked `!== undefined` plus the type
guard, so `extractApiKey: ''` passed, propagated into the per-call
options as `apiKey: ''`, and produced `Authorization: Bearer ` with no
token — surfacing as a confusing server-side 401 instead of a
constructor-time `ValidationError`. Add an explicit empty-string check.
2. `extractErrorMessage` in `src/http.ts` checked snake_case (`error_message`,
`error_description`) and generic message fields but not `errorMessage`
(camelCase) — the field DWS Extract returns on every 4xx/5xx. Result:
the server's specific message (e.g. `"invalid mode: 'vlm'"`) was
silently replaced by the generic `HTTP <status>: <statusText>` string.
Add `errorMessage` to the priority list.
3. `parse()` accepted `mode='text' + output.format='spatial'` and let the
server reject with 400. The Python sibling client adds a client-side
`ValidationError` for this case (after reviewer feedback). The TS
`parseElements()` wrapper blocked it at the type level via `Exclude`,
but the low-level `parse()` did not. Add a pre-flight runtime guard.
4. `RequestTypeMap` JSDoc on `/extraction/parse` claimed `instructions`
was optional for multipart upload, but the type definition marks it
required and the implementation always passes it (an empty object when
no options are supplied). Update the comment to match the type.
5. `parse()` `@param options.language` JSDoc described the field as
"string or array of ISO 639-2 codes". The underlying spec also accepts
lowercase language names (`'english'`, `'german'`) and `+`-joined
multilingual strings (`'eng+spa'`). Document all four accepted forms.
Adds three unit tests (empty-string `extractApiKey`, `errorMessage`
extraction, text+spatial pre-flight rejection). 292 tests pass.1 parent 121ee7d commit cf56d22
16 files changed
Lines changed: 3213 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
11 | 35 | | |
12 | 36 | | |
13 | 37 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
461 | 461 | | |
462 | 462 | | |
463 | 463 | | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
464 | 522 | | |
465 | 523 | | |
466 | 524 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
136 | 307 | | |
137 | 308 | | |
138 | 309 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
455 | 455 | | |
456 | 456 | | |
457 | 457 | | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
458 | 564 | | |
459 | 565 | | |
460 | 566 | | |
| |||
0 commit comments