Skip to content

fix: drop inferred ingest pipeline for CSV and auto-detect delimited format#62

Merged
walterra merged 2 commits intomainfrom
fix-csv-infer-mappings
Mar 3, 2026
Merged

fix: drop inferred ingest pipeline for CSV and auto-detect delimited format#62
walterra merged 2 commits intomainfrom
fix-csv-infer-mappings

Conversation

@walterra
Copy link
Owner

@walterra walterra commented Mar 3, 2026

Summary

  • Drop the ingest pipeline returned by _text_structure/find_structure when the detected format is delimited (CSV). Client-side parsing already handles CSV splitting, so the server-side pipeline is redundant and can cause conflicts.
  • Expose detectedFormat from the inference result so main.js can auto-switch sourceFormat to csv when the structure is detected as delimited.
  • Only set params.format when sourceFormat is not ndjson, and fall through to the actual sourceFormat value instead of hardcoding ndjson as the default.

Test plan

  • Updated infer_mappings.test.js to assert no ingest pipeline is created for CSV
  • Run yarn test locally (requires Docker)
  • CI passes on all ES version matrix entries

walterra added 2 commits March 3, 2026 23:27
…format

CSV files are parsed client-side, so the ingest pipeline returned by
_text_structure/find_structure is unnecessary and can cause conflicts.
Also auto-switch sourceFormat to csv when the inferred structure is
delimited.

Made-with: Cursor
@walterra walterra merged commit 238f525 into main Mar 3, 2026
6 checks passed
@walterra walterra deleted the fix-csv-infer-mappings branch March 3, 2026 22:40
@walterra-dev-release walterra-dev-release bot mentioned this pull request Mar 3, 2026
walterra pushed a commit that referenced this pull request Mar 3, 2026
# Releases

## node-es-transformer@1.2.2

### Patch Changes

- [#62](#62)
[`238f525`](238f525)
- Fix CSV mapping inference to skip ingest pipeline and auto-detect
delimited format

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant