Skip to content

Comments

docs: add data pipeline testing spike findings#475

Merged
mikelane merged 2 commits intomainfrom
issue-435-data-pipeline-spike
Feb 7, 2026
Merged

docs: add data pipeline testing spike findings#475
mikelane merged 2 commits intomainfrom
issue-435-data-pipeline-spike

Conversation

@mikelane
Copy link
Owner

@mikelane mikelane commented Feb 7, 2026

Summary

  • Researched data engineering stacks (Airflow 320M downloads, Prefect 32M, Dagster 15M) and testing pain points
  • Assessed dioxide API fitness for data pipeline use cases (no API changes needed)
  • Evaluated framework integrations: recommends Airflow guide + lightweight module, skip Dagster/Prefect
  • Produced priority-ranked content and DX improvement recommendations with T-shirt size estimates
  • Includes a complete ETL pipeline code sketch (ports, prod adapters, test fakes, service, tests)

Key Findings

The existing dioxide API (@adapter.for_(), @service, Profile, fresh_container) already handles data pipeline dependency injection well. The gap is documentation and examples that speak to data engineers, not API changes.

Top recommendations:

  1. P0: Data pipeline testing guide in docs/guides/
  2. P1: Runnable data pipeline example in examples/data_pipeline/
  3. P2: Use data pipeline as case study for mock-vs-fake comparison (Write Mock vs Fake comparison page with code examples #394)
  4. P3: dev.to article targeting data engineers

Test plan

  • Document is well-structured with actionable next steps
  • Code sketch uses correct dioxide API patterns
  • No slop (verified: no em-dashes, semicolons, or filler words)
  • Next steps are concrete enough to file as individual issues

Fixes #435

Generated with Claude Code

@codecov
Copy link

codecov bot commented Feb 7, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.85%. Comparing base (9f37e0b) to head (ad30780).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #475   +/-   ##
=======================================
  Coverage   91.85%   91.85%           
=======================================
  Files          21       21           
  Lines        1903     1903           
  Branches      372      372           
=======================================
  Hits         1748     1748           
  Misses         91       91           
  Partials       64       64           
Flag Coverage Δ
python-3.13 91.85% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted file tree graph

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- Fix PyPI download numbers with verified pypistats.org data (Feb 2026)
- Correct TaskFlow API attribution (Airflow 2.0, not 3.0)
- Use Container(profile=...) autoscan in Airflow example
- Add @pytest.mark.asyncio and async to test methods using async fixtures
@mikelane mikelane merged commit a3a602f into main Feb 7, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spike: Investigate data pipeline testing DX and documentation strategy

1 participant