Skip to content

Add orchestration assets and ops for de-identification dataset jobs - #2241

Merged
maziyarpanahi merged 6 commits into
masterfrom
feature/om-488-orchestration-deid-assets
Aug 12, 2026
Merged

Add orchestration assets and ops for de-identification dataset jobs#2241
maziyarpanahi merged 6 commits into
masterfrom
feature/om-488-orchestration-deid-assets

Conversation

@maziyarpanahi

Copy link
Copy Markdown
Owner

Closes #845.

Summary

  • Adds an optional orchestration adapter with a configurable batch op and daily-partitioned redacted dataset asset.
  • Validates canonical policy profiles, redacts configured columns with process_batch, and emits count- and label-only metadata.
  • Adds synthetic in-memory tests and a runnable example while keeping imports safe without the optional dependency.

Validation

  • .venv/bin/python -m pytest tests/unit/integrations/test_dagster_assets.py -q — 3 passed
  • .venv/bin/ruff check openmed/integrations/dagster_assets.py tests/unit/integrations/test_dagster_assets.py examples/dagster/definitions.py
  • .venv/bin/ruff format --check openmed/integrations/dagster_assets.py tests/unit/integrations/test_dagster_assets.py examples/dagster/definitions.py
  • /Users/maziyar/.local/bin/uv lock --check
  • git diff --check

@maziyarpanahi maziyarpanahi added roadmap-v2 OpenMed V2 roadmap backlog feature New capability P2 Medium labels Aug 5, 2026
@maziyarpanahi

Copy link
Copy Markdown
Owner Author

Thank you @maziyarpanahi. I reviewed this against #845 and current master, then added maintainer follow-ups cdbedeb4, d097f83f, and 1341bb01.

What changed:

  • Made non-text Dagster asset values fail closed before de-identification.
  • Completed the optional dependency, lockfile, permissive-license, and stable installation guidance.
  • Integrated the current SQLAlchemy dependency state without dropping either optional extra.

Validation:

  • .venv/bin/python -m pytest tests/unit/integrations/test_dagster_assets.py -q — 5 passed on the final head with Dagster installed.
  • Combined Dagster and SQLAlchemy focused tests — 8 passed.
  • .venv/bin/python -m pytest tests/unit -q — 11,593 passed, 89 skipped on the exact aggregate release tree.
  • uv lock --check, Ruff lint, Ruff format check, package build, and dependency-license policy — passed.

This now satisfies the issue and is ready to merge.

@maziyarpanahi
maziyarpanahi merged commit 0cfa83f into master Aug 12, 2026
9 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New capability P2 Medium roadmap-v2 OpenMed V2 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add orchestration assets and ops for de-identification dataset jobs

1 participant