docs: add Dagster migration guide #20092
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- add comprehensive Dagster to Prefect migration guide - reorganize migration docs under "other orchestrators" group - move Airflow guide into the new structure - add index page for orchestrator migrations 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
the pytest-markdown-docs plugin requires a blank line before
{/* pmd-metadata: notest */} comments for them to work properly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- drastically shortened from ~1000 lines to ~260 lines - fixed concept mapping: Assets → Prefect Assets (Cloud), not "tasks with caching" - added cache policies as proper caching mechanism - removed fake dagster_type=dict syntax - made all Prefect examples self-contained and runnable (tests pass) - linked to install page instead of hardcoding pip - added honest notes about what Prefect doesn't have (partition state tracking) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- researched actual dagster docs (assets, resources, i/o managers, partitions, sensors) - read prefect source code (@materialize, asset_deps, blocks, caching) - added proper dagster assets → prefect assets (cloud) mapping - honest about gaps: i/o managers have no prefect equivalent - partitions → parameters + .map() with note about no state tracking - all 7 prefect examples are runnable (tests pass) - much more concise (~400 lines vs ~1000) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- add CodeGroup for side-by-side Dagster/Prefect comparisons - add narrative explaining why each concept matters to data engineers - use congruent examples (same function names, same domain) - fix I/O Managers: show result persistence as Prefect's answer - fix Resources: show custom block definition, not pre-built block - update concept mapping table (I/O Managers → Result persistence) - all 7 Prefect examples pass markdown tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- remove unnecessary persist_result=True (implied by result_storage) - remove result_serializer="json" (pickle is default, matches dagster example) - remove confusing "read inputs explicitly" statement - simplify partitions example to just show passing a parameter - move backfill .map() example to a brief snippet 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
rewrote the guide after actually running dagster in sandbox, exploring the UI, materializing partitions, and reading both frameworks' docs. key improvements: - i/o managers section correctly explains transparent read/write behavior - partitions section simplified to show dagster's state tracking vs prefect's parameter approach - removed unnecessary options (persist_result=True, result_serializer) - all examples now use congruent domains and function names 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Contributor
|
This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment. |
Contributor
|
This pull request was closed because it has been stale for 14 days with no activity. If this pull request is important or you have more to add feel free to re-open it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
adds a dagster migration guide to the "migrate from other orchestrators" section.
structure:
concept mapping:
@materialize.map()🤖 Generated with Claude Code