feat(lineage): add lineage visualization across datasets, charts and dashboards - #40912
feat(lineage): add lineage visualization across datasets, charts and dashboards#40912rusackas wants to merge 11 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #40912 +/- ##
==========================================
- Coverage 79.41% 79.34% -0.08%
==========================================
Files 2895 2898 +3
Lines 167947 168402 +455
Branches 38896 39003 +107
==========================================
+ Hits 133382 133619 +237
- Misses 32065 32276 +211
- Partials 2500 2507 +7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Code Review Agent Run #9dcb5b
Actionable Suggestions - 4
-
superset/dashboards/api.py - 1
- Missing @handle_api_exception decorator · Line 528-534
-
superset-frontend/src/features/lineage/LineageView.tsx - 2
- Missing tests for new component · Line 1-377
- CWE-561: Remove any type · Line 496-496
-
superset-frontend/src/features/lineage/LineageModal.tsx - 1
- Unconditional hook calls waste resources · Line 40-46
Additional Suggestions - 9
-
superset-frontend/src/features/lineage/LineageModal.tsx - 1
-
Missing unit tests for new component · Line 1-79The new LineageModal component lacks unit tests. Per project guidelines, new features should include tests covering success paths, error scenarios, and edge cases.
-
-
superset/charts/api.py - 1
-
CWE-285: Improper Authorization · Line 353-356ChartDAO.get_by_id_or_uuid() applies ChartFilter which returns no results for unauthorized users, causing ChartNotFoundError to be raised. This design conflates 'not found' with 'access denied'. Consider refactoring to raise ChartAccessDeniedError when filters deny access.
-
-
superset-frontend/src/components/Datasource/components/DatasourceEditor/DatasourceEditor.tsx - 1
-
Redundant Loading state · Line 432-438The `DatasetLineageTab` wrapper calls `useDatasetLineage(datasourceId ?? 0)` even when `datasourceId` is undefined, triggering a spurious `/api/v1/dataset/0/lineage` request. The check at line 434 then renders `` anyway, duplicating `LineageView`'s own loading state. Since the parent at line 2536 already has a non-null `datasource.id`, either pass the resource as a prop or return `null` for falsy `datasourceId`.
-
-
tests/integration_tests/charts/api_tests.py - 1
-
Unused import with noqa · Line 53-53The `client` import on line 53 has `# noqa: F401` but is not referenced anywhere in the diff. Either remove it or add a comment explaining why it's needed as a module-level import.
-
-
superset-frontend/src/features/charts/ChartCard.tsx - 1
-
Missing test identifier · Line 105-126The new lineage menu item is missing a `data-test` attribute, making it inconsistent with DashboardCard (which has `data-test="dashboard-card-option-lineage-button"` at line 112). This will hinder automated testing of the ChartCard component.
-
-
superset/dashboards/schemas.py - 1
-
Missing allow_none for optional field · Line 564-567The `slug` field should specify `allow_none=True` to match the API behavior where `dash.slug` can be None. Without this, marshmallow serialization may raise an error when encountering null values.
Code suggestion
--- a/superset/dashboards/schemas.py +++ b/superset/dashboards/schemas.py @@ -563,7 +563,7 @@ class DashboardLineageDashboardSchema(Schema): class DashboardLineageDashboardSchema(Schema): id = fields.Integer() title = fields.String() - slug = fields.String() + slug = fields.String(allow_none=True) published = fields.Boolean()
-
-
superset-frontend/src/explore/components/useExploreAdditionalActionsMenu/index.tsx - 1
-
Inconsistent test attribute naming · Line 1042-1042The `data-test` attribute `view-lineage-menu-item` conflicts with the existing convention in `DashboardCard.tsx` (line 112) which uses `dashboard-card-option-lineage-button` for similar lineage menu items. Use `view-lineage-menu-item-button` to maintain consistent test selector naming across the explore and dashboard features.
-
-
tests/integration_tests/dashboards/api_tests.py - 1
-
Unused import in test file · Line 72-72The `lineage_test_data` fixture is imported but never used in this file. The `inject_expected_dashboard_lineage` fixture (line 70-71) already depends on `lineage_test_data` internally, so this direct import is redundant.
Code suggestion
--- tests/integration_tests/dashboards/api_tests.py +++ tests/integration_tests/dashboards/api_tests.py @@ -68,8 +68,7 @@ ) from tests.integration_tests.fixtures.client import client # noqa: F401 from tests.integration_tests.fixtures.lineage import ( inject_expected_dashboard_lineage, # noqa: F401 - lineage_test_data, # noqa: F401 ) from tests.integration_tests.fixtures.world_bank_dashboard import (
-
-
superset-frontend/src/hooks/apiResources/index.ts - 1
-
Missing unit tests for lineage hooks · Line 32-32The `lineage.ts` module exports 3 API hooks but lacks a corresponding test file. Other modules in this directory (dashboards, datasets, queries, etc.) have dedicated `.test.ts` files following the established pattern. Per BITO.md rule [11730], new tools should include comprehensive unit tests covering success paths, error scenarios, validation failures, and edge cases.
-
Filtered by Review Rules
Bito filtered these suggestions based on rules created automatically for your feedback. Manage rules.
-
superset/dashboards/api.py - 1
- chart_ids ordering mismatch with test · Line 604-604
-
superset-frontend/src/features/datasets/AddDataset/EditDataset/index.tsx - 1
- Missing translation key · Line 55-56
-
superset-frontend/src/features/lineage/LineageView.tsx - 2
- CWE-1057: any type violates TS standards · Line 342-342
- CWE-1057: any type in Map generic · Line 496-496
-
superset/charts/api.py - 1
- CWE-862: Missing Authorization Handler · Line 317-325
-
superset-frontend/src/dashboard/components/Header/useHeaderActionsDropdownMenu.tsx - 1
- Missing menu handler case · Line 249-249
Review Details
-
Files reviewed - 23 · Commit Range:
23123ce..23123ce- superset-frontend/src/components/Datasource/components/DatasourceEditor/DatasourceEditor.tsx
- superset-frontend/src/dashboard/components/Header/useHeaderActionsDropdownMenu.tsx
- superset-frontend/src/dashboard/types.ts
- superset-frontend/src/explore/components/useExploreAdditionalActionsMenu/index.tsx
- superset-frontend/src/features/charts/ChartCard.tsx
- superset-frontend/src/features/dashboards/DashboardCard.tsx
- superset-frontend/src/features/datasets/AddDataset/EditDataset/index.tsx
- superset-frontend/src/features/lineage/LineageModal.tsx
- superset-frontend/src/features/lineage/LineageView.tsx
- superset-frontend/src/features/lineage/index.ts
- superset-frontend/src/hooks/apiResources/index.ts
- superset-frontend/src/hooks/apiResources/lineage.ts
- superset/charts/api.py
- superset/charts/schemas.py
- superset/constants.py
- superset/dashboards/api.py
- superset/dashboards/schemas.py
- superset/datasets/api.py
- superset/datasets/schemas.py
- tests/integration_tests/charts/api_tests.py
- tests/integration_tests/dashboards/api_tests.py
- tests/integration_tests/datasets/api_tests.py
- tests/integration_tests/fixtures/lineage.py
-
Files skipped - 0
-
Tools
- Eslint (Linter) - ✔︎ Successful
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
-
/review- Manually triggers a full AI review. -
/pause- Pauses automatic reviews on this pull request. -
/resume- Resumes automatic reviews. -
/resolve- Marks all Bito-posted review comments as resolved. -
/abort- Cancels all in-progress reviews.
Refer to the documentation for additional commands.
Configuration
This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.
Documentation & Help
✅ Deploy Preview for superset-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
Code Review Agent Run #e7785a
Actionable Suggestions - 1
-
superset-frontend/src/hooks/apiResources/apiResources.ts - 1
- Missing unit tests for skip parameter · Line 89-89
Additional Suggestions - 1
-
superset/charts/api.py - 1
-
Test coverage gap for permission filter · Line 389-395The existing test `test_get_chart_lineage` uses `ADMIN_USERNAME` and will continue to pass. Consider adding a test with a non-admin user who has access to only some dashboards to verify the permission filtering is effective.
-
Review Details
-
Files reviewed - 12 · Commit Range:
23123ce..f8f4010- superset-frontend/src/dashboard/components/Header/useHeaderActionsDropdownMenu.tsx
- superset-frontend/src/features/datasets/AddDataset/EditDataset/index.tsx
- superset-frontend/src/features/lineage/LineageView.tsx
- superset-frontend/src/hooks/apiResources/apiResources.ts
- superset-frontend/src/hooks/apiResources/lineage.ts
- superset/charts/api.py
- superset/dashboards/api.py
- superset/datasets/api.py
- tests/integration_tests/charts/api_tests.py
- tests/integration_tests/dashboards/api_tests.py
- tests/integration_tests/datasets/api_tests.py
- superset-frontend/src/hooks/apiResources/lineage.test.ts
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- Eslint (Linter) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
-
/review- Manually triggers a full AI review. -
/pause- Pauses automatic reviews on this pull request. -
/resume- Resumes automatic reviews. -
/resolve- Marks all Bito-posted review comments as resolved. -
/abort- Cancels all in-progress reviews.
Refer to the documentation for additional commands.
Configuration
This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.
Documentation & Help
Code Review Agent Run #8416a4Actionable Suggestions - 0Additional Suggestions - 1
Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help |
|
This might warrant an extension, but the work was already part done, thus the adoption... we can consider merging this, or leaving it behind... or @qf-jonathan can reopen theirs and I can close this out if they want to pick it up and drive it. |
There was a problem hiding this comment.
Code Review Agent Run #7c9e20
Actionable Suggestions - 2
-
superset-frontend/src/features/lineage/LineageView.tsx - 2
- SEMANTIC_DUPLICATION between useMemo hooks · Line 380-589
- Missing i18n for 8 hardcoded legend labels · Line 594-617
Additional Suggestions - 10
-
superset-frontend/src/hooks/apiResources/apiResources.ts - 1
-
Missing test coverage for skip param · Line 191-198The `useApiV1Resource` hook now forwards `skip` to `useApiResourceFullBody` but lacks dedicated unit tests. While the underlying hook is tested, adaptive rule [11730] requires coverage for new API entry points. Add tests similar to lines 101-141: one for `skip=true` and one for the toggle behavior.
-
-
superset-frontend/src/features/lineage/LineageView.tsx - 2
-
CWE-835: Missing sink for optional relationship · Line 537-556When `chart.dataset_id` is undefined, the chart appears orphaned in the sankey even though the upstream dataset references it via `chart_ids`. Add fallback logic to connect charts via their containing dataset's `chart_ids` array.
-
CWE-477: Reload instead of router navigation · Line 633-645Line 637 uses `window.location.href = getEntityUrl(selectedNode)` to navigate after clicking Open. This causes a full page reload instead of a client-side navigation, resetting React state.
-
-
tests/integration_tests/datasets/api_tests.py - 1
-
Unused import dead code · Line 58-58The `client` fixture import is unused throughout this file. Only line 58 imports it, but grep confirms no reference to `client` exists beyond that import. Dead import increases maintenance burden and may confuse readers about actual test dependencies. Remove it.
-
-
superset/datasets/schemas.py - 1
-
Missing OpenAPI field metadata · Line 253-304New DatasetLineage schemas lack OpenAPI field descriptions (metadata). The file consistently uses `metadata={"description": "..."}` on all field definitions (e.g., lines 233-244), but lines 253-304 are missing this documentation, reducing OpenAPI spec quality.
-
-
tests/integration_tests/charts/api_tests.py - 1
-
Unused import introduced by diff · Line 53-53The module-level `client` fixture (from `tests.integration_tests.fixtures.client`) is not referenced by either of the new test methods — both use `self.client` inherited from the test class base. The import is dead code in the context of these additions.
-
-
superset-frontend/src/hooks/apiResources/apiResources.test.ts - 1
-
Missing test coverage for skip reset · Line 101-141Consider adding a test case for toggling skip back to true (false→true) to complement the existing test for true→false. This would cover the scenario where a user cancels an operation mid-fetch, ensuring the loading state is preserved and no stale data appears.
-
-
superset-frontend/src/features/lineage/LineageModal.tsx - 1
-
Add unit tests for new component · Line 1-79This new component lacks unit tests. Per project guidelines, comprehensive unit tests should cover success paths and error scenarios for all entity types the component supports.
-
-
superset-frontend/src/explore/components/useExploreAdditionalActionsMenu/index.tsx - 1
-
Missing test coverage · Line 1082-1097The new VIEW_LINEAGE feature is missing test coverage. The existing test file (useExploreAdditionalActionsMenu.test.tsx) has no tests for this functionality. Per project guidelines, comprehensive unit tests should cover success paths, error scenarios, and edge cases for new features. ([11730](https://cwe.mitre.org/data/definitions/11730.html))
-
-
superset/dashboards/schemas.py - 1
-
Missing OpenAPI field descriptions · Line 638-692The new lineage schemas lack OpenAPI field descriptions. Existing schemas in this file (e.g., `DashboardGetResponseSchema`) use `metadata={"description": ...}` for API documentation. Adding descriptions improves the OpenAPI spec generated for the `/dashboard/{id}/lineage` endpoint.
-
Filtered by Review Rules
Bito filtered these suggestions based on rules created automatically for your feedback. Manage rules.
-
superset/charts/schemas.py - 8
- Missing type annotations and docstring · Line 1841-1844
- Missing type annotations and docstring · Line 1847-1853
- Missing type annotations and docstring · Line 1856-1859
- Missing type annotations and docstring · Line 1862-1865
- Missing type annotations and docstring · Line 1868-1870
- Missing type annotations and docstring · Line 1873-1875
- Missing type annotations and docstring · Line 1878-1879
- Missing type annotations and docstring · Line 1882-1885
-
superset-frontend/src/components/Datasource/components/DatasourceEditor/DatasourceEditor.tsx - 1
- CWE-754: Hook Called Before Validation · Line 420-424
-
superset-frontend/src/hooks/apiResources/apiResources.ts - 2
- Add explicit type annotation to skip · Line 89-89
- Add explicit type annotation to skip · Line 193-193
-
superset/datasets/api.py - 1
- Non-deterministic chart_ids ordering · Line 952-956
Review Details
-
Files reviewed - 26 · Commit Range:
60dacb5..ea5c33f- superset-frontend/src/components/Datasource/components/DatasourceEditor/DatasourceEditor.tsx
- superset-frontend/src/dashboard/components/Header/useHeaderActionsDropdownMenu.tsx
- superset-frontend/src/dashboard/types.ts
- superset-frontend/src/explore/components/useExploreAdditionalActionsMenu/index.tsx
- superset-frontend/src/features/charts/ChartCard.tsx
- superset-frontend/src/features/dashboards/DashboardCard.tsx
- superset-frontend/src/features/datasets/AddDataset/EditDataset/index.tsx
- superset-frontend/src/features/lineage/LineageModal.tsx
- superset-frontend/src/features/lineage/LineageView.tsx
- superset-frontend/src/features/lineage/index.ts
- superset-frontend/src/hooks/apiResources/apiResources.test.ts
- superset-frontend/src/hooks/apiResources/apiResources.ts
- superset-frontend/src/hooks/apiResources/index.ts
- superset-frontend/src/hooks/apiResources/lineage.test.ts
- superset-frontend/src/hooks/apiResources/lineage.ts
- superset/charts/api.py
- superset/charts/schemas.py
- superset/constants.py
- superset/dashboards/api.py
- superset/dashboards/schemas.py
- superset/datasets/api.py
- superset/datasets/schemas.py
- tests/integration_tests/charts/api_tests.py
- tests/integration_tests/dashboards/api_tests.py
- tests/integration_tests/datasets/api_tests.py
- tests/integration_tests/fixtures/lineage.py
-
Files skipped - 0
-
Tools
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- Eslint (Linter) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
-
/review- Manually triggers a full AI review. -
/pause- Pauses automatic reviews on this pull request. -
/resume- Resumes automatic reviews. -
/resolve- Marks all Bito-posted review comments as resolved. -
/abort- Cancels all in-progress reviews.
Refer to the documentation for additional commands.
Configuration
This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.
Documentation & Help
|
@rusackas At Airbnb, we have systems to display lineage information with much richer information. I wonder if this should be an extension of Superset instead of part of the core? |
|
Agreed @michael-s-molina - this probably ought to be an extension, and would warrant a SIP if not. I've converted it to draft for now. If y'all have an Extension that's better and on track to be open-sourced (and doesn't require additional hosted tooling) then we can probably just close this effort in favor of that effort. |
|
The chart lineage endpoint returns the upstream dataset block (schema, table_name, database_name, backend, ids) unconditionally, whereas the dashboard lineage endpoint in this same PR gates that block behind |
|
@sha174n good catch, fixed. The chart endpoint now redacts schema/table/database the same way the dashboard one does when the caller can't access the dataset. |
867327c to
08cb0f6
Compare
…dashboards Adopts #37701 by @qf-jonathan. Adds interactive Sankey-based lineage that shows the data relationships for a dataset, chart, or dashboard: three REST endpoints (/api/v1/lineage/{dataset|chart|dashboard}/<id>) plus a LineageView/LineageModal frontend surfaced from the dataset editor and the chart/dashboard menus. Rebased onto current master and updated for drift since the original branch: - repointed lineage imports from the removed '@apache-superset/core/ui' to '@apache-superset/core/translation' and '/theme' - re-applied the dataset lineage tab onto the migrated (TypeScript) DatasourceEditor Closes #37701 Co-authored-by: Jonathan Alberth Quispe Fuentes <qf.jonathan@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Wrap untranslated "Details" JSX text in t() (fixes pre-commit custom rules + babel-extract regression) - Filter dataset/chart/dashboard lineage endpoints by the current user's permissions so they never expose charts, dashboards, or datasource metadata the user cannot access (mirrors existing related_objects / get_datasets redaction patterns) - Register DashboardLineageResponseSchema in openapi_spec_component_schemas to resolve the dangling $ref in the generated API spec - Key Sankey graph nodes and the node-details map by a stable unique identity (type:id) so entities sharing a display name no longer collapse into a single node; keep the human-readable title as a separate label - Add a skip mechanism to useApiV1Resource / lineage hooks so empty-id callers (e.g. LineageModal) no longer fire requests against invalid endpoints like /api/v1/chart//lineage - Defer the dataset edit page lineage fetch until the Lineage tab is active - Make "View lineage" available in dashboard view mode, not only edit mode - Compare data["result"] in lineage integration tests to match the result-wrapped API contract Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace `Record<string, any>` on additionalInfo with an explicit value union, and type the Sankey node click handler params instead of `any`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pass '' instead of 0 as the fallback so isEmptyId skips the request rather than calling /api/v1/dataset/0/lineage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- mark redacted dashboard lineage dataset fields nullable in the schema - filter dataset lineage chart_ids by chart access - responsive Sankey width and fallback dataset->dashboard edges - empty state for unsaved datasets; i18n legend labels; typed test mocks Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Mark database_name/slug as allow_none in lineage response schemas to match the nullable values the endpoints actually return. - Defer LineageModal's lineage fetch until the modal opens (beforeOpen + the hooks' skip flag) so rendering it inside a dropdown no longer fires the endpoint on menu open. - selectinload Slice.dashboards / Dashboard.slices in get_related_objects to avoid N+1 queries when building dataset lineage. - Drop the unreachable dataset branch in getEntityUrl (no standalone dataset page) and add type hints to the new lineage locals. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirrors the dashboard lineage endpoint's gating: schema/table/database name are only exposed when the caller can access the underlying datasource, so the chart endpoint no longer leaks that info unconditionally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merges duplicate @superset-ui/core/components import, replaces direct window.location navigation with the redirect() helper and drops the hard-coded /superset/ dashboard path per navigationUtils invariants, and fixes new drill_info RBAC tests calling insert_dataset() with a nonexistent owners kwarg instead of editor_user_ids. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These four tests were unrelated to this PR's lineage feature, called insert_dashboard(roles=...) which no longer exists (roles was replaced by the Subject-based editors/viewers model upstream), and asserted DASHBOARD_RBAC-gated behavior that can_drill_dataset_via_dashboard_access never implements (it only checks EMBEDDED_SUPERSET guest access and ENABLE_VIEWERS promiscuous mode). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
32877a4 to
f75d48d
Compare
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.10.
| Benchmark suite | Current: f75d48d | Previous: 4a18556 | Ratio |
|---|---|---|---|
spa entrypoint (JS) |
9277950 bytes |
8332413 bytes |
1.11 |
This comment was automatically generated by workflow using github-action-benchmark.
SUMMARY
Adopts #37701 by @qf-jonathan — thank you for the substantial work here! 🙏
Adds interactive Sankey-based lineage that lets users explore data relationships:
Implementation: three REST endpoints (
/api/v1/lineage/{dataset|chart|dashboard}/<id>) with permissions + integration tests, and aLineageView/LineageModalfrontend (ECharts Sankey) surfaced from the dataset editor and the chart/dashboard action menus.What changed vs. the original PR
Rebased onto current master and updated for drift accumulated since the original branch (Feb):
constants.py,ChartCard,useHeaderActionsDropdownMenu, and the chart integration tests.@apache-superset/core/uito@apache-superset/core/translationand/theme(the package layout changed).DatasourceEditor(it was.jsxwhen the original PR was written).TESTING INSTRUCTIONS
Open a dataset → Lineage tab; open a chart/dashboard menu → Lineage; verify the Sankey renders upstream/downstream relationships.
Backend:
pytest tests/integration_tests/{charts,dashboards,datasets}/api_tests.py -k lineageVerification status (honest notes for reviewers)
oxlintand the backend integration tests weren't run in my environment (no DB); CI will exercise both.ADDITIONAL INFORMATION
Closes #37701
🤖 Generated with Claude Code