Rdf ingestion mvp #15741

stephengoldbaum · 2025-12-19T18:36:23Z

Summary

This PR introduces a new RDF ingestion source for DataHub, enabling ingestion of RDF/OWL ontologies (Turtle, RDF/XML, JSON-LD, N3, N-Triples) with a focus on business glossaries. The source extracts glossary terms, term hierarchies, and relationships from RDF files using standard vocabularies like SKOS, OWL, and RDFS.

What's New

Core Features

RDF Ingestion Source (type: rdf) - Native DataHub plugin for RDF/OWL ontologies
Multiple Format Support - Turtle, RDF/XML, JSON-LD, N3, N-Triples
Flexible Source Loading - Files, directories (with recursive option), URLs, and comma-separated file lists
Glossary Term Extraction - Converts skos:Concept and owl:Class to DataHub GlossaryTerms
Glossary Node Hierarchy - Auto-creates glossary nodes from IRI path hierarchies
Term Relationships - Extracts skos:broader and skos:narrower relationships as isRelatedTerms
Stateful Ingestion - Full support for stale entity removal via stateful_ingestion config
Platform Instance Support - Configurable platform instances via platform_instance config

Architecture

Modular Entity Processing - Clean separation with extractors, converters, and MCP builders
Dependency-Based Processing - Topological sort for correct entity processing order
Test Connection Support - Implements test_connection() for connection validation

Capabilities

The source supports the following DataHub capabilities:

Capability	Status	Notes
Glossary Terms	✅	Enabled by default
Glossary Nodes	✅	Auto-created from IRI path hierarchies
Term Relationships	✅	Supports `skos:broader` and `skos:narrower`
Detect Deleted Entities	✅	Requires `stateful_ingestion.enabled: true`
Platform Instance	✅	Supported via `platform_instance` config
Extract Descriptions	✅	Enabled by default (from `skos:definition` or `rdfs:comment`)
Data Domain	❌	Not applicable (domains used internally for hierarchy)
Dataset Profiling	❌	Not applicable
Extract Lineage	❌	Not in MVP
Extract Ownership	❌	Not supported
Extract Tags	❌	Not supported

Testing

Test Coverage

126 unit tests - Comprehensive coverage of core functionality, error handling, and edge cases
16 integration tests - End-to-end testing with golden file validation
Test scenarios include:
- Simple glossary ingestion
- Glossary with relationships
- Glossary with domains
- Multiple RDF formats (Turtle, RDF/XML, JSON-LD)
- Recursive directory ingestion
- Export filtering (export_only, skip_export)
- Stateful ingestion with stale entity removal
- Error handling (missing files, malformed RDF, invalid formats)
- Large file performance warnings
- Path traversal protection

Test Files

tests/unit/rdf/ - Unit tests for individual components
tests/integration/rdf/ - Integration tests with golden file validation
All tests passing ✅

Documentation

User Documentation

docs/sources/rdf/rdf.md - Comprehensive user guide (489 lines)
- Quickstart guide
- Configuration reference
- RDF format and source types
- Dialects and selective export
- Stateful ingestion guide
- Example RDF files
- IRI-to-URN mapping
- Glossary node hierarchy
- Supported vocabularies
- Limitations and troubleshooting

Recipe Examples

docs/sources/rdf/rdf_recipe.yml - Example recipes for basic and stateful ingestion

Integration Test Documentation

tests/integration/rdf/README.md - Detailed guide for running integration tests

Configuration Example

source:
type: rdf
config:
source: ./glossary.ttl
format: turtle
environment: PROD
stateful_ingestion:
enabled: true
remove_stale_metadata: true
export_only:
- glossary## Files Changed

Technical Notes

Security & Performance

URL Loading Security - Timeout limits, size limits, and redirect limits for safe URL loading
Path Traversal Protection - Configurable enforcement to prevent access outside intended directories
Memory Efficiency - Generator patterns for work unit generation and streaming for large URL downloads
Format Validation - Validates RDF formats before processing

Code Quality

Thread-Safe Registry - Entity registry uses double-checked locking pattern for thread safety
Component Validation - Validates registered components for entity type consistency
Type Safety - Complete type hints with proper forward references for MCP return types
Error Handling - Granular error reporting with structured logs and context
URN Generation - Standardized URN format using dot notation, proper encoding, and GUID fallback for non-ASCII characters

New Files

src/datahub/ingestion/source/rdf/ingestion/rdf_source.py - Main source implementation
src/datahub/ingestion/source/rdf/core/rdf_loader.py - RDF loading utilities with security
src/datahub/ingestion/source/rdf/core/urn_generator.py - URN generation with encoding
src/datahub/ingestion/source/rdf/entities/base.py - Base interfaces for entity processing
src/datahub/ingestion/source/rdf/entities/registry.py - Thread-safe entity registry
docs/sources/rdf/rdf.md - User documentation
docs/sources/rdf/rdf_recipe.yml - Recipe examples
tests/integration/rdf/test_rdf_source.py - Integration tests
tests/unit/rdf/ - Unit tests (multiple files)

Modified Files

setup.py - Added RDF source to entry points (line 862)

Breaking Changes

None - This is a new feature addition with no breaking changes to existing functionality.

Support Status

The RDF source is marked as INCUBATING (SupportStatus.INCUBATING), indicating it's ready for community adoption but may have minor version changes in future releases based on feedback.

Checklist

github-actions · 2025-12-19T18:36:36Z

Linear: ING-1308

codecov · 2025-12-19T18:38:58Z

Codecov Report

❌ Patch coverage is 55.55556% with 1004 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...rc/datahub/ingestion/source/rdf/core/rdf_loader.py	21.22%	360 Missing ⚠️
...tahub/ingestion/source/rdf/ingestion/rdf_source.py	62.99%	158 Missing ⚠️
...datahub/ingestion/source/rdf/core/urn_generator.py	22.38%	104 Missing ⚠️
...ion/source/rdf/entities/glossary_term/extractor.py	49.75%	102 Missing ⚠️
.../datahub/ingestion/source/rdf/entities/registry.py	59.66%	73 Missing ⚠️
...rc/datahub/ingestion/source/rdf/dialects/router.py	38.70%	38 Missing ⚠️
.../src/datahub/ingestion/source/rdf/dialects/fibo.py	61.29%	36 Missing ⚠️
...c/datahub/ingestion/source/rdf/dialects/generic.py	50.98%	25 Missing ⚠️
...c/datahub/ingestion/source/rdf/dialects/bcbs239.py	59.09%	18 Missing ⚠️
...n/source/rdf/entities/glossary_term/mcp_builder.py	85.43%	15 Missing ⚠️
... and 10 more

📢 Thoughts on this report? Let us know!

alwaysmeticulous · 2025-12-19T18:42:27Z

✅ Meticulous spotted 0 visual differences across 976 screens tested: view results.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Expected differences? Click here. Last updated for commit bfdc088. This comment will update as new commits are pushed.}

codecov · 2025-12-19T18:45:28Z

Bundle Report

Changes will increase total bundle size by 3.34kB (0.01%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	29.51MB	3.34kB (0.01%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
`assets/index-*.js`	3.34kB	19.34MB	0.02%

Files in assets/index-*.js:

./src/app/ingestV2/source/builder/sources.json → Total Size: 39.45kB
./src/app/ingestV2/source/builder/RecipeForm/rdf.ts → Total Size: 2.86kB
./src/app/ingestV2/source/builder/RecipeForm/constants.ts → Total Size: 12.0kB
./src/app/ingestV2/source/builder/constants.ts → Total Size: 7.2kB

- Add RDF ingestion source for glossary terms, domains, and relationships - Streamlined architecture: extractors return DataHub AST directly - Removed unnecessary abstraction layers (RDF AST, converters where not needed) - Support for SKOS, OWL, and other RDF vocabularies - Comprehensive test coverage with 128 passing tests - UI integration for RDF source configuration

- Remove build_relationship_mcps() method from GlossaryTermMCPBuilder - Update tests to use RelationshipMCPBuilder directly - Clean separation: glossary_term handles terms, relationship handles relationships - Refactor _generate_workunits_from_ast to reduce complexity

…ance - Enhance error handling in RDFSource to provide actionable messages for missing files, malformed RDF, and invalid formats. - Implement unit tests to verify error handling behavior and ensure graceful degradation. - Update glossary term URN generation to use dot notation for hierarchical paths. - Improve logging for large file processing and ensure consistent URN formats across glossary nodes and terms. - Refactor methods to yield MCPs for memory efficiency during processing.

…uidance - Added helper text for various RDF source fields including source, format, extensions, recursive processing, environment, and dialect. - These enhancements aim to provide clearer instructions and examples for users configuring RDF ingestion settings.

- Introduced new RDF platform entry in capability_summary.json with detailed capabilities including deletion detection, tags, ownership, lineage, data profiling, domains, descriptions, and platform instance support. - Each capability includes a description and support status to enhance clarity for users configuring RDF ingestion.

- Changed the warning filter to ignore specific SQLAlchemy warnings. - Added a new dependency for RDF support in the setup configuration.

…ters - Introduced a new documentation file for the RDF ingestion source. - Updated type hints across various classes to use `Optional` for context parameters, enhancing code clarity and type safety. - Adjusted method signatures in `EntityExtractor`, `EntityConverter`, `EntityMCPBuilder`, and related classes to reflect these changes.

- Included "rdf" in both base development and full test development requirements in setup.py to ensure proper support for RDF ingestion.

…_summary.json

This reverts commit 48a0118.

- Added unit tests for duplicate term definition handling, ensuring correct extraction behavior for same URIs and properties. - Implemented comprehensive validation tests for RDF source configuration, covering required fields, type checks, and value constraints. - Introduced connection testing unit tests to verify functionality and error handling for various scenarios, including file existence and RDF format validation. - Developed edge case tests to handle scenarios like empty files, circular relationships, and special characters in paths. - Enhanced error handling tests to ensure actionable feedback for file not found, invalid format, and unsupported extensions.

- Updated test_duplicate_handling.py to use URIRef and Literal for RDF terms, enhancing readability and maintainability. - Modified test_rdf_config.py to specify type for config_dict, improving type safety. - Adjusted test_rdf_source_errors.py to refine error assertion logic for better clarity in failure messages.

…enhance glossary term extraction - Added `extract_custom_properties` method to `FIBODialect` for extracting FIBO-specific properties from URIs. - Updated `GlossaryTermExtractor` to utilize dialect-specific logic for identifying glossary terms and extracting custom properties. - Refactored glossary term extraction to delegate type checking and property extraction to the dialect, improving modularity and maintainability. - Enhanced RDF source configuration to ensure dialect instance is always provided for consistent behavior across dialects.

…iguration - Introduced `include_provisional` option in `RDFSourceConfig` to control the inclusion of provisional/work-in-progress terms. - Updated `FIBODialect` to filter out provisional terms based on the new configuration. - Enhanced `DialectRouter` to pass the `include_provisional` setting to dialect instances. - Modified `RDFSource` to ensure the `include_provisional` setting is correctly utilized during dialect initialization.

…ndividual support - Introduced integration tests for FIBO dialect to validate inclusion and exclusion of provisional terms based on configuration. - Added unit tests for FIBO dialect to ensure proper filtering of glossary terms by maturity level. - Implemented unit tests for NamedIndividual extraction, verifying support across different dialects and ensuring correct handling of SKOS properties. - Enhanced RDF source configuration tests to validate the `include_provisional` setting and its default behavior.

…visional filtering - Introduced a new RDF file `example_fibo_maturity.ttl` containing various maturity levels for terms, including released, provisional, and no maturity. - Created a corresponding YAML configuration file `test_example_recipe.yml` to test the filtering of provisional terms based on the `include_provisional` setting.

…e warning filters - Modified the `_get_dialect` method in `GlossaryTermExtractor` to provide a default `GenericDialect` when no dialect is specified, enhancing flexibility for testing and usage. - Updated warning filters in `setup.cfg` to ignore specific SQLAlchemy warnings, improving log clarity. - Adjusted unit tests to reflect changes in dialect handling, ensuring accurate behavior when no dialect is provided.

….Class and RDFS.Class - Updated the logic in `GenericDialect` to exclude ontology construct types while allowing OWL.Class and RDFS.Class to coexist with SKOS.Concept, enhancing compatibility with RDF standards.

- Updated the rdflib dependency in setup.py to specify a version range of >=6.0.0,<7.0.0, ensuring compatibility with existing RDF handling features.

- Updated the rdflib dependency in setup.py to specify an exact version of 6.3.2, ensuring compatibility with existing RDF handling features and preventing potential issues with future releases.

…er, and URN generator - Introduced new unit tests for various edge cases in RDF dialects (Generic, FIBO, Default), including handling of empty graphs, missing labels, and special characters. - Added tests for the RDF loader to cover format validation, file handling, URL loading, and zip file scenarios. - Implemented edge case tests for the URN generator, focusing on IRI parsing and platform normalization. - Enhanced overall test coverage to ensure robustness and reliability of RDF processing components.

… requests_file - Modified the RDF plugin dependencies in setup.py to add specific versions of requests (2.32.5) and requests_file (3.0.1) alongside rdflib (6.3.2), ensuring compatibility and stability for RDF processing.

…n in documentation generation - Added checks and warnings for missing platforms and plugins when processing README and documentation files, ensuring robustness in the documentation generation process. - Improved logging to provide clearer feedback when encountering issues with platform or plugin names during the generation of custom documentation.

github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX labels Dec 19, 2025

github-actions bot added the community-contribution PR or Issue raised by member(s) of DataHub Community label Dec 19, 2025

github-actions bot deployed to datahub-wheels (Preview) December 19, 2025 18:38 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) December 19, 2025 18:41 View deployment

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Dec 19, 2025

vercel bot deployed to Preview December 19, 2025 18:48 View deployment

stephengoldbaum force-pushed the rdf-ingestion-mvp branch from 83de506 to 4ba2ba2 Compare January 6, 2026 17:38

github-actions bot deployed to datahub-wheels (Preview) January 6, 2026 17:40 View deployment

vercel bot had a problem deploying to Preview January 6, 2026 17:46 Failure

stephengoldbaum added 16 commits January 17, 2026 16:13

Bringing RDF ingestion to DataHub standards compliance

25ade49

fix(ingestion): update warning filters and add RDF dependency

5c48c53

- Changed the warning filter to ignore specific SQLAlchemy warnings. - Added a new dependency for RDF support in the setup configuration.

chore(setup): add RDF dependency to base and test requirements

b465791

- Included "rdf" in both base development and full test development requirements in setup.py to ensure proper support for RDF ingestion.

Update capability_summary.json with RDF plugin

508f90a

Update RDF documentation link and ensure newline at end of capability…

a93f9ae

…_summary.json

Fix capability_summary.py to ensure consistent plugin ordering

380bcc9

Revert "Fix capability_summary.py to ensure consistent plugin ordering"

8437f3b

This reverts commit 48a0118.

Fix broken markdown links in RDF documentation

2f88e1f

stephengoldbaum added 5 commits January 17, 2026 16:26

Regenerate capability_summary.json to include RDF source

a6e752d

stephengoldbaum force-pushed the rdf-ingestion-mvp branch from 4ba2ba2 to 0238c98 Compare January 18, 2026 01:20

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 01:22 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 01:25 View deployment

vercel bot had a problem deploying to Preview January 18, 2026 01:27 Failure

stephengoldbaum added 2 commits January 17, 2026 22:26

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 03:35 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 03:38 View deployment

fix(rdf): restrict rdflib version to ensure compatibility

56c784c

- Updated the rdflib dependency in setup.py to specify a version range of >=6.0.0,<7.0.0, ensuring compatibility with existing RDF handling features.

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 03:41 View deployment

fix(rdf): pin rdflib version to 6.3.2 for compatibility

83c1ed3

- Updated the rdflib dependency in setup.py to specify an exact version of 6.3.2, ensuring compatibility with existing RDF handling features and preventing potential issues with future releases.

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 03:45 View deployment

vercel bot had a problem deploying to Preview January 18, 2026 03:46 Failure

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 04:00 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 04:04 View deployment

vercel bot had a problem deploying to Preview January 18, 2026 04:05 Failure

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 04:09 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 04:13 View deployment

vercel bot had a problem deploying to Preview January 18, 2026 04:14 Failure

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 04:34 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 04:38 View deployment

vercel bot had a problem deploying to Preview January 18, 2026 04:39 Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rdf ingestion mvp #15741

Rdf ingestion mvp #15741

Uh oh!

stephengoldbaum commented Dec 19, 2025

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

codecov bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

alwaysmeticulous bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

codecov bot commented Dec 19, 2025 •

edited

Loading

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Rdf ingestion mvp #15741

Are you sure you want to change the base?

Rdf ingestion mvp #15741

Uh oh!

Conversation

stephengoldbaum commented Dec 19, 2025

Summary

What's New

Core Features

Architecture

Capabilities

Testing

Test Coverage

Test Files

Documentation

User Documentation

Recipe Examples

Integration Test Documentation

Configuration Example

Technical Notes

Security & Performance

Code Quality

New Files

Modified Files

Breaking Changes

Support Status

Checklist

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

codecov bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alwaysmeticulous bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle Report

Affected Assets, Files, and Routes:

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Dec 19, 2025 •

edited

Loading

alwaysmeticulous bot commented Dec 19, 2025 •

edited

Loading

codecov bot commented Dec 19, 2025 •

edited

Loading