feat(ingestion): add Omni BI platform source (INCUBATING) (#16564)

Author: bearsandrhinos

metadata-ingestion/docs/sources/omni/README.md

@@ -0,0 +1,18 @@
+## Overview
+
+Omni is a cloud-native business intelligence platform. Learn more in the [official Omni documentation](https://docs.omni.co/).
+
+The DataHub integration for Omni covers BI entities such as dashboards, charts, semantic datasets, and related ownership context. Depending on module capabilities, it can also capture features such as lineage, usage, profiling, ownership, tags, and stateful deletion detection.
+
+## Concept Mapping
+
+| Omni Concept    | DataHub Concept                                               | Notes                                                                                 |
+| --------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
+| `Folder`        | [Container](../../metamodel/entities/container.md)            | SubType `"Folder"`                                                                    |
+| `Dashboard`     | [Dashboard](../../metamodel/entities/dashboard.md)            | Published document with `hasDashboard=true`                                           |
+| `Tile`          | [Chart](../../metamodel/entities/chart.md)                    | Each query presentation within a dashboard                                            |
+| `Topic`         | [Dataset](../../metamodel/entities/dataset.md)                | SubType `"Topic"` — the semantic join graph entry point                               |
+| `View`          | [Dataset](../../metamodel/entities/dataset.md)                | SubType `"View"` — semantic layer table with dimensions and measures as schema fields |
+| `Workbook`      | [Dataset](../../metamodel/entities/dataset.md)                | SubType `"Workbook"` — unpublished personal exploration document                      |
+| Warehouse table | [Dataset](../../metamodel/entities/dataset.md)                | Native platform entity (e.g. Snowflake, BigQuery); linked as upstream of Omni Views   |
+| Document owner  | [User (a.k.a CorpUser)](../../metamodel/entities/corpuser.md) | Propagated as `TECHNICAL_OWNER` to Dashboard and Chart entities                       |

metadata-ingestion/docs/sources/omni/omni_post.md

@@ -0,0 +1,59 @@
+### Capabilities
+
+Use the **Important Capabilities** table above as the source of truth for supported features and whether additional configuration is required.
+
+#### Physical table lineage
+
+Omni Views reference physical warehouse tables via `sql_table_name` in model YAML. The connector resolves each reference to a DataHub dataset URN using the `connection_to_platform` mapping. If `normalize_snowflake_names: true` (default), database, schema, and table name components are uppercased to match the casing used by the DataHub Snowflake connector.
+
+#### Column-level lineage
+
+When `include_column_lineage: true` (default), the connector emits `FineGrainedLineage` entries by parsing `sql` expressions in model YAML and matching field references to known view columns. This enables precise field-level impact analysis across the full chain:
+
+```
+physical_table.column → semantic_view.field → dashboard_tile.field
+```
+
+#### Schema metadata
+
+For each Omni Semantic View, the connector emits a `SchemaMetadata` aspect containing one `SchemaField` per dimension and measure defined in model YAML:
+
+- **Dimensions**: emitted with inferred native type (string, date, timestamp, number, boolean)
+- **Measures**: emitted with aggregation type and native type `NUMBER`
+- Field descriptions are extracted from the YAML `description` attribute when present
+
+#### Model and document filtering
+
+Use `model_pattern` and `document_pattern` to restrict ingestion to specific models or dashboards:
+
+```yaml
+model_pattern:
+  allow:
+    - "^prod-.*"
+  deny:
+    - ".*-dev$"
+
+document_pattern:
+  allow:
+    - ".*"
+```
+
+### Limitations
+
+- Access Filters, User Attributes, and Cache schedules are not yet ingested.
+- Column lineage is limited to fields that appear in model YAML `sql` expressions; complex or fully derived expressions may not fully resolve.
+- Large organizations with many models may approach Omni API rate limits; tune `max_requests_per_minute` accordingly.
+- True end-to-end integration tests require a live Omni environment; the test suite uses deterministic mock API responses.
+
+### Troubleshooting
+
+If ingestion fails, validate credentials, permissions, and connectivity first. Then review the ingestion report and logs for source-specific errors.
+
+Common issues:
+
+| Symptom                                          | Likely Cause                                          | Resolution                                                                    |
+| ------------------------------------------------ | ----------------------------------------------------- | ----------------------------------------------------------------------------- |
+| `403 Forbidden` on `/v1/connections`             | API key lacks connection read scope                   | Ingestion continues with config fallbacks; physical lineage may be incomplete |
+| Physical tables not linked to warehouse entities | `connection_to_platform` not configured               | Add connection mapping for each Omni connection ID                            |
+| Snowflake URN mismatch                           | Case mismatch between Omni and DataHub Snowflake URNs | Ensure `normalize_snowflake_names: true` (default)                            |
+| Column lineage empty                             | View YAML has no `sql` expressions                    | Expected for views using direct `sql_table_name` without field-level SQL      |

metadata-ingestion/docs/sources/omni/omni_pre.md

@@ -0,0 +1,38 @@
+### Overview
+
+The `omni` module ingests metadata from the [Omni](https://omni.co/) BI platform into DataHub. It is intended for production ingestion workflows and supports the following:
+
+- Folders (as Containers), Dashboards, and Chart tiles
+- Semantic layer: Models, Topics, and Views with schema fields (dimensions and measures)
+- Physical warehouse tables with upstream lineage stitched to existing DataHub entities
+- Column-level (fine-grained) lineage from semantic view fields back to warehouse columns
+- Ownership propagated from the Omni document API
+
+Lineage is emitted as a five-hop chain:
+
+```
+Folder → Dashboard → Chart (tile) → Topic → Semantic View → Physical Table
+```
+
+### Prerequisites
+
+Before running ingestion, ensure you have the following:
+
+1. **An Omni Organization API key** with read access to models, documents, and connections. Generate API keys in Omni Admin → API Keys.
+
+2. **Connection mapping configuration** if you want physical table lineage to stitch with existing warehouse entities in DataHub. You will need to map each Omni connection ID to the corresponding DataHub platform name, platform instance, and database name:
+
+```yaml
+connection_to_platform:
+  "conn_abc123": "snowflake"
+connection_to_platform_instance:
+  "conn_abc123": "my_snowflake_account"
+connection_to_database:
+  "conn_abc123": "ANALYTICS_PROD"
+```
+
+Connection IDs can be found by calling the Omni `/v1/connections` API or from the Omni Admin UI.
+
+:::note
+If the Omni API key does not have permission to list connections (`403 Forbidden`), the connector will fall back to the `connection_to_platform` config overrides and continue ingestion without failing.
+:::

metadata-ingestion/docs/sources/omni/omni_recipe.yml

@@ -0,0 +1,46 @@
+source:
+  type: omni
+  config:
+    # Coordinates
+    base_url: "https://your-org.omniapp.co/api"
+
+    # Credentials
+    api_key: "${OMNI_API_KEY}"
+
+    # Connection → warehouse stitching
+    # Map Omni connection IDs to DataHub platform names so that physical table
+    # URNs match what was ingested by your warehouse source connector.
+    connection_to_platform:
+      "conn_abc123": "snowflake"
+
+    # Optional: map connection IDs to platform instances
+    # connection_to_platform_instance:
+    #   "conn_abc123": "my_snowflake_account"
+
+    # Optional: override the database name inferred from the Omni connection
+    # connection_to_database:
+    #   "conn_abc123": "ANALYTICS_PROD"
+
+    # Optional: include workbook-only documents (not just published dashboards)
+    # include_workbook_only: false
+
+    # Optional: filter which models to ingest
+    # model_pattern:
+    #   allow:
+    #     - ".*"
+
+    # Optional: filter which documents (dashboards/workbooks) to ingest
+    # document_pattern:
+    #   allow:
+    #     - ".*"
+
+    # Optional: disable column-level lineage
+    # include_column_lineage: true
+
+    # Optional: stateful ingestion with stale entity removal
+    stateful_ingestion:
+      enabled: true
+      remove_stale_metadata: true
+
+sink:
+  # sink configs

metadata-ingestion/pyproject.toml

@@ -910,6 +910,11 @@ okta = [
     "okta~=1.7.0,<2.0.0",
 ]
 
+omni = [
+    "PyYAML>=5.4",
+    "requests<3.0.0",
+]
+
 oracle = [
     "acryl-datahub-classify==0.0.11",
     "acryl-great-expectations==0.15.50.1",
@@ -1486,6 +1491,7 @@ all = [
     "pyspark~=3.5.6,<4.0.0",
     "python-ldap>=2.4,<4.0.0",
     "python-liquid>=2.0.0,<3.0.0",
+    "PyYAML>=5.4",
     "rdflib==6.3.2",
     "redash-toolbelt<0.2.0",
     "redshift-connector>=2.1.5,<3.0.0",
@@ -1651,7 +1657,7 @@ dev = [
     "python-json-logger>=2.0.0,<5.0.0",
     "python-ldap>=2.4,<4.0.0",
     "python-liquid>=2.0.0,<3.0.0",
-    "PyYAML<7.0.0",
+    "PyYAML>=5.4,<7.0.0",
     "rdflib==6.3.2",
     "redash-toolbelt<0.2.0",
     "redshift-connector>=2.1.5,<3.0.0",
@@ -1842,7 +1848,7 @@ docs = [
     "python-json-logger>=2.0.0,<5.0.0",
     "python-ldap>=2.4,<4.0.0",
     "python-liquid>=2.0.0,<3.0.0",
-    "PyYAML<7.0.0",
+    "PyYAML>=5.4,<7.0.0",
     "rdflib==6.3.2",
     "redash-toolbelt<0.2.0",
     "redshift-connector>=2.1.5,<3.0.0",
@@ -2111,6 +2117,7 @@ cassandra = "datahub.ingestion.source.cassandra.cassandra:CassandraSource"
 neo4j = "datahub.ingestion.source.neo4j.neo4j_source:Neo4jSource"
 vertexai = "datahub.ingestion.source.vertexai.vertexai:VertexAISource"
 hex = "datahub.ingestion.source.hex.hex:HexSource"
+omni = "datahub.ingestion.source.omni.omni:OmniSource"
 
 [project.entry-points."datahub.ingestion.transformer.plugins"]
 pattern_cleanup_ownership = "datahub.ingestion.transformer.pattern_cleanup_ownership:PatternCleanUpOwnership"

metadata-ingestion/setup.py

@@ -633,6 +633,7 @@
     },
     "flink": {"requests<3.0.0", "tenacity>=8.0.1,<9.0.0"},
     "grafana": {"requests<3.0.0", *sqlglot_lib},
+    "omni": {"requests<3.0.0", "PyYAML>=5.4"},
     "glue": aws_common | cachetools_lib | sqlglot_lib,
     # hdbcli is supported officially by SAP, sqlalchemy-hana is built on top but not officially supported
     "hana": sql_common
@@ -958,6 +959,7 @@
             "neo4j",
             "vertexai",
             "mssql-odbc",
+            "omni",
         ]
         if plugin
         for dependency in plugins[plugin]
@@ -1118,6 +1120,7 @@
         "neo4j = datahub.ingestion.source.neo4j.neo4j_source:Neo4jSource",
         "vertexai = datahub.ingestion.source.vertexai.vertexai:VertexAISource",
         "hex = datahub.ingestion.source.hex.hex:HexSource",
+        "omni = datahub.ingestion.source.omni.omni:OmniSource",
     ],
     "datahub.ingestion.transformer.plugins": [
         "pattern_cleanup_ownership = datahub.ingestion.transformer.pattern_cleanup_ownership:PatternCleanUpOwnership",

metadata-ingestion/src/datahub/ingestion/autogenerated/connector_registry/datahub.json

@@ -2520,6 +2520,62 @@
       "platform_name": "Okta",
       "support_status": "CERTIFIED"
     },
+    "omni": {
+      "capabilities": [
+        {
+          "capability": "LINEAGE_FINE",
+          "description": "Field-level lineage when include_column_lineage=true",
+          "subtype_modifier": null,
+          "supported": true
+        },
+        {
+          "capability": "DESCRIPTIONS",
+          "description": "Enabled by default",
+          "subtype_modifier": null,
+          "supported": true
+        },
+        {
+          "capability": "DELETION_DETECTION",
+          "description": "Enabled by default via stateful ingestion",
+          "subtype_modifier": null,
+          "supported": true
+        },
+        {
+          "capability": "OWNERSHIP",
+          "description": "Document owner extracted from Omni API",
+          "subtype_modifier": null,
+          "supported": true
+        },
+        {
+          "capability": "PLATFORM_INSTANCE",
+          "description": "Supported via connection_to_platform_instance config",
+          "subtype_modifier": null,
+          "supported": true
+        },
+        {
+          "capability": "SCHEMA_METADATA",
+          "description": "Dimensions and measures extracted as schema columns",
+          "subtype_modifier": null,
+          "supported": true
+        },
+        {
+          "capability": "LINEAGE_COARSE",
+          "description": "Dashboard \u2192 Tile \u2192 Topic \u2192 View \u2192 DB Table",
+          "subtype_modifier": null,
+          "supported": true
+        },
+        {
+          "capability": "TEST_CONNECTION",
+          "description": "Enabled by default",
+          "subtype_modifier": null,
+          "supported": true
+        }
+      ],
+      "classname": "datahub.ingestion.source.omni.omni.OmniSource",
+      "platform_id": "omni",
+      "platform_name": "Omni",
+      "support_status": "INCUBATING"
+    },
     "openapi": {
       "capabilities": [
         {
@@ -4238,4 +4294,4 @@
       "support_status": "CERTIFIED"
     }
   }
-}
+}