-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Description
Affected module
Ingestion Framework
Describe the bug
Metadata ingestion CLI for Google BigQuery (GBQ) fails during the CreateTable validation step when a table has a Foreign Key constraint where the local column name differs from the referenced column name.
The error returned by the OM API is:
Failed to ingest CreateTableRequest [table_name] due to api request failure: Invalid column name found in table constraint.
This happens because the BigQuery ingestion query for foreign constraints is returning the referenced column name as the constrained column name. When the API tries to validate the table, it looks for a column that doesn't exist in the local table schema, triggering a validation error in EntityRepository.java.
To Reproduce
- In GBQ, create tableA with column Id (PK).
- Create tableB with column tableA_Id (FK).
- Define a foreign key in tableB where tableA_Id references tableA.Id.
- Run the ingestion with includeConstraints: true.
- The ingestion fails because it tries to create a constraint on tableB using the name Id instead of tableA_Id.
Expected behavior
The ingestion should map the local column name (the one residing in the table being ingested) to the constraint, not the name of the column it points to in the remote table.
Version:
- OS: Linux (Ubuntu)
- Python version: 3.10
- OpenMetadata version: 1.9.1
- OpenMetadata Ingestion package version: openmetadata-ingestion~=1.9.1
Additional context
Affected Query: queries.py (Line 78)
Validation Logic: EntityRepository.java (Line 5027)
Metadata
Metadata
Assignees
Labels
Type
Projects
Status