Skip to content

Fix: PowerBI .pbit parser fails on multiline DAX expressions #25510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
AntoineGlacet wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix: PowerBI .pbit parser fails on multiline DAX expressions #25510

AntoineGlacet wants to merge 1 commit into from
+116 −6

Conversation

@AntoineGlacet
Copy link
Contributor

@AntoineGlacet AntoineGlacet commented Jan 25, 2026

Summary

Fixes #25483 - PowerBI .pbit file ingestion fails when DAX expressions span multiple lines.

Problem

The PowerBI ingestion connector expected expression fields to be strings, but .pbit files store multiline DAX as JSON arrays. This caused Pydantic validation errors during parsing:

validation error for PowerBiTable
measures.0.expression
  Input should be a valid string [type=string_type, input_value=['', 'VAR PL_Net_Selectio.../ 1000000', ')', '', ''], input_type=list]

Root Cause

Three Pydantic models had strict str typing for expression fields:

  • PowerBiMeasures.expression
  • PowerBITableSource.expression
  • DatasetExpression.expression

When .pbit files contained multiline DAX, the JSON parser returned lists instead of strings, causing validation failures.

Solution

  1. Updated type definitions to accept both str and List[str]:

    expression: Optional[Union[str, List[str]]] = None

  2. Added Pydantic validators to normalize lists to newline-joined strings:

    @field_validator("expression")
@classmethod
def normalize_expression(cls, v):
    return "\n".join(v) if isinstance(v, list) else v

  3. Added defensive null check in metadata.py to prevent errors when expression is None.

Changes

  • ingestion/src/metadata/ingestion/source/dashboard/powerbi/models.py (+31, -5)
    • Updated 3 model expression fields to accept Union[str, List[str]]
    • Added 3 field validators for automatic normalization
  • ingestion/src/metadata/ingestion/source/dashboard/powerbi/metadata.py (+6, -1)
    • Added null check before processing expressions
  • ingestion/tests/unit/topology/dashboard/test_powerbi_table_measures.py (+85)
    • Added integration test with real multiline DAX
    • Added 9 unit tests covering all edge cases

Testing

Unit Tests

pytest tests/unit/topology/dashboard/test_powerbi_table_measures.py -v
# ✅ 13/13 tests passed

Docker Integration Testing

Tested with production .pbit file (Monthly Financial_trusted.pbit):

  • 72 multiline DAX measures - ✅ All parsed successfully
  • 32 multiline source expressions - ✅ All normalized correctly
  • 0 Pydantic validation errors - ✅ No failures

Test Coverage

  • ✅ String expressions (backward compatibility)
  • ✅ List expressions (new behavior)
  • ✅ Empty lists (edge case)
  • ✅ None values (edge case)
  • ✅ All three affected models tested

Backward Compatibility

✅ 100% backward compatible - String expressions continue working exactly as before. The Union type and validators only activate when lists are encountered.

Checklist

  • Unit tests added and passing
  • Integration tested with real .pbit files
  • Follows Pydantic v2 best practices
  • No breaking changes
  • Defensive coding (null checks)
  • Consistent with existing code patterns

@AntoineGlacet
Fix PowerBI .pbit parser for multiline DAX expressions
8d370e8 
Fixes open-metadata#25483

Problem:
- PowerBI .pbit files store multiline DAX expressions as JSON arrays (one string per line)
- Pydantic validation failed with 'Input should be a valid string [type=string_type, input_value=[...], input_type=list]'
- Parser could not ingest .pbit files with multiline DAX measures, source expressions, or dataset expressions

Solution:
- Added Pydantic field_validator decorators to normalize list expressions to multiline strings
- Updated PowerBiMeasures.expression to accept Union[str, List[str]]
- Updated PowerBITableSource.expression to accept Union[str, List[str]]
- Updated DatasetExpression.expression to accept Union[str, List[str]]
- Made expression optional for PowerBiMeasures to handle measures without expressions
- Updated _get_child_measures() to handle None expression values

Testing:
- Successfully parsed .pbit file with 41 tables, 93 measures (72 multiline), 32 multiline sources
- Added 9 new unit tests for validators
- Added integration test case for multiline DAX expressions
- All existing tests pass (backward compatible)

Files changed:
- ingestion/src/metadata/ingestion/source/dashboard/powerbi/models.py
- ingestion/src/metadata/ingestion/source/dashboard/powerbi/metadata.py
- ingestion/tests/unit/test_powerbi_table_measures.py
@github-actions
Copy link
Contributor

github-actions bot commented Jan 25, 2026

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@AntoineGlacet AntoineGlacet mentioned this pull request Jan 25, 2026
Open
@gitar-bot
Copy link

gitar-bot bot commented Jan 25, 2026

Code Review 👍 Approved with suggestions 0 resolved / 2 findings

Well-structured fix for multiline DAX parsing with good test coverage. One behavioral change (required→optional field) should be verified as intentional.

⚠️ Bug: PowerBiMeasures.expression changed from required to optional

📄 ingestion/src/metadata/ingestion/source/dashboard/powerbi/models.py:139

The expression field in PowerBiMeasures was previously a required field (expression: str), but has been changed to optional (expression: Optional[Union[str, List[str]]] = None).

This is a potential breaking change:

  1. API contract change: Existing code that creates PowerBiMeasures objects may rely on expression being guaranteed non-None
  2. Data integrity: Measures without expressions may not be valid business objects

While the null check in metadata.py handles the consumer side gracefully, consider whether this was intentional. If a measure must always have an expression, the type should remain required:

expression: Union[str, List[str]]  # Required, accepts both types

If making it optional is intentional (e.g., some .pbit files have measures without expressions), then document this behavior change.

💡 Quality: Type annotation doesn't reflect post-validation state

📄 ingestion/src/metadata/ingestion/source/dashboard/powerbi/models.py:139 📄 ingestion/src/metadata/ingestion/source/dashboard/powerbi/models.py:155 📄 ingestion/src/metadata/ingestion/source/dashboard/powerbi/models.py:187

The type annotation Optional[Union[str, List[str]]] suggests the field can be either a string or a list after model creation. However, the @field_validator with mode="before" always converts lists to strings, so the actual runtime type is always Optional[str].

This mismatch could confuse consumers of the model who might check isinstance(measure.expression, list) based on the type hint.

Consider updating the type annotation to reflect the post-validation state:

expression: Optional[str] = None

@field_validator("expression", mode="before")
@classmethod
def normalize_expression(cls, v: Union[str, List[str], None]) -> Optional[str]:
    if isinstance(v, list):
        return "\n".join(v)
    return v

This makes the type annotation accurate while still allowing list inputs via the validator's parameter type.

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change:

Auto-apply Compact 
gitar auto-apply:on         

gitar display:verbose

Was this helpful? React with 👍 / 👎 | Gitar

gitar-bot[bot]
gitar-bot bot reviewed Jan 25, 2026
View reviewed changes
ingestion/src/metadata/ingestion/source/dashboard/powerbi/models.py
name: str
expression: str
expression: Optional[Union[str, List[str]]] = None
description: Optional[str] = None
Copy link

@gitar-bot gitar-bot bot Jan 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Bug: PowerBiMeasures.expression changed from required to optional

Details

The expression field in PowerBiMeasures was previously a required field (expression: str), but has been changed to optional (expression: Optional[Union[str, List[str]]] = None).

This is a potential breaking change:

  1. API contract change: Existing code that creates PowerBiMeasures objects may rely on expression being guaranteed non-None
  2. Data integrity: Measures without expressions may not be valid business objects

While the null check in metadata.py handles the consumer side gracefully, consider whether this was intentional. If a measure must always have an expression, the type should remain required:

expression: Union[str, List[str]]  # Required, accepts both types

If making it optional is intentional (e.g., some .pbit files have measures without expressions), then document this behavior change.

Was this helpful? React with 👍 / 👎

Copy link
Contributor Author

@AntoineGlacet AntoineGlacet Jan 25, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an intentional change as some measures can have no expressions and it frequently happen in real reports, used in business.

@AntoineGlacet AntoineGlacet marked this pull request as ready for review January 25, 2026 10:24
@AntoineGlacet AntoineGlacet requested a review from a team as a code owner January 25, 2026 10:24
@harshach harshach added the safe to test Add this label to run secure Github workflows on PRs label Jan 25, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 25, 2026

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 25, 2026

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion:trivy (debian 12.12)

Vulnerabilities (4)

Package Vulnerability ID Severity Installed Version Fixed Version
libpam-modules CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2
libpam-modules-bin CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2
libpam-runtime CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2
libpam0g CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
com.squareup.okhttp3:okhttp CVE-2021-0341 🚨 HIGH 3.12.12 4.9.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219
org.lz4:lz4-java CVE-2025-12183 🚨 HIGH 1.8.0 1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (18)

Package Vulnerability ID Severity Installed Version Fixed Version
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3
aiohttp CVE-2025-69223 🚨 HIGH 3.12.12 3.13.3
aiohttp CVE-2025-69223 🚨 HIGH 3.13.2 3.13.3
apache-airflow CVE-2025-68438 🚨 HIGH 3.1.5 3.1.6
apache-airflow CVE-2025-68675 🚨 HIGH 3.1.5 3.1.6
azure-core CVE-2026-21226 🚨 HIGH 1.37.0 1.38.0
jaraco.context CVE-2026-23949 🚨 HIGH 5.3.0 6.1.0
jaraco.context CVE-2026-23949 🚨 HIGH 5.3.0 6.1.0
jaraco.context CVE-2026-23949 🚨 HIGH 6.0.1 6.1.0
pyasn1 CVE-2026-23490 🚨 HIGH 0.6.1 0.6.2
ray CVE-2025-62593 🔥 CRITICAL 2.47.1 2.52.0
starlette CVE-2025-62727 🚨 HIGH 0.48.0 0.49.1
urllib3 CVE-2025-66418 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2025-66471 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2026-21441 🚨 HIGH 1.26.20 2.6.3
wheel CVE-2026-24049 🚨 HIGH 0.45.1 0.46.2
wheel CVE-2026-24049 🚨 HIGH 0.45.1 0.46.2
wheel CVE-2026-24049 🚨 HIGH 0.45.1 0.46.2

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO

No Vulnerabilities Found

@github-actions
Copy link
Contributor

github-actions bot commented Jan 25, 2026

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion-base-slim:trivy (debian 12.13)

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
com.squareup.okhttp3:okhttp CVE-2021-0341 🚨 HIGH 3.12.12 4.9.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219
org.lz4:lz4-java CVE-2025-12183 🚨 HIGH 1.8.0 1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (10)

Package Vulnerability ID Severity Installed Version Fixed Version
apache-airflow CVE-2025-68438 🚨 HIGH 3.1.5 3.1.6
apache-airflow CVE-2025-68675 🚨 HIGH 3.1.5 3.1.6
jaraco.context CVE-2026-23949 🚨 HIGH 5.3.0 6.1.0
jaraco.context CVE-2026-23949 🚨 HIGH 6.0.1 6.1.0
starlette CVE-2025-62727 🚨 HIGH 0.48.0 0.49.1
urllib3 CVE-2025-66418 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2025-66471 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2026-21441 🚨 HIGH 1.26.20 2.6.3
wheel CVE-2026-24049 🚨 HIGH 0.45.1 0.46.2
wheel CVE-2026-24049 🚨 HIGH 0.45.1 0.46.2

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/extended_sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/lineage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data_aut.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage_aut.yaml

No Vulnerabilities Found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gitar-bot gitar-bot[bot] gitar-bot[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

powerBI .pbit file parser fail on multiline DAX expressions

2 participants

@AntoineGlacet @harshach