OM 1.4.3 dbt ingestion issue NoneType has no attribute __root__

[geoHeil]

For a postgres container

services:
  db:
    image: postgres:16.3-alpine3.20
    restart: always
    shm_size: 128mb
    environment:
      POSTGRES_PASSWORD: minidemo
      POSTGRES_USER: minidemo
    ports:
      - "5432:5432"
    command: ["postgres",
              "-c", "shared_preload_libraries=pg_stat_statements",
              "-c", "pg_stat_statements.track=top", # all, top
              "-c", "pg_stat_statements.max=10000",
    ]
    volumes:
      - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql

and dbt project like

sources.yaml:

version: 2

sources:
  - name: minidemo
    database: minidemo
    schema: default
    tables:
      - name: sample
        columns:
          - name: foo
            tests:
              - unique
              - not_null

schema.yaml:

version: 2

models:
  - name: my_first_dbt_model
    description: "A starter dbt model"
    columns:
      - name: id
        description: "The primary key for this table"
        tests:
          - unique
          - not_null

  - name: my_second_dbt_model
    description: '{{ doc("orders_status") }}'
    +tags: "foo_tag"

    meta:
      owner: "georg.heiler"
      openmetadata:
        tier: 'Tier.Tier2'
      my_custom_meta: mymeta
    columns:
      - name: id
        description: "A starter dbt model"
        tests:
          - unique
          - not_null

# step 1
  - name: sample_foo
    description: '{{ doc("orders_status") }}'

# step 2
    columns:
      - name: foo
        #description: foo the bar
        # step 3 change this again
        description: baz the bar

and OM configuraton:

pg basic ingest:

source:
  type: postgres
  serviceName: minidemo_local
  serviceConnection:
    config:
      type: Postgres
      username: minidemo
      authType:
        password: minidemo
      hostPort: localhost:5432
      database: minidemo

  sourceConfig:
    config:
      type: DatabaseMetadata
      markDeletedTables: true
      markDeletedStoredProcedures: true
      includeTables: true
      includeViews: true
      schemaFilterPattern:
        excludes:
        - information_schema
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: DEBUG
  openMetadataServerConfig:
    hostPort: "https://om.com/api"
    authProvider: openmetadata
    securityConfig:
      jwtToken: "$TOKEN"

and dbt configuration like

source:
  type: dbt
  serviceName: minidemo_local
  sourceConfig:
    config:
      type: DBT
      dbtConfigSource:
        dbtConfigType: local
        dbtCatalogFilePath: minidemo_dbt\target\catalog.json
        dbtManifestFilePath: minidemo_dbt\target\manifest.json
        dbtRunResultsFilePath: minidemo_dbt\target\run_results.json
      dbtUpdateDescriptions: true
      includeTags: true
      dbtClassificationName: dbtTags
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: DEBUG  # DEBUG, INFO, WARNING or ERROR
  openMetadataServerConfig:
    hostPort: "https://om.com/api"
    authProvider: openmetadata
    securityConfig:
      jwtToken: "$TOKEN"

when ingesting with

metadata ingest -c admh_connections\01_pg_catalog.yaml
metadata ingest -c admh_connections\02_dbt.yaml

the following error is observed

+--------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| From   | Entity Name                                                                    | Message                                                                                                    | Stack Trace
                                                                                                                                        |
+========+================================================================================+============================================================================================================+==============================================================================================================================================================================+
| dbt    |                                                                                | Failed to parse the query  to capture lineage: 'NoneType' object has no attribute '__root__'               | Traceback (most recent call last):
                                                                                                                                        |
|        |                                                                                |                                                                                                            |   File "minidemo\lib\site-packages\metadata\ingestion\source\database\dbt\metadata.py", line 675, in create_dbt_query_lineage |
|        |                                                                                |                                                                                                            |     self.config.serviceConnection.__root__.config.type.value                                                                                                                 |
|        |                                                                                |                                                                                                            | AttributeError: 'NoneType' object has no attribute '__root__'. Did you mean: '__bool__'?                                                                                     |
+--------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| dbt    | /*                                                                             | Failed to parse the query /*                                                                               | Traceback (most recent call last):
                                                                                                                                        |
|        |     Welcome to your first dbt model!                                           |     Welcome to your first dbt model!                                                                       |   File "minidemo\lib\site-packages\metadata\ingestion\source\database\dbt\metadata.py", line 675, in create_dbt_query_lineage |
|        |     Did you know that you can also configure models directly within SQL files? |     Did you know that you can also configure models directly within SQL files?                             |     self.config.serviceConnection.__root__.config.type.value                                                                                                                 |
|        |     This will override configurations stated in dbt_project.yml                |     This will override configurations stated in dbt_project.yml                                            | AttributeError: 'NoneType' object has no attribute '__root__'. Did you mean: '__bool__'?                                                                                     |
|        |                                                                                |                                                                                                            |
                                                                                                                                        |
|        |     Try changing "table" to "view" below                                       |     Try changing "table" to "view" below                                                                   |
                                                                                                                                        |
|        | */                                                                             | */                                                                                                         |
                                                                                                                                        |
|        |                                                                                |                                                                                                            |
                                                                                                                                        |
|        |                                                                                |                                                                                                            |
                                                                                                                                        |
|        |                                                                                |                                                                                                            |
                                                                                                                                        |
|        | with source_data as (                                                          | with source_data as (                                                                                      |
                                                                                                                                        |
|        |                                                                                |                                                                                                            |
                                                                                                                                        |
|        |     select 1 as id                                                             |     select 1 as id                                                                                         |
                                                                                                                                        |
|        |     union all                                                                  |     union all                                                                                              |
                                                                                                                                        |
|        |     select null as id                                                          |     select null as id                                                                                      |
                                                                                                                                        |
|        |                                                                                |                                                                                                            |
                                                                                                                                        |
|        | )                                                                              | )                                                                                                          |
                                                                                                                                        |
|        |                                                                                |                                                                                                            |
                                                                                                                                        |
|        | select *                                                                       | select *                                                                                                   |
                                                                                                                                        |
|        | from source_data                                                               | from source_data                                                                                           |
                                                                                                                                        |
|        |                                                                                |                                                                                                            |
                                                                                                                                        |
|        | /*                                                                             | /*                                                                                                         |
                                                                                                                                        |
|        |     Uncomment the line below to remove records with null `id` values           |     Uncomment the line below to remove records with null `id` values                                       |
                                                                                                                                        |
|        | */                                                                             | */                                                                                                         |
                                                                                                                                        |
|        |                                                                                |                                                                                                            |
                                                                                                                                        |
|        | -- where id is not null                                                        | -- where id is not null to capture lineage: 'NoneType' object has no attribute '__root__'                  |
                                                                                                                                        |
+--------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| dbt    | select * from "minidemo"."demo"."sample"                                       | Failed to parse the query                                                                                  | Traceback (most recent call last):
                                                                                                                                        |
|        |                                                                                | select * from "minidemo"."demo"."sample" to capture lineage: 'NoneType' object has no attribute '__root__' |   File "minidemo\lib\site-packages\metadata\ingestion\source\database\dbt\metadata.py", line 675, in create_dbt_query_lineage |
|        |                                                                                |                                                                                                            |     self.config.serviceConnection.__root__.config.type.value                                                                                                                 |
|        |                                                                                |                                                                                                            | AttributeError: 'NoneType' object has no attribute '__root__'. Did you mean: '__bool__'?                                                                                     |
+--------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| dbt    |                                                                                | Failed to parse the query  to capture lineage: 'NoneType' object has no attribute '__root__'               | Traceback (most recent call last):
                                                                                                                                        |
|        |                                                                                |                                                                                                            |   File "minidemo\lib\site-packages\metadata\ingestion\source\database\dbt\metadata.py", line 675, in create_dbt_query_lineage |
|        |                                                                                |                                                                                                            |     self.config.serviceConnection.__root__.config.type.value                                                                                                                 |
|        |                                                                                |                                                                                                            | AttributeError: 'NoneType' object has no attribute '__root__'. Did you mean: '__bool__'?                                                                                     |
+--------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[2024-06-18 12:54:20] INFO     {metadata.Utils:logger:184} - Success %: 81.82
[2024-06-18 12:54:20] INFO     {metadata.Utils:logger:184} - Workflow finished in time: 6.07s
[2024-06-18 12:54:20] INFO     {metadata.Utils:logger:184} - Workflow finished with failures
Traceback (most recent call last):
  File "minidemo\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "minidemo\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "minidemo\Scripts\metadata.exe\__main__.py", line 7, in <module>
    sys.exit(metadata())
  File "minidemo\lib\site-packages\metadata\cmd.py", line 162, in metadata
    RUN_PATH_METHODS[metadata_workflow](path)
  File "minidemo\lib\site-packages\metadata\cli\ingest.py", line 51, in run_ingest
    workflow.raise_from_status()
  File "minidemo\lib\site-packages\metadata\workflow\workflow_status_mixin.py", line 125, in raise_from_status
    raise err
  File "minidemo\lib\site-packages\metadata\workflow\workflow_status_mixin.py", line 122, in raise_from_status
    self.raise_from_status_internal(raise_warnings)
  File "minidemo\lib\site-packages\metadata\workflow\ingestion.py", line 150, in raise_from_status_internal
    raise WorkflowExecutionError(
metadata.config.common.WorkflowExecutionError: dbt reported errors: dbt Summary: [9 Records, [0 Updated Records, 0 Warnings, 4 Errors, 0 Filtered]

[geoHeil]

the logs show

[2024-06-18 12:54:16] WARNING  {metadata.Ingestion:status:76} - Failed to parse the query /*
    Welcome to your first dbt model!
    Did you know that you can also configure models directly within SQL files?
    This will override configurations stated in dbt_project.yml

    Try changing "table" to "view" below
*/



with source_data as (

    select 1 as id
    union all
    select null as id

)

select *
from source_data

/*
    Uncomment the line below to remove records with null `id` values
*/

-- where id is not null to capture lineage: 'NoneType' object has no attribute '__root__'
[2024-06-18 12:54:16] DEBUG    {metadata.Ingestion:status:77} - Traceback (most recent call last):
  File "minidemo\lib\site-packages\metadata\ingestion\source\database\dbt\metadata.py", line 675, in create_dbt_query_lineage
    self.config.serviceConnection.__root__.config.type.value
AttributeError: 'NoneType' object has no attribute '__root__'. Did you mean: '__bool__'? 

[geoHeil]

the real issue is hidden here:

[2024-06-18 12:54:12] DEBUG    {metadata.Ingestion:ingestion:200} - Traceback (most recent call last):
  File "minidemo\lib\site-packages\metadata\workflow\ingestion.py", line 187, in _retrieve_service_connection_if_needed
    self.config.source.serviceConnection = ServiceConnection(
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ServiceConnection
__root__
  none is not an allowed value (type=type_error.none.not_allowed)

[2024-06-18 12:54:12] ERROR    {metadata.Ingestion:ingestion:201} - Unknown error getting service connection for service name [minidemo_local] using the secrets manager provider [SecretsManagerProvider.db]: 1 validation error for ServiceConnection

[geoHeil]

answer see slack

[harshach]

cc @OnkarVO7

[geoHeil]

we had been in discussion already / resolved now

[harshach]

@OnkarVO7 lets post the answer here and close the discussion

[OnkarVO7]

While running the query lineage of the dbt ingestion we check for the dialect of the query to parse it using the service connection information stored in the db.
The above error got generated because the service connection was not stored as a result of passing storeServiceConnection: false in the metadata ingestion.
Once the dbt ingestion has the service connection details and gets the dialect information this error does not get generated

[geoHeil]

additionally for a service connection that initially was created with false and later set to true the change of these was NOT honoured by the OM server 1.4.3 only a deletion and re-creation worked (or explicitly passing in the connection arguments)