Feature: Add support for Data Profiling Scan

[syou6162]

resolves dbt-labs/dbt-adapters#543

Problem

Dataplex data profiling lets you identify common statistical characteristics of the columns in your BigQuery tables. This information helps you to understand and analyze your data more effectively.

• https://cloud.google.com/dataplex/docs/data-profiling-overview?hl=en

If you are managing tables with dbt, it is natural to want to configure Data Profile Scan in a yaml file. If data profiling could be set within dbt after the table is created, it would make it easier for dbt users to use the data profiling function.

Solution

I created this pull request to add support for Data Profiling Scan. If you write the following in dbt_project.yml and then run dbt run, the Data Profile Scan settings will be configured automatically.

models:
  +on_schema_change: "sync_all_columns"
  my_project:
    +persist_docs:
      relation: true
      columns: true
    sandbox:
      +schema: sandbox
      +materialized: table
      +data_profile_scan:
        location: us-central1
        sampling_percent: 10
        enabled: "{{ target.name == 'prod'}}"

You can also specify Data Profile Scan settings for individual model files, rather than dbt_project.yml.

version: 2
models:
  - name: my_table
    config:
      data_profile_scan:
        location: us-central1
        scan_id: my_profile_scan
        sampling_percent: 10
        row_filter: "TRUE"

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[syou6162]

Data Profile Scan is sometimes used for purposes other than dbt. It is important to have a way to tell whether the information in Data Profile Scan was created via dbt when updating/deleting it mechanically using cli or sdk. You can use scan_id, but I added the managed_by label because it is easier to handle when structured like labels.

[syou6162]

@mikealfare @colin-rogers-dbt Could you review this pull request? It seems that many users are waiting for this feature to be incorporated into dbt-bigquery.

I can also make a pull request to fix the documentation for BigQuery configurations, so please let me know if you need this 👍. If you need it, it would be helpful if you could let me know if you need it before this pull request is merged or if it would be sufficient after it is merged.

[github-actions]

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.