Skip to content

Benefits: Add row-level security to benefits mart table#4943

Open
charlie-costanzo wants to merge 1 commit intomainfrom
benefits-create-row-access-policy
Open

Benefits: Add row-level security to benefits mart table#4943
charlie-costanzo wants to merge 1 commit intomainfrom
benefits-create-row-access-policy

Conversation

@charlie-costanzo
Copy link
Member

Description

Agencies accessing benefits data through Metabase should only be able to see their own data. This PR adds a benefits_row_access_policy macro (following the same pattern used for payments data) and applies it as a post-hook on fct_benefits_events.

Each agency's service account is restricted to rows where event_properties_transit_agency matches their agency name. Internal team members retain full access as long as they are added to the DOT_DDS_Data_Pipeline_and_Warehouse_Users GCP group. Service accounts for CI, Composer, and Metabase team access are also included in the unrestricted policy.

Changes
Added benefits_row_access_policy() macro to create_row_access_policy.sql with per-agency filters for MST, SacRT, SBMTD, Nevada County Connects, VCTC, El Dorado Transit, and SLO RTA
Applied the macro as a post_hook on fct_benefits_events

Resolves #4789

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

tests will happen post-merge

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

  • No action required
  • Actions required (specified below)
    Create service account for RABA and add to row access policies

Testing

  • Confirm fct_benefits_events builds successfully in dbt
  • Verify an agency service account can only query its own rows
  • Verify a DOT_DDS_Data_Pipeline_and_Warehouse_Users member can query all rows

@charlie-costanzo charlie-costanzo force-pushed the benefits-create-row-access-policy branch from e8163e7 to 02eba3b Compare March 17, 2026 16:30
@github-actions
Copy link

github-actions bot commented Mar 17, 2026

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 5 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "4IG6kQ==" -> (known after apply)
!~      detect_md5hash      = "wbkSni83tARcGK5kRNNksw==" -> "different hash"
!~      generation          = 1773763143594609 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "wbkSni83tARcGK5kRNNksw==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["macros/create_row_access_policy.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "1E7mXQ==" -> (known after apply)
!~      detect_md5hash      = "aQlE89hGJ3yNIunbAYKhOg==" -> "different hash"
!~      generation          = 1768507755630034 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/macros/create_row_access_policy.sql"
!~      md5hash             = "aQlE89hGJ3yNIunbAYKhOg==" -> (known after apply)
        name                = "data/warehouse/macros/create_row_access_policy.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/mart/benefits/fct_benefits_events.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "qf0CBA==" -> (known after apply)
!~      detect_md5hash      = "yv1Ds8wqYdajqIdhXyAMQA==" -> "different hash"
!~      generation          = 1764872245825491 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/mart/benefits/fct_benefits_events.sql"
!~      md5hash             = "yv1Ds8wqYdajqIdhXyAMQA==" -> (known after apply)
        name                = "data/warehouse/models/mart/benefits/fct_benefits_events.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/mart/gtfs_schedule_latest/dim_stops_latest.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "RX9QrA==" -> (known after apply)
!~      detect_md5hash      = "N3LXHDngyMQ/GoIPU/W9ow==" -> "different hash"
!~      generation          = 1751416667776220 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql"
!~      md5hash             = "N3LXHDngyMQ/GoIPU/W9ow==" -> (known after apply)
        name                = "data/warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "pVsw4g==" -> (known after apply)
!~      detect_md5hash      = "A09mrJtuFMpL0Nx4Vn3XKA==" -> "different hash"
!~      generation          = 1773763145090036 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "A09mrJtuFMpL0Nx4Vn3XKA==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 5 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1656

@github-actions
Copy link

github-actions bot commented Mar 17, 2026

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 5 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "/PRoeg==" -> (known after apply)
!~      detect_md5hash      = "OaKB1cp3DCgNoIxK9XQX4g==" -> "different hash"
!~      generation          = 1773763149532212 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "OaKB1cp3DCgNoIxK9XQX4g==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["macros/create_row_access_policy.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "1E7mXQ==" -> (known after apply)
!~      detect_md5hash      = "aQlE89hGJ3yNIunbAYKhOg==" -> "different hash"
!~      generation          = 1768514562367243 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/macros/create_row_access_policy.sql"
!~      md5hash             = "aQlE89hGJ3yNIunbAYKhOg==" -> (known after apply)
        name                = "data/warehouse/macros/create_row_access_policy.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/mart/benefits/fct_benefits_events.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "qf0CBA==" -> (known after apply)
!~      detect_md5hash      = "yv1Ds8wqYdajqIdhXyAMQA==" -> "different hash"
!~      generation          = 1764958875947203 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/mart/benefits/fct_benefits_events.sql"
!~      md5hash             = "yv1Ds8wqYdajqIdhXyAMQA==" -> (known after apply)
        name                = "data/warehouse/models/mart/benefits/fct_benefits_events.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/mart/gtfs_schedule_latest/dim_stops_latest.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "RX9QrA==" -> (known after apply)
!~      detect_md5hash      = "N3LXHDngyMQ/GoIPU/W9ow==" -> "different hash"
!~      generation          = 1749663114136574 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql"
!~      md5hash             = "N3LXHDngyMQ/GoIPU/W9ow==" -> (known after apply)
        name                = "data/warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "gj3vWw==" -> (known after apply)
!~      detect_md5hash      = "IlOXGL2cEy0Jd/PMXGUGhg==" -> "different hash"
!~      generation          = 1773763150875515 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "IlOXGL2cEy0Jd/PMXGUGhg==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 5 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1656

@github-actions
Copy link

Warehouse report 📦

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data-pipeline-ingestion-and-modeling Ingesting, parsing and modeling data. Evan Siroky is product owner.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extend payments row-based security patterns to mart benefits tables

1 participant