Feature/refactor grants (#19)

* Enhance logging in grant_object macro to include detailed information about grants and roles being processed

* Refactor grant_object macro to improve logging and ensure unique role privileges are processed correctly

* Refactor grant management macros for consistency and improved logging

- Updated grant_share_read and grant_share_read_specific_schema macros to implement early exits and structured logging.
- Refactored ownership macros to include early exits and enhanced logging for ownership changes.
- Introduced shared helper macros for grant management to reduce redundancy and improve maintainability.
- Updated documentation in grants.yml and other macro YAML files for clarity and consistency.
- Added a new grants_smoke_test macro for lightweight validation of grant macros in dry-run mode.
- Standardized argument signatures and descriptions across macros for better usability.

* Add grant_schema_procedure_usage macros for managing procedure usage privileges

* Fix grant_procedure_usage macro to correctly access procedure names from query results

* Update README.md

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* Update macros/grants/_helpers.sql

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

Author: jonhopper-dataengineers

CHANGELOG.md

@@ -1,6 +1,10 @@
 # Data Engineers Snowflake DataOps Utils Project Changelog
 This file contains the changelog for the Data Engineers Snowflake DataOps Utils project, detailing updates, fixes, and enhancements made to the project over time.
 
+## v0.3.9 2024-06-10 - Grant Object For Procedures
+
+* added macro `grant_procedure_usage` to enable the ability to grant usage of a stored procedure to a role
+
 ## v0.3.8.5 2025-08-25 - Grant Usage to Application
 
 * added grant usage to application for `sp_sync_` in the `grant_privileges` macro

README.md

@@ -1,53 +1,54 @@
 This [dbt](https://github.com/dbt-labs/dbt) package contains macros that can be (re)used across dbt projects.
 
-> require-dbt-version: [">=1.6.0", "<2.0.0"]
+> require-dbt-version: [">=1.8.0", "<2.0.0"]
 ----
 
 ## Installation Instructions
 Add the following to your packages.yml file
 ```
   - git: https://github.com/DataEngineersNZ/dbt-snowflake-datops-utils.git
-    revision: "0.3.8"
+    revision: "0.3.8.2"
 ```
 ----
 
 ## Contents
 
-**Checks**
+Below is a catalogue of publicly supported macros grouped by domain. Internal helpers (those with docs.show: false or purely supportive behavior) are intentionally excluded. Where helpful, a short description is inlined; consult the YAML files for full argument metadata.
 
-- `get_populated_array`
-- `get_populated_array_value_as_string`
-- `get_populated_array_value_as_number`
-- `get_populated_numeric_value`
-- `get_populated_string_value`
+**checks**
+
+- `get_populated_array` – first non-empty array from two candidates
+- `get_populated_array_value_as_string` – join first non-empty array
+- `get_populated_array_value_or_string_as_string` – array joined or fallback string
+- `get_populated_numeric_value` – first numeric else 0
+- `get_populated_string_value` – first string else ''
 
 **clean**
 
-- `clean_functions`
-- `clean_generic`
-- `clean_models`
-- `clean_objects`
-- `clean_schemas`
-- `clean_stale_models`
-- `drop_object`
+- `clean_functions` – drop orphaned UDFs
+- `clean_generic` – drop orphaned streams/tasks/stages
+- `clean_models` – drop orphaned tables/views/external tables
+- `clean_objects` – orchestrate all clean macros
+- `clean_schemas` – drop schemas not in project
+- `clean_stale_models` – drop models older than N days
 
 **database**
 
-- `database_clone`
-- `database_desctroy`
-- `schema_clone`
+- `database_clone` – zero-copy clone a database
+- `database_destroy` – drop database
+- `schema_clone` – zero-copy clone a schema
 
-**dependancies**
+**dependencies** (non-lineage referencing)
 
-- `depends_on_ref`
-- `depends_on_source`
+- `depends_on_ref` – commented reference to model
+- `depends_on_source` – commented reference to source
 
 **dynamic_tables**
 
-- `target_lag_environment`
-- `target_warehouse_environment`
+- `target_lag_environment` – lag by environment
+- `target_warehouse_environment` – warehouse by environment
 
-**grants**
+**grants** (see refactored patterns section below)
 
 - `grant_database_ownership`
 - `grant_integration_ownership`
@@ -56,15 +57,23 @@ Add the following to your packages.yml file
 - `grant_object`
 - `grant_privileges`
 - `grant_schema_monitor`
+- `grant_schema_monitor_specific`
 - `grant_schema_operate`
-- `grant_schema_onwership`
+- `grant_schema_operate_specific`
+- `grant_schema_ownership`
+- `grant_schema_procedure_usage`
+- `grant_schema_procedure_usage_specific`
 - `grant_schema_read`
+- `grant_schema_read_specific`
 - `grant_share_read`
+- `grant_share_read_specific_schema`
 - `grant_usage_to_application`
+- `grants_smoke_test` – CI/dry-run validation harness
 
 **merge**
 
 - `get_merge_statement`
+- `get_default_merge_statement`
 
 **modelling**
 
@@ -96,23 +105,80 @@ Add the following to your packages.yml file
 
 **schema**
 
-- `generate_schema_name`
+- `generate_schema_name` (override)
 - `model_ref`
 - `model_source`
-- `ref`
-- `source`
+- `ref` (enhanced include_database)
+- `source` (enhanced include_database)
 
 **tags**
 
 - `apply_meta_as_tags`
-- `model_columns_contains_tag_meta`
-- `set_column_tag_value`
 
 **tasks**
 
 - `enable_dependent_tasks`
 - `execute_task`
 
+### Grants Management (Refactored Patterns)
+
+Recent refactors introduced a consistent pattern across grant-related macros for clarity, auditability, and safety:
+
+Key characteristics:
+- Early exit guards: macros skip execution outside `run` / `run-operation` contexts.
+- Logging only for top-level macros: operational macros write human-readable summaries instead of returning data structures.
+- Statement batching with consistent formatting and explicit counts (revokes vs grants).
+- Ownership helper macros still return statement lists internally (consumed by `grant_schema_ownership`).
+- Optional dry-run mode to preview changes.
+
+Dry-run mode:
+Set a project or CLI var `grants_dry_run: true` to log all statements without executing them for the following macros:
+`grant_schema_monitor`, `grant_schema_operate`, `grant_share_read`, `grant_share_read_specific_schema`, `grant_privileges`.
+
+Example CLI usage:
+```
+dbt run-operation grant_schema_operate --args '{"exclude_schemas": [], "grant_roles": ["OPS_SUPPORT"]}' --vars '{"grants_dry_run": true}'
+```
+
+Example project-level configuration (`dbt_project.yml`):
+```yaml
+vars:
+  grants_dry_run: true  # disable to allow execution
+```
+
+Sample log output pattern:
+```
+grant_schema_operate: processing 5 schemas for roles: OPS_SUPPORT
+revoke operate on TASK in schema MY_DB.MY_SCHEMA.MY_TASK from role OLD_ROLE;
+grant operate on all tasks in schema MY_DB.MY_SCHEMA to role ops_support;
+grant_schema_operate_specific summary: 1 revokes, 2 grants (dry_run=True)
+```
+
+Recommended workflow:
+1. Run with `grants_dry_run: true` and review logs in CI.
+2. Approve changes, re-run with dry-run disabled to apply.
+
+High-level macro intent summary:
+- `grant_schema_read*`: Ensures read usage, SELECT/REFERENCE privileges, optional future grants.
+- `grant_schema_monitor*`: Grants MONITOR on tasks/pipes + schema usage.
+- `grant_schema_operate*`: Grants OPERATE on tasks/pipes + schema usage.
+- `grant_schema_procedure_usage*`: Grants USAGE on all procedures + schema usage, with future grants.
+- `grant_share_read*`: Manages secure view exposure to outbound shares (revokes unmanaged, grants managed).
+- `grant_object`: Reconciles privilege sets on specific objects (TABLE/VIEW/PROCEDURE/FUNCTION/etc).
+- `grant_privileges`: Environment-aware bundle orchestrator.
+
+Notes:
+- Privilege diffing avoids redundant grants.
+- Revokes are only issued for privileges outside desired scope (or for unmanaged grantees when revocation is enabled).
+- Ownership grants always use `revoke current grants` to move ownership cleanly.
+
+Future enhancement ideas (not yet implemented):
+- Generic unified privilege macro parameterized by privilege type.
+- Aggregated dry-run report macro producing a JSON artifact.
+- Caching of SHOW results across macros within a single run-operation invocation.
+
+Contributions welcome. Keep macro signatures stable to avoid breaking downstream usage.
+
 ---
 
 ### Tagging macros

macros/checks/checks.yml

@@ -2,7 +2,7 @@ version: 2
 
 macros:
   - name: get_populated_array
-    description: This macro returns an array of values based on two input string arrays
+    description: Return the first non-empty array among two candidate string arrays.
     docs:
       show: true
     arguments:
@@ -14,7 +14,7 @@ macros:
         description: Column to fall back on if col_to_check is not populated
 
   - name: get_populated_array_value_as_string
-    description: This macro returns a string of values based on two input string arrays
+    description: Join the first non-empty array (of two candidates) into a delimited string.
     docs:
       show: true
     arguments:
@@ -26,7 +26,7 @@ macros:
         description: Column to fall back on if col_to_check is not populated
 
   - name: get_populated_array_value_or_string_as_string
-    description: This macro returns a string of values based on an input string array and a fall back string option
+    description: Return array contents as string if present else fall back to provided string value.
     docs:
       show: true
     arguments:
@@ -38,7 +38,7 @@ macros:
         description: Text based Column to fall back on if col_to_check is not populated
 
   - name: get_populated_numeric_value
-    description: This macro returns a numeric value based on two input numeric inputs. If both are empty, it returns 0
+    description: Return first non-null numeric value from two candidates else 0.
     docs:
       show: true
     arguments:
@@ -50,7 +50,7 @@ macros:
         description: Column to fall back on if col_to_check is not populated
 
   - name: get_populated_string_value
-    description: This macro returns a numeric value based on two input numeric inputs. If both are empty, it returns 0
+    description: Return first non-empty string value from two candidates else empty string.
     docs:
       show: true
     arguments:

macros/clean/clean.yml

@@ -2,7 +2,7 @@ version: 2
 
 macros:
   - name: clean_functions
-    description: This macro compares dbt user defined functions with those deployed and removes from snowflake if they no longer exist in dbt.
+    description: Reconcile dbt-defined user defined functions vs deployed; drop any orphaned in Snowflake.
     docs:
       show: true
     arguments:
@@ -14,7 +14,7 @@ macros:
         description: specifies if the macro should run in dry run mode
 
   - name: clean_generic
-    description: This macro compares dbt streams or tasks with those deployed and removes from snowflake if they no longer exist in dbt.
+    description: Reconcile streams / tasks / stages vs dbt definitions; drop those no longer present.
     docs:
       show: true
     arguments:
@@ -29,7 +29,7 @@ macros:
         description: specifies if the macro should run in dry run mode
 
   - name: clean_models
-    description: This macro compares dbt tables, external tables and views with those deployed and removes from snowflake if they no longer exist in dbt.
+    description: Reconcile dbt tables, external tables & views with deployed objects; remove orphans.
     docs:
       show: true
     arguments:
@@ -41,7 +41,7 @@ macros:
         description: specifies if the macro should run in dry run mode
 
   - name: clean_objects
-    description: This macro runs all the other clean macros and should be used on a post-hook event
+    description: Convenience orchestrator to run all clean_* macros (suitable for post-hook usage).
     docs:
       show: true
     arguments:
@@ -57,7 +57,7 @@ macros:
 
 
   - name: clean_schemas
-    description: This macro compares dbt schemas with those deployed and removes from snowflake if they no longer exist in dbt.
+    description: Reconcile schema list vs dbt; drop schemas no longer defined.
     docs:
       show: true
     arguments:
@@ -108,7 +108,7 @@ macros:
         description: arguments of the sql object
 
   - name: clean_stale_models
-    description: This macro removes stale models from snowflake
+    description: Drop models in a schema older than N days (stale artifact cleanup).
     docs:
       show: true
     arguments:

macros/database/database.yml

@@ -2,6 +2,7 @@ version: 2
 
 macros:
   - name: database_clone
+    description: Clone (zero-copy) a source database into a new destination database optionally transferring ownership.
     arguments:
       - name: source_database
         description: Source database name, i.e. database to be cloned
@@ -11,6 +12,7 @@ macros:
         description: "[Optional] The new owner role of the newly created object"
 
   - name: schema_clone
+    description: Clone (zero-copy) a source schema into a destination schema; database defaults to target if not supplied.
     arguments:
       - name: source_schema
         description: Source schema name, i.e. schema to be cloned
@@ -24,6 +26,7 @@ macros:
         description: "[Optional] The new owner role of the newly created object"
 
   - name: database_destroy
+    description: Drop the supplied database (destructive operation).
     arguments:
       - name: database_name
         description: Database to drop

macros/dependancies/dependancies.yml

@@ -2,7 +2,7 @@ version: 2
 
 macros:
   - name: depends_on_ref
-    description: This macro adds a commented refrence to a reference model
+    description: This macro adds a commented reference to a reference model so downstream lineage remains implicit.
     docs:
       show: true
     arguments:
@@ -14,7 +14,7 @@ macros:
         description: the reference model to add
 
   - name: depends_on_source
-    description: This macro adds a commented refrence to a source model
+    description: This macro adds a commented reference to a source model so downstream lineage remains implicit.
     docs:
       show: true
     arguments:

macros/dynamic_tables/dynamic_tables.yml

@@ -2,7 +2,7 @@ version: 2
 
 macros:
   - name: target_lag_environment
-    description: This macro sets the lag duration for the target environment
+    description: Set the dynamic table target lag duration based on the active target (prod/test/other) to tune freshness vs cost.
     docs:
       show: true
     arguments:
@@ -16,6 +16,6 @@ macros:
         type: number
         description: specifies the lag duration for other environments
   - name: target_warehouse_environment
-    description: This macro sets the warehouse to be used for the dynamic table for the target environment
+    description: Select the warehouse to be used for dynamic tables based on target environment (e.g. smaller warehouse for dev/test).
     docs:
       show: true