Dev/skruk/enh dynamic tables grant

.github/copilot-instructions.md

@@ -286,7 +286,7 @@ Store alongside proposal. Must be reviewed and accepted before Phase 3.
    - **Commit** — small, frequent commits per task
 1. **After all tasks**:
    - Run full test suite and `make lint`
-   - Run `scripts/dev/build.sh`
+   - Run `scripts/dev/build_docs.sh` to test build and update documentation (PLUGINS.md, SEMANTICS.md)
    - Update `docs/CHANGELOG.md` (highlights) and `docs/DEVLOG.md` (technical details)
    - Review changeset, open PR
 

.github/workflows/ci.yml

@@ -81,7 +81,7 @@ jobs:
       - name: Install dependencies
         run: |
           pip install -r requirements.txt
-          pip install pytest-mock
+          pip install pytest-mock flake8 black pylint
 
       - name: Install system dependencies
         run: |
@@ -91,6 +91,32 @@ jobs:
       - name: Run bash tests
         run: pytest test/core/test_bash_scripts.py -v
 
+  test-bash-slow:
+    runs-on: ubuntu-latest
+    if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/devel' || startsWith(github.ref, 'refs/heads/release/')
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11.12"
+
+      - name: Install dependencies
+        run: |
+          pip install -r requirements.txt
+          pip install pytest-mock flake8 black pylint
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y bats jq gawk pandoc zip
+          npm install -g markdownlint-cli2
+
+      - name: Run slow bash tests (build/package/compile)
+        run: pytest test/core/test_bash_scripts.py -v --run-slow
+
   test-core:
     runs-on: ubuntu-latest
     steps:

docs/CHANGELOG.md

@@ -16,6 +16,7 @@ Released on TBD
 
 ### Fixed in 0.9.4
 
+- **Dynamic Tables — Grant Granularity**: `P_GRANT_MONITOR_DYNAMIC_TABLES()` now derives grant scope from the `include` pattern. `DB.%.%` grants at database level, `DB.SCHEMA.%` at schema level, and `DB.SCHEMA.TABLE` on a specific named table only — eliminating previous over-granting when a schema or table was explicitly specified.
 - **Span Timestamp Handling**: Fixed spans being re-processed after agent restart due to incorrect timestamp being recorded as last-processed marker
 - **OTLP Compliance**: Fixed log `observed_timestamp` field to use nanoseconds per OTLP specification
 

docs/DEVLOG.md

@@ -63,6 +63,26 @@ This file documents detailed technical changes, internal refactorings, and devel
 
 ### Bug Fixes — Technical Details
 
+#### Dynamic Tables Grant — Schema-Level Granularity (BDX-640)
+
+- **Issue**: `P_GRANT_MONITOR_DYNAMIC_TABLES()` always granted `MONITOR` at **database level**, even when the `include` pattern specified a particular schema (e.g. `PROD_DB.ANALYTICS.%`). This caused the procedure to over-grant: a user expecting grants only on `PROD_DB.ANALYTICS` received grants on all schemas in `PROD_DB`.
+- **Root cause**: The CTE extracted only `split_part(value, '.', 0)` (the database part) and the schema part was never inspected.
+- **Fix**: Three-pass approach in `032_p_grant_monitor_dynamic_tables.sql`:
+  1. **Database pass** — `split_part(value, '.', 1) = '%'` → `GRANT … IN DATABASE`.
+  2. **Schema pass** — `split_part(value, '.', 1) != '%'` and `split_part(value, '.', 2) = '%'` → `GRANT … IN SCHEMA db.schema`.
+  3. **Table pass** — `split_part(value, '.', 1) != '%'` and `split_part(value, '.', 2) != '%'` → `GRANT … ON DYNAMIC TABLE db.schema.table` (no FUTURE grant — not supported by Snowflake at individual table level).
+- **Grant matrix**:
+
+  | Include pattern               | Grant level                         |
+  | ----------------------------- | ----------------------------------- |
+  | `%.%.%`                       | All databases                       |
+  | `PROD_DB.%.%`                 | Database `PROD_DB`                  |
+  | `PROD_DB.ANALYTICS.%`         | Schema `PROD_DB.ANALYTICS`          |
+  | `PROD_DB.ANALYTICS.ORDERS_DT` | Table `PROD_DB.ANALYTICS.ORDERS_DT` |
+
+- **Files changed**: `032_p_grant_monitor_dynamic_tables.sql`, `bom.yml`, `config.md`
+- **Tests added**: `test/bash/test_grant_monitor_dynamic_tables.bats` — structural content checks covering both grant paths
+
 #### Log ObservedTimestamp Unit Correction
 
 - **Issue**: OTel log `observed_timestamp` field was sent in milliseconds

docs/PLUGINS.md

@@ -203,7 +203,7 @@ plugins:
 
 ### Enabling the Budgets plugin
 
-1. Set `IS_ENABLED` to `true` in your configuration file.
+1. Set `is_enabled` to `true` in your configuration file.
 1. For **account budget only** (no custom budgets): no additional grants needed — `SNOWFLAKE.BUDGET_VIEWER` is already granted.
 1. For **custom budgets**: configure `monitored_budgets` and run `P_GRANT_BUDGET_MONITORING()` (admin scope required), or grant privileges
    manually (see below).
@@ -417,7 +417,18 @@ plugins:
 > default, when the `admin` scope is installed, this is handled by the `P_GRANT_MONITOR_DYNAMIC_TABLES()` procedure, which is executed with
 > the elevated privileges of the `DTAGENT_ADMIN` role (created only when the `admin` scope is installed), via the
 > `APP.TASK_DTAGENT_DYNAMIC_TABLES_GRANTS` task. The schedule for this task can be configured separately using the
-> `PLUGINS.DYNAMIC_TABLES.SCHEDULE_GRANTS` configuration option. Alternatively, you may choose to:
+> `PLUGINS.DYNAMIC_TABLES.SCHEDULE_GRANTS` configuration option.
+
+The grant granularity is derived automatically from the `include` pattern:
+
+| Include pattern               | Grant level | SQL issued                                                 |
+| ----------------------------- | ----------- | ---------------------------------------------------------- |
+| `%.%.%` or `PROD_DB.%.%`      | Database    | `GRANT MONITOR ON ALL/FUTURE DYNAMIC TABLES IN DATABASE …` |
+| `PROD_DB.ANALYTICS.%`         | Schema      | `GRANT MONITOR ON ALL/FUTURE DYNAMIC TABLES IN SCHEMA …`   |
+| `PROD_DB.ANALYTICS.ORDERS_DT` | Table       | `GRANT MONITOR ON DYNAMIC TABLE …` (no FUTURE grant)       |
+
+Alternatively, you may choose to grant the required permissions manually, using the appropriate
+`GRANT MONITOR ON ALL/FUTURE DYNAMIC TABLES IN …` statement, depending on the desired granularity.
 
 ### Dynamic Tables Bill of Materials
 
@@ -437,14 +448,17 @@ The following tables list the Snowflake objects that this plugin delivers data f
 
 #### Objects referenced by the `Dynamic Tables` plugin
 
-| Name                                             | Type          | Privileges | Granted to     | Comment                                                                    |
-| ------------------------------------------------ | ------------- | ---------- | -------------- | -------------------------------------------------------------------------- |
-| SHOW DATABASES                                   | command       | USAGE      |                |                                                                            |
-| ALL DYNAMIC TABLES IN DATABASE $database         | dynamic table | MONITOR    | DTAGENT_VIEWER | We grant that on every database selected in configuration or all (default) |
-| ALL FUTURE TABLES IN DATABASE $database          | table         | MONITOR    | DTAGENT_VIEWER | We grant that on every database selected in configuration or all (default) |
-| INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY | view          | USAGE      | DTAGENT_VIEWER |                                                                            |
-| INFORMATION_SCHEMA.DYNAMIC_TABLE_GRAPH_HISTORY   | view          | USAGE      | DTAGENT_VIEWER |                                                                            |
-| INFORMATION_SCHEMA.DYNAMIC_TABLES                | view          | USAGE      | DTAGENT_VIEWER |                                                                            |
+| Name                                                  | Type          | Privileges | Granted to     | Comment                                                                                                                  |
+| ----------------------------------------------------- | ------------- | ---------- | -------------- | ------------------------------------------------------------------------------------------------------------------------ |
+| SHOW DATABASES                                        | command       | USAGE      |                |                                                                                                                          |
+| ALL DYNAMIC TABLES IN DATABASE $database              | dynamic table | MONITOR    | DTAGENT_VIEWER | Granted when include pattern has wildcard schema (e.g. DB.%.%)                                                           |
+| ALL FUTURE DYNAMIC TABLES IN DATABASE $database       | dynamic table | MONITOR    | DTAGENT_VIEWER | Granted when include pattern has wildcard schema (e.g. DB.%.%)                                                           |
+| ALL DYNAMIC TABLES IN SCHEMA $database.$schema        | dynamic table | MONITOR    | DTAGENT_VIEWER | Granted when include pattern has specific schema (e.g. DB.ANALYTICS.%)                                                   |
+| ALL FUTURE DYNAMIC TABLES IN SCHEMA $database.$schema | dynamic table | MONITOR    | DTAGENT_VIEWER | Granted when include pattern has specific schema (e.g. DB.ANALYTICS.%)                                                   |
+| DYNAMIC TABLE $database.$schema.$table                | dynamic table | MONITOR    | DTAGENT_VIEWER | Granted when include pattern specifies an exact table name (e.g. DB.ANALYTICS.ORDERS_DT); no FUTURE grant at table level |
+| INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY      | view          | USAGE      | DTAGENT_VIEWER |                                                                                                                          |
+| INFORMATION_SCHEMA.DYNAMIC_TABLE_GRAPH_HISTORY        | view          | USAGE      | DTAGENT_VIEWER |                                                                                                                          |
+| INFORMATION_SCHEMA.DYNAMIC_TABLES                     | view          | USAGE      | DTAGENT_VIEWER |                                                                                                                          |
 
 <a name="event_log_info_sec"></a>
 

pytest.ini

@@ -4,4 +4,5 @@ addopts = --ignore-glob=**/otel_*_test.py
 log_cli = false
 log_cli_level = DEBUG
 markers =
-    xdist_group: mark test to be executed in named group
+    xdist_group: mark test to be executed in named group
+    slow: mark test as slow (build/package integration tests, skipped by default)

requirements.txt

@@ -83,3 +83,6 @@ protobuf>=5.29.6,<6.0.0
 urllib3>=2.6.3
 # CVE-2026-24049: Wheel path traversal vulnerability
 wheel>=0.46.2
+
+# dependencies
+chardet>=3.0.2,<6.0.0

src/dtagent/plugins/budgets.config/config.md

@@ -7,7 +7,7 @@
 
 ### Enabling the Budgets plugin
 
-1. Set `IS_ENABLED` to `true` in your configuration file.
+1. Set `is_enabled` to `true` in your configuration file.
 1. For **account budget only** (no custom budgets): no additional grants needed — `SNOWFLAKE.BUDGET_VIEWER` is already granted.
 1. For **custom budgets**: configure `monitored_budgets` and run `P_GRANT_BUDGET_MONITORING()` (admin scope required), or grant
    privileges manually (see below).

src/dtagent/plugins/dynamic_tables.config/bom.yml

@@ -23,12 +23,27 @@ references:
     type: dynamic table
     privileges: MONITOR
     granted to: DTAGENT_VIEWER
-    comment: We grant that on every database selected in configuration or all (default)
-  - name: ALL FUTURE TABLES IN DATABASE $database
-    type: table
+    comment: Granted when include pattern has wildcard schema (e.g. DB.%.%)
+  - name: ALL FUTURE DYNAMIC TABLES IN DATABASE $database
+    type: dynamic table
+    privileges: MONITOR
+    granted to: DTAGENT_VIEWER
+    comment: Granted when include pattern has wildcard schema (e.g. DB.%.%)
+  - name: ALL DYNAMIC TABLES IN SCHEMA $database.$schema
+    type: dynamic table
+    privileges: MONITOR
+    granted to: DTAGENT_VIEWER
+    comment: Granted when include pattern has specific schema (e.g. DB.ANALYTICS.%)
+  - name: ALL FUTURE DYNAMIC TABLES IN SCHEMA $database.$schema
+    type: dynamic table
+    privileges: MONITOR
+    granted to: DTAGENT_VIEWER
+    comment: Granted when include pattern has specific schema (e.g. DB.ANALYTICS.%)
+  - name: DYNAMIC TABLE $database.$schema.$table
+    type: dynamic table
     privileges: MONITOR
     granted to: DTAGENT_VIEWER
-    comment: We grant that on every database selected in configuration or all (default)
+    comment: Granted when include pattern specifies an exact table name (e.g. DB.ANALYTICS.ORDERS_DT); no FUTURE grant at table level
   - name: INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY
     type: view
     privileges: USAGE

src/dtagent/plugins/dynamic_tables.config/config.md

@@ -1,5 +1,13 @@
 > **IMPORTANT**: For this plugin to function correctly, `MONITOR on DYNAMIC TABLES` must be granted to the `DTAGENT_VIEWER` role.
 > By default, when the `admin` scope is installed, this is handled by the `P_GRANT_MONITOR_DYNAMIC_TABLES()` procedure, which is executed with the elevated privileges of the `DTAGENT_ADMIN` role (created only when the `admin` scope is installed), via the `APP.TASK_DTAGENT_DYNAMIC_TABLES_GRANTS` task.
 > The schedule for this task can be configured separately using the `PLUGINS.DYNAMIC_TABLES.SCHEDULE_GRANTS` configuration option.
-> Alternatively, you may choose to:
->
+
+The grant granularity is derived automatically from the `include` pattern:
+
+| Include pattern               | Grant level | SQL issued                                                 |
+| ----------------------------- | ----------- | ---------------------------------------------------------- |
+| `%.%.%` or `PROD_DB.%.%`      | Database    | `GRANT MONITOR ON ALL/FUTURE DYNAMIC TABLES IN DATABASE …` |
+| `PROD_DB.ANALYTICS.%`         | Schema      | `GRANT MONITOR ON ALL/FUTURE DYNAMIC TABLES IN SCHEMA …`   |
+| `PROD_DB.ANALYTICS.ORDERS_DT` | Table       | `GRANT MONITOR ON DYNAMIC TABLE …` (no FUTURE grant)       |
+
+Alternatively, you may choose to grant the required permissions manually, using the appropriate `GRANT MONITOR ON ALL/FUTURE DYNAMIC TABLES IN …` statement, depending on the desired granularity.