Draft
Conversation
…te context section
…sts and validation processes
…and prompt file locations (#58)
* feat: Standardize timestamp handling across APIs - Refactored timestamp handling to use a unified approach with smart unit detection, ensuring all SQL views produce nanoseconds. - Enhanced `validate_timestamp()` to work internally in nanoseconds, with added precision and strict parameter validation. - Introduced `process_timestamps_for_telemetry()` utility for consistent timestamp processing across logs and events. - Updated documentation to clarify timestamp requirements for Dynatrace and OTLP standards. - Improved build scripts to automatically activate the virtual environment. - Comprehensive updates to tests for new timestamp validation logic and utility functions. * feat: Enhance timestamp validation to support skipping range checks * feat: Improve timestamp handling by preserving out-of-range observed_timestamp values * feat: Add comprehensive documentation for markdown style, changelog, and development log * feat: Simplify Python environment instructions and enhance documentation clarity
* Update copilot instructions and VSCode settings for improved linting and prompt file locations
* feat: migrate test fixtures to NDJSON format and enhance plugin development docs
* refactor(tests): remove all pickle/pkl references, clean up backward-compat aliases
- Remove stale LFS entries for .pkl and *_results.txt from .gitattributes
- Update copilot-instructions.md: fixture pattern, test mode desc, _utils.py row, test strategy
- Update CONTRIBUTING.md: rename 'Regenerate test data (Pickles)' section to NDJSON
- Add CHANGELOG entry for NDJSON fixture migration
- Rename PICKLE_CONF shell variable to CONF_FLAG in test_core.sh
- Remove backward-compat aliases _pickle_data_history, _pickle_all,
_safe_get_unpickled_entries, should_pickle from test/_utils.py
- Fix stale docstring reference ('pickled data') in telemetry_test_sender()
- Add R0911 (too-many-return-statements) to test/.pylintrc disable list (pre-existing)
* refactor(tests): replace pickle references with safe data fixture terminology and update related documentation
* fix(docs): correct spelling and update test data format from pickle to NDJSON
* feat: Standardize timestamp handling across APIs (#57) * feat: Standardize timestamp handling across APIs - Refactored timestamp handling to use a unified approach with smart unit detection, ensuring all SQL views produce nanoseconds. - Enhanced `validate_timestamp()` to work internally in nanoseconds, with added precision and strict parameter validation. - Introduced `process_timestamps_for_telemetry()` utility for consistent timestamp processing across logs and events. - Updated documentation to clarify timestamp requirements for Dynatrace and OTLP standards. - Improved build scripts to automatically activate the virtual environment. - Comprehensive updates to tests for new timestamp validation logic and utility functions. * feat: Enhance timestamp validation to support skipping range checks * feat: Improve timestamp handling by preserving out-of-range observed_timestamp values * feat: Add comprehensive documentation for markdown style, changelog, and development log * feat: Simplify Python environment instructions and enhance documentation clarity * test fixtures clean (#59) * feat: migrate test fixtures to NDJSON format and enhance plugin development docs * refactor(tests): remove all pickle/pkl references, clean up backward-compat aliases - Remove stale LFS entries for .pkl and *_results.txt from .gitattributes - Update copilot-instructions.md: fixture pattern, test mode desc, _utils.py row, test strategy - Update CONTRIBUTING.md: rename 'Regenerate test data (Pickles)' section to NDJSON - Add CHANGELOG entry for NDJSON fixture migration - Rename PICKLE_CONF shell variable to CONF_FLAG in test_core.sh - Remove backward-compat aliases _pickle_data_history, _pickle_all, _safe_get_unpickled_entries, should_pickle from test/_utils.py - Fix stale docstring reference ('pickled data') in telemetry_test_sender() - Add R0911 (too-many-return-statements) to test/.pylintrc disable list (pre-existing) * refactor(tests): replace pickle references with safe data fixture terminology and update related documentation * fix(docs): correct spelling and update test data format from pickle to NDJSON * Fix path to CHANGELOG in release workflow for extracting release notes * Fix documentation 093 (#60) * fixing smaller issues in documentation * Update documentation to include Dynatrace Subscription compatibility details * Update documentation and comments for BizEvents, DavisEvents, and GenericEvents classes to reflect API changes and clarify usage * Fix Shares Plugin to correctly report inbound shares with deleted databases - Set `IS_REPORTED` to TRUE for deleted-DB shares in P_GET_SHARES procedure - Add user-friendly message in V_INBOUND_SHARE_TABLES view for deleted database scenario - Format rs_unavailable RESULTSET declaration for better readability - Update test fixtures and expectations to include deleted-DB share test case - Update CHANGELOG to clarify the fix scope and behavior
* refactor: Fix plugin disable logic to properly handle disabled-by-default plugins - Change condition from `(IS_DISABLED or (IS_DISABLED_BY_DEFAULT and not IS_ENABLED))` to `((IS_DISABLED or IS_DISABLED_BY_DEFAULT) and not IS_ENABLED)` - Ensure disabled-by-default plugins can be enabled via configuration - Prevent tasks from running when plugin is disabled or disabled-by-default without explicit enablement * refactor: Update budget JSON field names to uppercase and fix timestamp conversion - Change JSON field references from lowercase to uppercase (e.g., `created_on` to `"CREATED_ON"`) - Fix timestamp conversion from text parsing to numeric division by 1000 - Update `database_name` field reference from `database_name` to `"DATABASE"` - Update `schema_name` field reference from `schema_name` to `"SCHEMA"` - Align field extraction with actual SYSTEM$SHOW_BUDGETS_IN_ACCOUNT() output format * refactor: Add identifier validation and proper quoting for custom budget monitoring - Add regex validation for budget identifiers (`^[A-Za-z_][A-Za-z0-9_$]*$` per part) - Skip invalid budget FQNs with warning log instead of failing - Uppercase all budget name parts (database, schema, budget) for consistency - Add proper identifier quoting in grant statements to prevent SQL injection - Update config.md to document identifier requirements and automatic uppercasing * refactor: Add identifier validation and proper quoting for shares and query history plugins - Add regex validation for database names (`^[A-Za-z_][A-Za-z0-9_$]*$`) in shares plugin - Add regex validation for query IDs (UUID format) in query history plugin - Skip invalid identifiers with warning log instead of failing - Uppercase database names for consistency in shares plugin - Add proper identifier quoting in SQL statements to prevent SQL injection - Sanitize share names by removing single quotes before * refactor(tests): Allow passing custom TestConfiguration to execute_telemetry_test - Add optional `config` parameter to execute_telemetry_test function - Use provided config if available, otherwise fall back to get_config() - Pass explicit config to execute_telemetry_test in test_budgets.py - Enables better test isolation and configuration control in telemetry tests * refactor: Use shutil to resolve bats executable path for improved compatibility * fix: Update plugin schedule call to include grants array for proper execution * refactor: Standardize formatting in budgets configuration documentation
* feat: Add cross-tenant monitoring support to Event Log plugin
- Introduce `plugins.event_log.cross_tenant_monitoring` config option (default: true) to enable reporting WARN/ERROR entries from all DTAGENT_*_DB instances
- Add `plugins.event_log.databases` config for filtering monitored databases using SQL LIKE patterns
- Implement F_EVENT_LOG_INCLUDE() function to centralize database filtering logic across logs, metrics, and spans views
- Update V_EVENT_LOG, V_EVENT_LOG_METRICS_INSTRUMENTED, and V_EVENT
* fix: Update changelog and config documentation for clarity and accuracy
* fix: Correct function signature and improve documentation clarity in event log configuration (based on PR review)
* fix: Update changelog to clarify fixes in Shares and Event Log plugins for version 0.9.4
#67) * feat: Add two-phase commit for query telemetry and configurable event log lookback window - Implement two-phase commit pattern in `_process_span_rows` to prevent silent data loss when trace export fails by only marking queries as processed after successful OTLP flush - Add `plugins.event_log.lookback_hours` configuration option (default: 24) to replace hardcoded 24-hour lookback window in `V_EVENT_LOG` - Add comprehensive test coverage for query hierarchy validation with nested stored procedure chains * refactor: Skip trace flush when spans are disabled to prevent unnecessary errors - Check for `NOT_ENABLED` attribute on spans object before attempting flush - Set `flush_succeeded` to `True` when spans are disabled to avoid false error reporting - Prevents "Problem flushing traces" errors when span collection is intentionally disabled * docs: Clarify cross-tenant monitoring configuration for event log plugin
* feat: Add two-phase commit for query telemetry and configurable event log lookback window - Implement two-phase commit pattern in `_process_span_rows` to prevent silent data loss when trace export fails by only marking queries as processed after successful OTLP flush - Add `plugins.event_log.lookback_hours` configuration option (default: 24) to replace hardcoded 24-hour lookback window in `V_EVENT_LOG` - Add comprehensive test coverage for query hierarchy validation with nested stored procedure chains * feat: Add authentication method attributes to Users plugin - Add `snowflake.user.has_mfa` attribute to indicate multi-factor authentication enrollment status - Add `snowflake.user.has_pat` attribute to indicate programmatic access token configuration - Add `snowflake.user.has_rsa` attribute to indicate RSA public key authentication configuration - Add `snowflake.user.has_workload_identity` attribute to indicate workload identity federation configuration - Update TMP_USERS and TMP_USERS_HELPER table * fix: Update log attribute adjustment to handle None values correctly fix: Update secure_objects_only field to use TRY_TO_BOOLEAN for proper type conversion
* fix: Correct span timestamp tracking to prevent re-processing after agent restart
- Track actual last row timestamp in `_process_span_rows()` instead of using `current_timestamp()` column expression
- Remove unused `current_timestamp` import from snowflake.snowpark.functions
- Add `lookback_hours` configuration parameter to event_log plugin (default: 24)
- Update all three event_log SQL views to use configurable lookback window
- Expand event_log configuration documentation with cost optimization guidance
* feat: Add support for SNOWFLAKE.TELEMETRY.EVENTS as account-level event table
- Remove 'snowflake.telemetry.events' from excluded event tables array in SETUP_EVENT_TABLE()
- Wrap GRANT SELECT statement in exception handler to silently ignore failures on read-only or Snowflake-managed tables
- Create DTAGENT_DB.STATUS.EVENT_LOG as view over snowflake.telemetry.events when configured
- Update CHANGELOG and DEVLOG with BDX-1172 fix details and behavior explanation
* feat: Add configurable lookback_hours parameter to all plugins with incremental views
- Add `lookback_hours` configuration parameter to data_schemas (4h), event_usage (6h), login_history (24h), tasks (4h), and warehouse_usage (24h) plugins
- Replace hardcoded lookback values in SQL views with `F_GET_CONFIG_VALUE('plugins.<plugin>.lookback_hours', <default>)` calls
- Unify tasks plugin to use single `lookback_hours` parameter for both serverless_tasks and task_versions views (previously 4h and 1
* refactor: Split tasks plugin lookback configuration into separate keys for serverless and version views
- Add `lookback_hours_versions` parameter (default: 720h = 1 month) to tasks plugin configuration
- Update `063_v_task_versions.sql` to use `plugins.tasks.lookback_hours_versions` instead of shared `lookback_hours`
- Preserve original defaults: `lookback_hours` remains 4h for serverless tasks, `lookback_hours_versions` uses 720h for task versions
- Update documentation to clarify separate look
* chore: Update markdown linting to use markdownlint-cli2 and adjust configuration tables for consistency
* feat: Update documentation and configuration for plugins
- Enhanced the `INSTALL.md` and `PLUGINS.md` files to clarify the `lookback_hours` parameter and its behavior across various plugins.
- Added detailed configuration options for the `data_schemas`, `event_log`, `event_usage`, `login_history`, `tasks`, and `warehouse_usage` plugins, ensuring consistency in descriptions and default values.
- Updated the SQL initialization script for the event log to improve clarity on table handling.
- Adjusted the `event_log-config.yml` to modify default values for `lookback_hours` and `retention_hours`.
* default lookback was 24h - adjusting to avoid product change
* feat: Update event log plugin to log warnings on grant failures and adjust lookback and retention hours in configuration
* feat: Enhance dynamic tables grant with schema-level granularity * refactor: Update documentation and tests to reflect three-pass grant logic for Dynamic Tables - Add table-level grant pass to `032_p_grant_monitor_dynamic_tables.sql` documentation - Update CHANGELOG and DEVLOG to clarify three-pass approach (database, schema, table) - Update grant matrix in config.md and PLUGINS.md to include table-level pattern example - Add DYNAMIC TABLE entry to bom.yml for exact table name grants - Fix PLUGINS.md Event Log description to clarify cross-tenant monitoring behavior * fixing formatting issues * fix: Update configuration documentation and SQL scripts for Budgets and Dynamic Tables plugins * fix: test data now contains "0/false" metric values * feat: add support for slow tests and update test configurations * refactor: update test scripts to use new paths and improve build fixture creation * feat: add separate CI job for slow bash tests and update test dependencies - Add new `test-bash-slow` job that runs only on main, devel, and release branches - Configure slow tests to run with `--run-slow` flag for build/package/compile operations - Install flake8, black, and pylint dependencies in both bash test jobs * fix: ensure build directory exists in compile tests and improve test output formatting - Create build/30_plugins and build/09_upgrade directories in test_compile.bats setup to prevent failures in fresh CI checkouts - Add stdout output to bash test failure messages for better debugging alongside existing stderr output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.