-
Notifications
You must be signed in to change notification settings - Fork 8
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Summary
Add declarative event_log and monitoring configuration support to LHP, enabling automatic event log table creation and cross-pipeline monitoring through lhp.yaml project-level settings.
Features
Event Log Configuration (lhp.yaml)
- New
event_log:section inlhp.yamlfor project-wide event log configuration - Supports
catalog,schema, andname_suffixfields with LHP token substitution - Automatically injects
event_logblock into generated bundle resource YAML for all pipelines - Pipeline-level override: define
event_log:inpipeline_config.yamlto fully replace the project setting - Pipeline-level opt-out: set
event_log: falseinpipeline_config.yamlto disable for a specific pipeline
Monitoring Pipeline
- New
monitoring:section inlhp.yamlto generate a synthetic monitoring pipeline - Automatically UNIONs all pipeline event log tables into a streaming table with materialized views
- Configurable:
pipeline_name,catalog,schema,streaming_table,materialized_views - Support for
__eventlog_monitoringalias inpipeline_config.yamlfor monitoring pipeline settings - Custom materialized views with user-defined SQL or file-based SQL paths
Bundle Resource Generation
- Template renders
event_log:block withname,schema,catalogfields - Monitoring pipeline generates its own bundle resource file
lhp generatecreates monitoring flowgroup Python code alongside regular pipelines
Tests
- Integration and unit tests for monitoring configuration and event log
- E2E tests with baselines for event log injection, override, disable, and monitoring variants
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request