Skip to content

Declarative event_log and pipeline monitoring support #82

@Mmodarre

Description

@Mmodarre

Summary

Add declarative event_log and monitoring configuration support to LHP, enabling automatic event log table creation and cross-pipeline monitoring through lhp.yaml project-level settings.

Features

Event Log Configuration (lhp.yaml)

  • New event_log: section in lhp.yaml for project-wide event log configuration
  • Supports catalog, schema, and name_suffix fields with LHP token substitution
  • Automatically injects event_log block into generated bundle resource YAML for all pipelines
  • Pipeline-level override: define event_log: in pipeline_config.yaml to fully replace the project setting
  • Pipeline-level opt-out: set event_log: false in pipeline_config.yaml to disable for a specific pipeline

Monitoring Pipeline

  • New monitoring: section in lhp.yaml to generate a synthetic monitoring pipeline
  • Automatically UNIONs all pipeline event log tables into a streaming table with materialized views
  • Configurable: pipeline_name, catalog, schema, streaming_table, materialized_views
  • Support for __eventlog_monitoring alias in pipeline_config.yaml for monitoring pipeline settings
  • Custom materialized views with user-defined SQL or file-based SQL paths

Bundle Resource Generation

  • Template renders event_log: block with name, schema, catalog fields
  • Monitoring pipeline generates its own bundle resource file
  • lhp generate creates monitoring flowgroup Python code alongside regular pipelines

Tests

  • Integration and unit tests for monitoring configuration and event log
  • E2E tests with baselines for event log injection, override, disable, and monitoring variants

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions