You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This proposal recommends overhauling Imbi's notification filtering system to support complex boolean logic using CEL (Common Expression Language) expressions. The current system uses simple equality checks with implicit OR logic, making it impossible to express common filtering requirements like "main branch AND completed status" or conditional fact routing.
Key Changes:
Add CEL expression support to notification_rules for per-fact filtering
Remove action type system (process/ignore) and precedence rules
Enable single webhook URL to conditionally update multiple facts
Benefits:
Solves inability to express AND/OR boolean logic
Eliminates need for duplicate webhook registrations
Simplifies mental model with rule-level filtering only
Human-readable filter syntax
Industry-standard expression language
Problem Statement
Current Architecture Limitations
The existing notification filter system has fundamental design constraints that prevent common use cases:
1. Cannot Express Boolean AND Logic
Problem: Each filter is evaluated independently with implicit OR semantics. There's no way to require multiple conditions to be true simultaneously.
Example: "Only process workflow runs on main branch that are completed"
-- Current system: CANNOT express this correctlyINSERT INTO notification_filters VALUES
('main-branch', 'github', 'workflow-status', '/workflow_run/head_branch', '==', 'main', 'process'),
('completed', 'github', 'workflow-status', '/action', '==', 'completed', 'process');
-- Result with default='process': Processes if branch=main OR action=completed (not AND!)-- Result with default='ignore': Still processes if branch=main OR action=completed (still OR!)
No workaround exists: The system always ORs filters of the same action type together. You cannot express AND logic.
2. Cannot Route Different Facts from Same Webhook
Problem: Filters apply at notification level, affecting ALL rules. To filter differently for different facts, you must register the webhook multiple times at different URLs.
Example: GitHub workflow status webhooks contain data for both build and deploy facts, but each should only update on specific workflows:
-- Current system: Must create TWO separate notifications-- Notification 1: workflow-buildINSERT INTO notification_filters VALUES
('build-only', 'github', 'workflow-build', '/workflow_run/name', '==', 'build', 'process');
INSERT INTO notification_rules VALUES
(build_status_id, 'github', 'workflow-build', '/workflow_run/conclusion');
-- Notification 2: workflow-deployINSERT INTO notification_filters VALUES
('deploy-only', 'github', 'workflow-deploy', '/workflow_run/name', '==', 'deploy', 'process');
INSERT INTO notification_rules VALUES
(deploy_status_id, 'github', 'workflow-deploy', '/workflow_run/conclusion');
-- Result: GitHub sends webhook to BOTH URLs, duplicate processing!
Impact:
Double the webhook traffic
Double the database queries (project lookups)
Double the OpenSearch updates
More complex GitHub webhook configuration
3. Action Type Precedence Creates Confusion
Problem: The action field (process vs ignore) with implicit precedence rules (ignore always wins) creates counterintuitive behavior.
Feature branch, not draft → PROCESSES (no filters match, uses default)
Main branch, draft → IGNORES (ignore wins over process)
Feature branch, draft → IGNORES (ignore matches)
Issue: The action='process' filter on main branch is effectively meaningless with default='process'—it would process anyway! Users expect it to mean "only process main" but it doesn't enforce that.
4. Limited Operations
Problem: Only supports == and != operations.
Cannot Express:
Numeric comparisons: "exit_code > 0"
String operations: "branch starts with 'release/'"
List membership: "workflow in ['build', 'deploy', 'test']"
Field existence: "has conclusion field"
Complex negation: "NOT (draft OR archived)"
5. No Complex Nested Logic
Problem: Cannot express "(A AND B) OR (C AND D)" type logic.
Example: "Process if (production deploy) OR (any release to staging)"
(environment == "production" AND workflow == "deploy")
OR
(environment == "staging" AND branch startsWith "release/")
This is impossible with the current system.
Current System Architecture
Database Schema
-- notification_filters: Accept/reject filters at notification levelCREATETABLEnotification_filters (
filter_name TEXTNOT NULL,
integration_name TEXTNOT NULL,
notification_name TEXTNOT NULL,
pattern TEXTNOT NULL, -- JSON pointer
operation notification_filter_operation_type NOT NULL, -- Only '==' or '!='
value TEXTNOT NULL,
action notification_action_type NOT NULL, -- 'ignore' or 'process'PRIMARY KEY (filter_name, integration_name, notification_name)
);
-- notification_rules: Fact update rules at notification levelCREATETABLEnotification_rules (
fact_type_id INTEGERNOT NULL,
integration_name TEXTNOT NULL,
notification_name TEXTNOT NULL,
pattern TEXTNOT NULL, -- JSON pointer for extractionPRIMARY KEY (fact_type_id, integration_name, notification_name)
);
Processing Flow
def_process_notification(self, integration_name, notification_name, body):
# 1. Evaluate ALL notification-level filtersifnotself._evaluate_filters(self._notification, body):
return# Early exit - blocks ALL rules# 2. If passed, process ALL rules for this notificationforprojectinawaitself._get_projects(self._notification, body):
updates= {}
forruleinnotification.rules: # No per-rule filtering!value=rule.pattern.resolve(body, unspecified)
ifvalueisnotunspecified:
updates[rule.fact_type_id] =valueifupdates:
awaitself._update_facts(project, updates)
Filter Evaluation Logic
def_evaluate_filters(self, notification, body) ->bool:
matches= []
forfilterinnotification.filters:
value=filter.pattern.resolve(body, unspecified)
ifvalueisnotunspecified:
if (value==filter.valueandfilter.operation=='==') or \
(value!=filter.valueandfilter.operation=='!='):
matches.append((filter, value))
# Separate by action typeignores= [fforf, vinmatchesiff.action=='ignore']
process= [fforf, vinmatchesiff.action=='process']
# Precedence: ignore always winsifignores:
returnFalseelifprocess:
returnTrueelse:
# No matches - use defaultreturnnotification.default_action=='process'
Key Issues:
Filters are evaluated independently (implicit OR within same action type)
Ignore action beats process action (hard-coded precedence)
No AND logic possible
All-or-nothing: filters block entire notification or allow all rules
Proposed Solution
High-Level Design
Replace the multi-filter system with rule-level CEL (Common Expression Language) expressions:
Rule Filters (rule-level): Optional expression per rule determining if that specific fact should be updated
Key Principles:
Expressions are boolean: TRUE = pass, FALSE = skip
CEL provides human-readable syntax with full boolean logic
Each rule independently decides if it should fire
No action types, no precedence rules, no implicit OR behavior
Database Schema Changes
-- Drop old complex systemDROPTABLE notification_filters;
DROPTYPE notification_action_type;
DROPTYPE notification_filter_operation_type;
-- Add rule-level filteringALTERTABLE notification_rules
ADD COLUMN filter_expression TEXT; -- Optional CEL expressionCOMMENT ON COLUMN notification_rules.filter_expression IS'Optional CEL expression that must evaluate to true for this rule to apply.';
CEL Expression Examples
CEL (Common Expression Language) is an industry-standard expression language used by Google Cloud, Kubernetes, Firebase, and others. It provides familiar syntax similar to Python/JavaScript.
// Check if field existshas(workflow_run.conclusion)// Null checkingworkflow_run.error==null// Ternary operator for optional fieldshas(workflow_run.environment) ? workflow_run.environment=="production" : false
Complex nested logic:
// (A AND B) OR (C AND D)(workflow_run.environment=="production"&&action=="deploy")||(workflow_run.environment=="staging"&&workflow_run.head_branch.startsWith("release/"))
Phase 1: Add CEL Expression Support (Non-Breaking)
Goal: Add CEL expression support to rules while maintaining backward compatibility with existing filters.
Changes:
-- Add rule-level filteringALTERTABLE notification_rules
ADD COLUMN filter_expression TEXT;
COMMENT ON COLUMN notification_rules.filter_expression IS'Optional CEL expression that must evaluate to true for this rule to apply.';
-- Mark old table as deprecated (don't drop yet)COMMENT ON TABLE notification_filters IS'DEPRECATED: Use filter_expression on notification_rules instead. Will be removed in future version.';
Processing logic:
asyncdef_process_notification(self, integration_name, notification_name, body):
# Check old filters for backward compatibilityifself._notification.filters: # Old systemifnotself._evaluate_filters(self._notification, body):
return# Process rules with optional rule-level filtersforprojectinawaitself._get_projects(self._notification, body):
updates= {}
forruleinself._notification.rules:
# Check rule-level filter if presentifrule.filter_expression:
ifnotevaluate_cel(rule.filter_expression, body):
continue# ... existing extraction and transformation logic
Impact:
Existing notifications and rules work unchanged
New rules can use CEL expressions for conditional updates
Solves conditional fact routing immediately
Both systems run in parallel during transition
Manual Migration:
Convert existing notification filters to CEL expressions on rules.
Phase 2: Remove Old System (Breaking)
Goal: Remove deprecated notification_filters table after all filters migrated to CEL.
Prerequisites:
All existing filters converted to CEL expressions
Documentation updated
Changes:
-- Drop old tableDROPTABLE notification_filters;
-- Drop deprecated typesDROPTYPE notification_action_type;
DROPTYPE notification_filter_operation_type;
-- Remove default_action (no longer needed)ALTERTABLE integration_notifications
DROP COLUMN default_action;
Concrete Examples
Example 1: GitHub Workflow Status
Scenario: Update build and deploy facts from single workflow_run webhook.
-- Notification: single webhook endpointINSERT INTO integration_notifications VALUES
('github', 'workflow-status');
-- Rule 1: Update build_status only for build jobs on mainINSERT INTO notification_rules VALUES
(build_status_id, 'github', 'workflow-status',
'/workflow_run/jobs/0/conclusion',
'workflow_run.jobs.exists(j, j.name == "build") && workflow_run.head_branch == "main" && action == "completed"');
-- Rule 2: Update deploy_status only for deploy jobs on mainINSERT INTO notification_rules VALUES
(deploy_status_id, 'github', 'workflow-status',
'/workflow_run/jobs/1/conclusion',
'workflow_run.jobs.exists(j, j.name == "deploy") && workflow_run.head_branch == "main" && action == "completed"');
Result: Single webhook conditionally updates both facts based on job presence and branch.
Example 2: Ignore Non-Production Deploys
Scenario: Only update deploy facts for production deployments, ignore staging/dev.
Configuration:
-- NotificationINSERT INTO integration_notifications VALUES
('github', 'deployment');
-- Rule: extract deploy status only for productionINSERT INTO notification_rules VALUES
(deploy_status_id, 'github', 'deployment',
'/deployment/status',
'deployment.environment == "production"');
Example 3: Complex Multi-Condition Routing
Scenario: Update different facts based on complex conditions:
test_coverage: Any test workflow on main, completed
deploy_status: Only deploy workflows to production, completed
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Executive Summary
This proposal recommends overhauling Imbi's notification filtering system to support complex boolean logic using CEL (Common Expression Language) expressions. The current system uses simple equality checks with implicit OR logic, making it impossible to express common filtering requirements like "main branch AND completed status" or conditional fact routing.
Key Changes:
notification_rulesfor per-fact filteringnotification_filterstable entirelyactiontype system (process/ignore) and precedence rulesBenefits:
Problem Statement
Current Architecture Limitations
The existing notification filter system has fundamental design constraints that prevent common use cases:
1. Cannot Express Boolean AND Logic
Problem: Each filter is evaluated independently with implicit OR semantics. There's no way to require multiple conditions to be true simultaneously.
Example: "Only process workflow runs on main branch that are completed"
No workaround exists: The system always ORs filters of the same action type together. You cannot express AND logic.
2. Cannot Route Different Facts from Same Webhook
Problem: Filters apply at notification level, affecting ALL rules. To filter differently for different facts, you must register the webhook multiple times at different URLs.
Example: GitHub workflow status webhooks contain data for both build and deploy facts, but each should only update on specific workflows:
Impact:
3. Action Type Precedence Creates Confusion
Problem: The
actionfield (processvsignore) with implicit precedence rules (ignore always wins) creates counterintuitive behavior.Example: With
default_action='process':Behavior:
Issue: The
action='process'filter on main branch is effectively meaningless withdefault='process'—it would process anyway! Users expect it to mean "only process main" but it doesn't enforce that.4. Limited Operations
Problem: Only supports
==and!=operations.Cannot Express:
5. No Complex Nested Logic
Problem: Cannot express "(A AND B) OR (C AND D)" type logic.
Example: "Process if (production deploy) OR (any release to staging)"
This is impossible with the current system.
Current System Architecture
Database Schema
Processing Flow
Filter Evaluation Logic
Key Issues:
Proposed Solution
High-Level Design
Replace the multi-filter system with rule-level CEL (Common Expression Language) expressions:
Key Principles:
Database Schema Changes
CEL Expression Examples
CEL (Common Expression Language) is an industry-standard expression language used by Google Cloud, Kubernetes, Firebase, and others. It provides familiar syntax similar to Python/JavaScript.
Basic comparisons:
Boolean logic:
List operations:
String operations:
Field existence and conditionals:
Complex nested logic:
Processing Flow (New)
Python Implementation
Dependencies:
Solution to Original Problems
1. Boolean AND Logic ✅
Before (impossible):
-- Cannot express "main AND completed"After:
2. Conditional Fact Routing ✅
Before (requires 2 notifications):
After (single notification):
3. No More Action Precedence Confusion ✅
Before:
After:
4. Rich Operations ✅
Before (only == and !=):
After (full CEL operators):
5. Complex Nested Logic ✅
Before (impossible):
-- Cannot express (A AND B) OR (C AND D)After:
Alternative Approaches Considered
Alternative 1: JSONLogic Expression Trees
Description: Use JSONLogic specification for structured JSON boolean expressions.
Example:
{ "and": [ {"==": [{"var": "workflow_run.head_branch"}, "main"]}, {"==": [{"var": "action"}, "completed"]} ] }Pros:
Cons:
Verdict: REJECTED - Poor developer experience for hand-editing. CEL is more readable.
Alternative 2: Extended Filter Groups (Backward Compatible)
Description: Keep existing filter system, add grouping table with AND/OR operators.
Schema:
Example:
Pros:
Cons:
Verdict: REJECTED - Adds complexity without fully solving problems. Half-measure.
Alternative 3: SQL WHERE Clause Syntax
Description: Use SQL-like WHERE clause syntax for filters.
Example:
Pros:
Cons:
Verdict: REJECTED - Security concerns, not designed for this use case.
Alternative 4: Python/JavaScript Expression Evaluation
Description: Allow arbitrary Python or JavaScript code execution.
Example:
Pros:
Cons:
Verdict: REJECTED - Unacceptable security risk.
Alternative 5: Keep Current System, Document Limitations
Description: Do nothing, document workarounds.
Workarounds:
default_action='ignore'with multiple process filtersPros:
Cons:
Verdict: REJECTED - Doesn't address fundamental architectural limitations.
Why CEL is the Right Choice
Industry Adoption:
Language Features:
Developer Experience:
Python Library:
Migration Strategy
Phase 1: Add CEL Expression Support (Non-Breaking)
Goal: Add CEL expression support to rules while maintaining backward compatibility with existing filters.
Changes:
Processing logic:
Impact:
Manual Migration:
Convert existing notification filters to CEL expressions on rules.
Phase 2: Remove Old System (Breaking)
Goal: Remove deprecated notification_filters table after all filters migrated to CEL.
Prerequisites:
Changes:
Concrete Examples
Example 1: GitHub Workflow Status
Scenario: Update build and deploy facts from single workflow_run webhook.
Webhook payload:
{ "action": "completed", "workflow_run": { "name": "CI/CD Pipeline", "head_branch": "main", "conclusion": "success", "jobs": [ {"name": "build", "conclusion": "success"}, {"name": "deploy", "conclusion": "success"} ] } }Configuration:
Result: Single webhook conditionally updates both facts based on job presence and branch.
Example 2: Ignore Non-Production Deploys
Scenario: Only update deploy facts for production deployments, ignore staging/dev.
Configuration:
Example 3: Complex Multi-Condition Routing
Scenario: Update different facts based on complex conditions:
test_coverage: Any test workflow on main, completeddeploy_status: Only deploy workflows to production, completedbuild_time: Any build on any branch (for metrics)Configuration:
Example 4: Common Preconditions in Multiple Rules
Scenario: Common precondition (must be completed, not draft) + specific routing per fact.
Configuration:
Result: Each rule includes common preconditions (completed, not draft) combined with rule-specific routing logic.
Trade-offs and Risks
Trade-offs
Risks
Risk 1: CEL Expression Errors
Risk 2: Performance Degradation
Risk 3: Security - Expression Injection
References
api/imbi/endpoints/integrations/notifications/processing.pyBeta Was this translation helpful? Give feedback.
All reactions