Alerting System Design

[ygrishajev]

Here I'd like to discuss

1. the state of BlockSpy service which we'd like to migrate into public console area and use to implement alerting/notifications
2. outline potential issues with the current design
3. suggest a roadmap to incrementally move towards a better design
4. decide which steps in the roadmap we'd like to undertake for an MVP

Relevant issues are created under Alerts & Notifications (Blockspy integration) milestone

[ygrishajev]

BlockSpy System Overview (Current Design with Issues Identified)

The BlockSpy service processes Akash blockchain events and notifies users when specific conditions are met. It consists of multiple components, each playing a role in processing blockchain data and delivering notifications. However, the current design has issues related to separation of concerns and efficiency, which need to be addressed.

---

1. User Alert Setup

Flow:

1. The User creates an alert through the Web interface.
2. The Web sends the alert configuration to the API.
3. The API stores the alert configuration in BlockSpyDB.
4. The API creates a monitored value in IndexerDB (!).

Purpose:

• This phase sets up custom alerts that the system will later evaluate against blockchain data.
• Monitored values are system-tracked values derived from user alert conditions.

Issues:

• API is storing monitored values in IndexerDB, which is meant to hold only raw blockchain data.
• Monitored values are a BlockSpy-specific entity and should exist only in BlockSpyDB, not IndexerDB.
• This causes tight coupling between BlockSpy and Indexer, making IndexerDB contain non-blockchain-related data.

---

2. Blockchain Indexing

Flow:

1. The Blockchain generates new block messages.
2. The Indexer reads these messages and stores them in IndexerDB.
3. The Indexer updates the monitored value in IndexerDB (!).

Purpose:

• This phase ensures that all blockchain events are indexed for quick retrieval.
• The IndexerDB serves as a queryable store of blockchain state, which BlockSpy can later use for processing alerts.

Issues:

• Indexer is updating monitored values that were originally created by the API.
• Monitored values are user-specific and belong to BlockSpyDB, so they should not be stored or modified in IndexerDB.
• This blurs the responsibility of Indexer, which should only index blockchain data, not manage user-specific entities.

---

3. Alert Processing and Notifications

Flow:

1. The Notifications service fetches new messages from IndexerDB (!).
2. It retrieves the corresponding alert conditions from BlockSpyDB.
3. It evaluates alerts against the fetched blockchain messages.
4. If conditions are met, it triggers an email notification via Novu.
5. Novu sends an email alert to the User.
6. The Notifications service marks the processed messages in IndexerDB (!).

Issues:

• Notifications is marking messages as processed in IndexerDB, but IndexerDB should only store blockchain data.
• Processing state should be tracked in BlockSpyDB, since it's relevant to BlockSpy's logic, not Indexer.
• Notifications directly fetches raw blockchain messages from IndexerDB, leading to tight coupling.
  • A better approach would be structured queries via an API instead of direct DB access.
• Duplicate Notifications Issue:
  • The current design stores a single processed flag per message, meaning if a message has multiple triggers and one fails, it will be resent to all users on reprocessing, causing duplicates.

---

4. Inefficient Matching Logic (Quadratic Complexity)

Flow:

1. Notifications fetch all messages from IndexerDB.
2. It then compares each message against all user triggers stored in BlockSpyDB.
3. This results in O(N * M) complexity, where:
   • N = number of messages,
   • M = number of user triggers.

Issues:

• This brute-force matching does not scale well as the number of messages and triggers grows.
• The process becomes extremely inefficient, consuming unnecessary compute power.
• The system wastes resources checking irrelevant messages against irrelevant triggers.

---

Summary of Issues Identified

Issue	Why It’s a Problem?
API creates monitored values in IndexerDB	IndexerDB should only store blockchain data, not user-defined entities.
Indexer updates monitored values in IndexerDB	Monitored values are user-specific and should be managed only by BlockSpyDB.
Notifications fetches new messages directly from IndexerDB	Increases coupling and inefficiently processes messages.
Notifications marks processed messages in IndexerDB	Processing state should be tracked in BlockSpyDB, not IndexerDB.
Matching logic has quadratic complexity (O(N * M))	Becomes inefficient at scale, wasting compute resources.
Duplicate notifications issue	If a message has multiple triggers and one fails, it will be resent to all users on reprocessing.

[ygrishajev]

Option 1: Migrating BlockSpy to Console (MVP Approach)

The first step in the migration is to reuse as much of the existing implementation as possible to quickly deliver an MVP and start gathering user feedback. This will help direct further improvements while ensuring minimal disruption.

---

Key Migration Changes

1. Auth Reconfiguration

✅ What Needs to Change?

• BlockSpy and Console both use Auth0, but BlockSpy must now use the Console’s Auth0 configuration.
• Update BlockSpy API to validate access tokens using the Console’s Auth0 tenant instead of its standalone configuration.

---

2. Removing userSettings Table

✅ What Needs to Change?

• userSettings in BlockSpy serves as a central user entity but is no longer needed.
• Instead, the user ID from Console userSettings should be used directly to associate all entities.

✅ Migration Approach:

• Modify queries to remove foreign key constraints related to userSettings.
• Ensure the user ID is retrieved from auth middleware and used without DB constraints.
• If necessary, the existing Console user DB/table can be queried for verification.

---

3. Ensuring DB Migration on Service Startup

✅ What Needs to Change?

• A new database for the notifications-api and notifications-processor services should be created.
• Tables from BlockSpy must be migrated to Console’s infrastructure.
• On service startup, ensure:
  • New schema updates are applied.
  • Old references (e.g., userSettings) are cleaned up.

---

4. Implementing Authorization with Permissions/Abilities

✅ What Needs to Change?

• Current Behavior: BlockSpy assumes that users pay for alerts and can subscribe to any blockchain event.
• New Behavior: In Console, users do not pay for alerts—so they should only be able to subscribe to their own resource alerts.
• Implement permissions-based access control similar to Console API.

✅ Migration Approach:

• Define permissions/abilities based on user ownership of resources.
• Ensure notifications-api follows the Console’s RBAC (Role-Based Access Control) model.
• Remove any logic that assumes a user can subscribe to any event.

---

5. Removing Stripe from the Codebase

✅ What Needs to Change?

• Stripe was used in BlockSpy for managing paid subscriptions.
• Since alerts will now be free for users in Console, Stripe must be fully removed.

✅ Migration Approach:

• Identify and remove all Stripe-related logic in:
  • Billing-related database tables.
  • Subscription validation logic.
  • Any UI or API references to payments.

---

New Services: notifications-api and notifications-processor

The migrated BlockSpy service will be split into two separate Console applications:

1. notifications-api

   • Handles user requests, alert creation, and subscription management.
   • Serves as the public-facing API for notifications.
   • Ensures RBAC-based access control aligned with Console permissions.

2. notifications-processor

   • Listens for new blockchain messages from IndexerDB.
   • Matches messages against user-defined alerts.
   • Sends notifications via Novu.
   • Tracks processing status to prevent duplicates.

This separation ensures better scalability, cleaner architecture, and proper division of responsibilities.

[baktun14]

Thanks for putting up a detailed approach with a sequence diagram!

Removing userSettings Table

We just need to make sure Blockspy keeps running as-is until we have it fully migrated because we're actively using the alerts.

New Behavior: In Console, users do not pay for alerts—so they should only be able to subscribe to their own resource alerts.

Just to confirm, this is what we want right? @anilmurty

My only concern with option 1 is to alter the current state of Blockspy, mainly the db because the code can change in a separate branch, while we do the migration.

[ygrishajev]

I didn't mean we're going to change current blockspy. The main thing is we should care about continuing its maintenance.
All infrastructure would be created anew. I'm referring to the updated codebase in console.

[baktun14]

Ok yea makes sense

[ygrishajev]

Alternative Approach for Notifications Service

This is an alternative solution to address the limitations of the existing approach. The key motivation is to improve scalability, maintainability, and flexibility.

Notifications Service Architecture

Overview

The Notifications Service is responsible for processing user-defined and system-generated notification conditions, matching them against blockchain messages and monitored values, and dispatching notifications via configured channels.

Event Flow

1. User Configuration: Users define notification conditions and delivery channels via the Notifications API.
2. Message Processing:
   • The Indexer publishes a message.created event.
   • The Message Matcher processes the event to check if it matches user-defined conditions.
   • If a match is found, a notification.send command is published.
3. Monitored Value Processing:
   • The Monitored Value Processor calculates and stores monitored values based on incoming messages.
   • It emits a monitored-value.created event when a monitored value is updated.
   • The Monitored Value Matcher evaluates stored conditions against monitored values.
   • If a match is found, a notification.send command is published.
4. Notification Dispatch:
   • The Notification Dispatcher subscribes to notification.send and sends notifications via the appropriate channel (e.g., email via Novu).

Alternative Approach

⚠️ Monitored Value Creation Can Potentially Be Skipped
Instead of a separate monitored-value.created event, the system can calculate monitored values and match them in a single handler when processing message.created events. This would reduce the number of event-handling steps and simplify the architecture at the cost of potential re-computation.

Scalability & Deployment

• Modular Execution: The same codebase should support multiple roles via environment variables:
  • MODULE=rest-api → Runs only the REST API.
  • MODULE=worker EVENT=message.created → Runs only the worker for message events.
  • MODULE=worker EVENT=monitored-value.created → Runs only the worker for monitored value events.
  • MODULE=worker EVENT=notification.send → Runs only the notification dispatcher.
• Decoupled Event Bus: The system must allow swapping event backends without modifying worker logic.

Acceptance Criteria

• User-defined and system-generated conditions trigger notifications.
• Workers process events asynchronously, with clear separation of concerns.
• Notifications are sent via multiple configurable channels.
• Monitored values can be calculated and matched either as separate steps or within a single event handler.
• Minimal infrastructure complexity by using PgBoss for event queuing.

Summary

By leveraging this approach, we achieve a modular and scalable notifications system that remains flexible and extensible.

• Workers operate independently, ensuring that each component can scale based on load.
• Event-driven architecture decouples components, allowing seamless integration of new event sources.
• Swappable event bus abstraction enables future adaptability without major refactoring.
• Separation of concerns ensures each service handles its own responsibility without cross-service dependencies.
• Eliminates misuse of Indexer DB, ensuring it remains focused on blockchain indexing, while notifications manage their own data.

This ensures a robust, maintainable, and future-proof solution, addressing existing pain points while keeping infrastructure complexity minimal. 🚀

[ygrishajev]

to reduce complexity of this approach we can as well make this not an app but rather a properly isolated module. This way we'd need to ensure it does not use any wallet signing logic to avoid more concurrency issues.