Skip to content

Changes to run queries(not changestreams) for data processing #6390

Merged
Cookiezaurs merged 28 commits intonewarchitecturefrom
feature/ingestion
Aug 15, 2025
Merged

Changes to run queries(not changestreams) for data processing #6390
Cookiezaurs merged 28 commits intonewarchitecturefrom
feature/ingestion

Conversation

@Cookiezaurs
Copy link
Contributor

No description provided.

@Cookiezaurs Cookiezaurs changed the title removed debug info Changes to run queries(not changestreams) for data processing Aug 7, 2025
Copy link
Member

@kanwarujjaval kanwarujjaval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR transitions from change stream-based data processing to query-based data processing for aggregation. It introduces new batch readers that process data in intervals rather than relying on real-time change streams for better performance and reliability.

  • Replaces changeStreamReader with dataBatchReader for periodic data aggregation
  • Updates aggregation plugins to use manual write batching instead of automatic batching
  • Implements new query infrastructure for handling segmented event data retrieval

Reviewed Changes

Copilot reviewed 21 out of 24 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
test/3.api.write/5.write.app.events.js Updates test validations to remove version/country metadata from expected results
plugins/times-of-day/api/aggregator.js Converts from change streams to batch reading with timezone-aware processing
plugins/server-stats/api/aggregator.js Migrates to batch processing and adds event type filtering
plugins/crashes/api/aggregator.js Removes debug logging statements
api/parts/queries/events.js Adds new segmented event data aggregation queries
api/parts/data/dataBatchReader.js New batch reader implementation for interval-based processing
api/parts/data/batcher.js Enhances write batcher with manual flush support
api/aggregator/processing.js Core aggregation logic using new batch readers
api/aggregator.js Restructures to use new query runner and manual batching

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@Cookiezaurs Cookiezaurs merged commit 62cf514 into newarchitecture Aug 15, 2025
1 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants