Skip to content

refactor: wire backfill pipeline and remove per-location Check documents#3313

Closed
gorkem-bwl wants to merge 1 commit intofeat/globalping-check-servicesfrom
feat/globalping-backfill-pipeline
Closed

refactor: wire backfill pipeline and remove per-location Check documents#3313
gorkem-bwl wants to merge 1 commit intofeat/globalping-check-servicesfrom
feat/globalping-backfill-pipeline

Conversation

@gorkem-bwl
Copy link
Contributor

Summary

  • Switch from creating separate Check documents per Globalping location to backfilling locationResults into the same Check document as the local check
  • Remove location from CheckMetadata, MonitorStatusResponse, metadata schema, location index, buildCheck(), toEntity(), and toDocument()
  • Change GlobalpingService return type to GlobalpingCheckResult[] (simplified shape without monitorId/teamId/type)
  • Rewrite SuperSimpleQueueHelper: after local check is buffered, fire Globalping checks and backfill results into the same document (in-buffer mutation if still buffered, DB updateOne if already flushed)
  • Rewrite findLocationChecksByMonitorId to $unwind locationResults instead of querying by metadata.location
  • Remove metadata.location filter from findUptimeDateRangeChecks

Test plan

  • npm run build in /server passes with zero errors
  • Monitor without Globalping: checks work exactly as before, no locationResults field
  • Monitor with Globalping enabled: local check writes immediately, then locationResults array gets backfilled after 5-15s
  • Details page location tabs show per-location charts from locationResults via $unwind
  • Backfill works both when check is still in buffer (mutated in-place) and when already flushed (DB updateOne)
  • Only local check data drives monitor status — locationResults has no effect on status pipeline

@gorkem-bwl gorkem-bwl force-pushed the feat/globalping-backfill-pipeline branch from 227141c to 0671981 Compare February 20, 2026 05:44
@gorkem-bwl gorkem-bwl force-pushed the feat/globalping-check-services branch 2 times, most recently from 4e7e341 to 325537e Compare February 20, 2026 06:15
@gorkem-bwl gorkem-bwl force-pushed the feat/globalping-backfill-pipeline branch 2 times, most recently from 9052c4f to de5adf5 Compare February 20, 2026 06:42
@gorkem-bwl gorkem-bwl force-pushed the feat/globalping-check-services branch from 325537e to 3b858de Compare February 20, 2026 06:42
Copy link
Member

@Br0wnHammer Br0wnHammer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my comments. @gorkem-bwl

$match: {
"metadata.monitorId": monitorObjectId,
createdAt: { $gte: startDate, $lte: endDate },
locationResults: { $exists: true, $not: { $size: 0 } },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The $exists: true + $not: { $size: 0 } combo can behave unexpectedly if locationResults is null (it exists but isn't an array, so $size won't work as intended). Safer to use:
locationResults: { $type: "array", $ne: [] }
This guarantees we're working with a non-empty array in all cases.

locationResults: { $exists: true, $not: { $size: 0 } },
},
},
{ $unwind: "$locationResults" },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perf: Since we're now doing $unwind on locationResults, it's important that the $match stage narrows the dataset as much as possible before that expansion happens.

const locCheck = this.checkService.buildCheck(locStatus);
if (locCheck) {
this.buffer.addToBuffer(locCheck);
.then((locationResults) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dual write path here (buffer mutation vs DB update) depends on whether the buffer has flushed by the time runChecks resolves. Can we add a comment explaining this is intentionally fire-and-forget? Also could a flush mid-callback cause a lost write?

const locCheck = this.checkService.buildCheck(locStatus);
if (locCheck) {
this.buffer.addToBuffer(locCheck);
.then((locationResults) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This .then() chain doesn't have a .catch(). If runChecks or updateLocationResults rejects, the error gets silently swallowed. Since this is fire-and-forget, we won't notice failures unless we explicitly log them.

## Changes
- Remove location field from CheckMetadata, MonitorStatusResponse, Check model metadata schema, and location index
- Remove location from checkService.buildCheck() and MongoChecksRepository toDocument/mapMetadata
- Rewrite SuperSimpleQueueHelper Step 2b: instead of creating separate Check documents per Globalping location, backfill locationResults into the same Check document after the local check is buffered
- Rewrite findLocationChecksByMonitorId to use $unwind on locationResults subdocument array instead of querying by metadata.location
- Remove metadata.location filter from findUptimeDateRangeChecks
- Pass locationResults through in toEntity

## Benefits
Single Check document per monitoring cycle now contains both local results
(top-level fields) and per-location Globalping results (locationResults array).
This eliminates document proliferation and simplifies queries.
@gorkem-bwl gorkem-bwl force-pushed the feat/globalping-backfill-pipeline branch from de5adf5 to a23e4de Compare February 20, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants