Skip to content

feat: add new space-diff-v2 table#592

Closed
BravoNatalie wants to merge 3 commits intomainfrom
feat/new-space-diff
Closed

feat: add new space-diff-v2 table#592
BravoNatalie wants to merge 3 commits intomainfrom
feat/new-space-diff

Conversation

@BravoNatalie
Copy link
Contributor

@BravoNatalie BravoNatalie commented Jan 15, 2026

According to this RFC#78 (related issue project-tracking#307)

This PR introduces a new space-diff table that uses cause as the sort key. The goal is to prevent duplicate diffs caused by past bugs (e.g. failed uploads), which led to inflated usage, slower queries, and “ghost” usage for spaces that should be empty after deletion.

Changes:

  1. Create a new space-diff-v2 table with:
    • cause as the sort key
    • A GSI for timestamp-based queries
  2. Enable dual writes:
    • On each diff event, write to both the existing table and the new table
    • Keep all readers (usage, reporting, billing) pointed at the existing table throughout January

For Future reference:
3. Cut over in February:

  • Switch usage reporting and billing reads to the new table
  • Keep the existing table as read-only historical storage

@seed-deploy
Copy link

seed-deploy bot commented Jan 15, 2026

View stack outputs
  • pr592-warm-upload-api-BillingDbStack

    Name Value
    customerTableName pr592-warm-upload-api-customer
    egressTrafficTableName pr592-warm-upload-api-egress-traffic-events
    spaceDiffArchiveTableName pr592-warm-upload-api-space-diff-archive
    spaceDiffTableName pr592-warm-upload-api-space-diff
    spaceDiffV2TableName pr592-warm-upload-api-space-diff-v2
    spaceSnapshotTableName pr592-warm-upload-api-space-snapshot
    usageTable pr592-warm-upload-api-usage
  • pr592-warm-upload-api-BillingStack

    Name Value
    ApiEndpoint https://ftdig32hm4.execute-api.us-east-2.amazonaws.com
    billingCronHandlerURL https://x7z6jkdf2d2zhzpo2yv27eczpi0znypz.lambda-url.us-east-2.on.aws/
    CustomDomain https://pr592.billing.warm.storacha.network
    EgressTrafficQueueURL https://sqs.us-east-2.amazonaws.com/505595374361/pr592-warm-upload-api-egress-traffic-queue
  • pr592-warm-upload-api-CarparkStack

    Name Value
    BucketName pr592-warm-upload-api-carpark-0
    Region us-east-2
  • pr592-warm-upload-api-RoundaboutStack

    Name Value
    ApiEndpoint https://staging.roundabout.web3.storage/
    CustomDomain Using ROUNDABOUT_API_URL - no custom domain
  • pr592-warm-upload-api-UcanInvocationStack

    Name Value
    agentIndexBucketName pr592-warm-upload-api-invocation-store-0
    agentMessageBucketName pr592-warm-upload-api-workflow-store-0
  • pr592-warm-upload-api-UploadApiStack

    Name Value
    ApiEndpoints ["https://6ev8rmcayl.execute-api.us-east-2.amazonaws.com"]
    CustomDomains ["https://pr592.up.warm.storacha.network"]
  • pr592-warm-upload-api-BusStack

  • pr592-warm-upload-api-FilecoinStack

  • pr592-warm-upload-api-UcanFirehoseStack

  • pr592-warm-upload-api-UploadDbStack

@seed-deploy seed-deploy bot temporarily deployed to pr592 January 16, 2026 14:16 Inactive
@seed-deploy seed-deploy bot temporarily deployed to pr592 January 16, 2026 14:16 Inactive
@BravoNatalie BravoNatalie linked an issue Jan 26, 2026 that may be closed by this pull request
Copy link
Member

@hannahhoward hannahhoward left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand why this can't be done with a GSI on cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redesign space diff table

2 participants