Skip to content

Feature/optimize publishing#125

Merged
rafmevis merged 4 commits intomainfrom
feature/optimize-publishing
Mar 6, 2026
Merged

Feature/optimize publishing#125
rafmevis merged 4 commits intomainfrom
feature/optimize-publishing

Conversation

@rafmevis
Copy link
Collaborator

@rafmevis rafmevis commented Mar 6, 2026

Summary by cubic

Switches publishing to a “dirty passport” pipeline: write paths mark affected passports dirty and a new projector builds snapshots in the background or on demand. Adds historical version compression and updates publish flows and jobs to flip status, mark dirty, project, and revalidate caches.

  • New Features

    • Added product_passports.dirty with index; mutations mark passports dirty (catalog edits, variant create/overrides, publish actions, integrations sync, bulk commit-to-production).
    • New projector: projectSinglePassport and projectDirtyPassports (+ hourly scheduled job) to materialize snapshots, clear dirty flags, and revalidate passports/barcodes; public DPP reads use on-demand projection.
    • Historical version compression: superseded versions are zstd-compressed (daily job); current version stays JSONB with transparent reads.
    • Publish router refactor: flip product status, ensure passport exists, mark dirty, project, then revalidate UPIDs and barcodes.
    • Removed catalog fan-out job and legacy per-variant publish; centralized snapshot logic in queries/products/projector. Integration sync returns affected product IDs; jobs mark those passports dirty.
  • Migration

    • Run DB migrations (adds product_passports.dirty; makes first_published_at nullable; adds compressed_snapshot, compressed_at, and nullable data_snapshot on product_passport_versions with presence check).
    • Deploy scheduled jobs: passportProjector (hourly) and compressPassportVersions (daily). Override cadence with PASSPORT_PROJECTOR_CRON and PASSPORT_VERSION_COMPRESSION_CRON if needed.

Written for commit bd802c3. Summary will update on new commits.

@supabase
Copy link

supabase bot commented Mar 6, 2026

Updates to Preview Branch (feature/optimize-publishing) ↗︎

Deployments Status Updated
Database Fri, 06 Mar 2026 13:01:14 UTC
Services Fri, 06 Mar 2026 13:01:14 UTC
APIs Fri, 06 Mar 2026 13:01:14 UTC

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

Tasks Status Updated
Configurations Fri, 06 Mar 2026 13:01:20 UTC
Migrations Fri, 06 Mar 2026 13:01:24 UTC
Seeding Fri, 06 Mar 2026 13:01:24 UTC
Edge Functions Fri, 06 Mar 2026 13:01:26 UTC

View logs for this Workflow Run ↗︎.
Learn more about Supabase for Git ↗︎.

Copy link

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6 issues found across 38 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/db/src/schema/products/product-passports.ts">

<violation number="1" location="packages/db/src/schema/products/product-passports.ts:131">
P2: The partial index already fixes `dirty = true`, so indexing `dirty` as a key is redundant and adds avoidable index/write overhead.</violation>
</file>

<file name="apps/api/src/trpc/routers/dpp-public/index.ts">

<violation number="1" location="apps/api/src/trpc/routers/dpp-public/index.ts:180">
P1: Inline projection in the public route can race under concurrent requests and throw unique-version errors, causing public 500s.</violation>
</file>

<file name="apps/api/src/trpc/routers/products/publish.ts">

<violation number="1" location="apps/api/src/trpc/routers/products/publish.ts:279">
P2: `totalProductsPublished` reports input count rather than actual updates. If some productIds don't exist or belong to another brand, the response will incorrectly claim they were published. Consider returning the actual affected row count from the UPDATE operation, or validating productIds exist beforehand.</violation>
</file>

<file name="apps/api/supabase/migrations/20260306111521_white_iron_lad.sql">

<violation number="1" location="apps/api/supabase/migrations/20260306111521_white_iron_lad.sql:1">
P1: Dropping `data_snapshot` NOT NULL without an alternative CHECK constraint allows snapshot-less versions to be stored.</violation>
</file>

<file name="packages/db/src/queries/dpp/public.ts">

<violation number="1" location="packages/db/src/queries/dpp/public.ts:339">
P2: `dirty` is hardcoded to `false` in `getPublicDppVersion`, which can return incorrect passport state.</violation>
</file>

<file name="apps/api/src/trpc/routers/products/index.ts">

<violation number="1" location="apps/api/src/trpc/routers/products/index.ts:144">
P3: Remove unreachable camelCase checks in bulk snapshot-change detection to avoid dead branches.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

@greptile-apps
Copy link

greptile-apps bot commented Mar 6, 2026

Greptile Summary

This PR replaces the previous "publish-on-write" model with a dirty-flag + deferred projection architecture. Rather than materializing passport snapshots synchronously on every product edit or via a delayed Trigger.dev fan-out job, mutations now mark affected published passports as dirty=true. Snapshots are then projected on-demand at public read time (inline via resolvePublicPassportResponse) or lazily by a new hourly scheduled passportProjector job. A companion compressPassportVersions job compresses superseded historical snapshots into zstd bytea storage to reclaim space.

Key changes:

  • Schema: product_passports.dirty boolean column added (partial index on (brand_id, dirty) WHERE dirty = true); first_published_at made nullable (set only on first actual materialization); product_passport_versions.data_snapshot made nullable; compressed_snapshot bytea + compressed_at columns added.
  • Publish routers: publish.variant and publish.product mark dirty then project inline; publish.bulk marks dirty and delegates to background job.
  • Catalog router: Drops the 45-second delayed Trigger.dev catalog-fan-out task; catalog mutations now call markPassportsDirtyByProductIds synchronously.
  • Public DPP router: On every read, if the passport is dirty and the product is published, it projects inline before serving — eliminating the race window between a catalog edit and the next background fan-out run.
  • Integration sync: syncProducts now accumulates affectedProductIds and the sync job marks their passports dirty after the run.
  • Removed: catalog-fan-out Trigger.dev task and all related enqueueCatalogFanOut call sites.

Two logic bugs identified:

  1. Silent projection failure on public readsresolvePublicPassportResponse does not check the error field returned by projectSinglePassport. If projection fails, the snapshot remains null and users silently receive 404s for published products.
  2. SQL NULL comparison in compression querybatchCompressSupersededVersions uses ne() against a nullable currentVersionId, which evaluates to NULL in SQL, permanently skipping compression for passports with a null current-version pointer.

Confidence Score: 3/5

  • Two logic bugs need to be addressed: silent projection failure hiding published products and SQL NULL comparison breaking compression for certain passports.
  • The dirty-flag + deferred-projection architecture is well-conceived and the schema changes are correct. However, two real runtime bugs exist: (1) resolvePublicPassportResponse silently drops projection errors causing published products to appear as 404s, and (2) batchCompressSupersededVersions uses ne() against a nullable column which evaluates to NULL in SQL, permanently skipping compression for passports with a null current-version pointer. Both impact critical paths and should be fixed before shipping.
  • apps/api/src/trpc/routers/dpp-public/index.ts (lines 175-189: add error check after projection), packages/db/src/queries/products/dpp-versions.ts (line 390: use OR logic to handle NULL currentVersionId)

Sequence Diagram

sequenceDiagram
    participant Client
    participant CatalogRouter
    participant ProductsRouter
    participant PublishRouter
    participant DB_MarkDirty as DB (mark-dirty)
    participant DB_Projector as DB (projector)
    participant PublicDPP as Public DPP Router
    participant ProjectorJob as Projector Job (hourly)

    Note over CatalogRouter,DB_MarkDirty: Catalog mutation path (inline dirty marking)
    Client->>CatalogRouter: update/delete manufacturer/material/etc.
    CatalogRouter->>DB_MarkDirty: markPassportsDirtyByProductIds()
    DB_MarkDirty-->>CatalogRouter: {marked, passportIds}
    CatalogRouter-->>Client: success

    Note over PublishRouter,DB_Projector: Single/product publish (inline projection)
    Client->>PublishRouter: publish.variant / publish.product
    PublishRouter->>DB_MarkDirty: markPassportDirty() / markPassportsDirtyByProductIds()
    PublishRouter->>DB_Projector: projectSinglePassport() / projectDirtyPassports()
    DB_Projector-->>PublishRouter: {snapshot, version, upids, barcodes}
    PublishRouter-->>Client: {success, version, passportsProjected}

    Note over PublishRouter,DB_MarkDirty: Bulk publish (deferred projection)
    Client->>PublishRouter: publish.bulk
    PublishRouter->>DB_MarkDirty: markPassportsDirtyByProductIds()
    PublishRouter-->>Client: {success, productsMarkedDirty}

    Note over PublicDPP,DB_Projector: Public read with inline dirty projection
    Client->>PublicDPP: getByPassportUpid / getByBarcode
    PublicDPP->>DB_Projector: getPublicDppByUpid()
    alt passport.dirty == true AND productStatus == published
        PublicDPP->>DB_Projector: projectSinglePassport()
        PublicDPP->>DB_Projector: getPublicDppByUpid() (re-fetch)
    end
    PublicDPP-->>Client: DPP snapshot + theme

    Note over ProjectorJob,DB_Projector: Background sweep (hourly)
    ProjectorJob->>DB_Projector: projectDirtyPassportsAllBrands()
    DB_Projector-->>ProjectorJob: {brands[], upids, barcodes}
    ProjectorJob->>ProjectorJob: revalidatePassports() + revalidateBarcodes() per brand
Loading

Last reviewed commit: 5732504

@github-actions
Copy link

github-actions bot commented Mar 6, 2026

🚀 Preview Deployment Summary

App: https://avelero-9jczi80of-avelero.vercel.app
API: https://pr-125-avelero-api.fly.dev
Web: https://avelero-website-1zszskx3s-avelero.vercel.app
Admin: https://avelero-admin-lb1fusbm0-avelero.vercel.app
DPP: https://avelero-passport-5v2zmc28y-avelero.vercel.app
Jobs: Deployed to Trigger.dev

Preview environments will auto-delete when PR closes.

Copy link

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 10 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/api/src/trpc/routers/products/publish.ts">

<violation number="1" location="apps/api/src/trpc/routers/products/publish.ts:288">
P2: `totalFailed` can be overcounted when `productIds` contains duplicates, producing incorrect bulk publish results.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

@github-actions
Copy link

github-actions bot commented Mar 6, 2026

🚀 Preview Deployment Summary

App: https://avelero-59winoy5n-avelero.vercel.app
API: https://pr-125-avelero-api.fly.dev
Web: https://avelero-website-7xr0ci7kz-avelero.vercel.app
Admin: https://avelero-admin-780d5xw19-avelero.vercel.app
DPP: https://avelero-passport-45ct9c3oe-avelero.vercel.app
Jobs: Deployed to Trigger.dev

Preview environments will auto-delete when PR closes.

@github-actions
Copy link

github-actions bot commented Mar 6, 2026

🚀 Preview Deployment Summary

App: https://avelero-j89jp75ml-avelero.vercel.app
API: https://pr-125-avelero-api.fly.dev
Web: https://avelero-website-n0yncor2h-avelero.vercel.app
Admin: https://avelero-admin-j3hdddvdi-avelero.vercel.app
DPP: https://avelero-passport-d5xue7f9g-avelero.vercel.app
Jobs: Deployed to Trigger.dev

Preview environments will auto-delete when PR closes.

@rafmevis rafmevis merged commit 3ed0f5e into main Mar 6, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants