Add real-time progress feedback to interview data export#678
Merged
Conversation
Defines the architecture for real-time progress feedback during interview data export, using SSE via Effect Stream/Queue and persistent toast notifications.
Clarify toast persistence requires explicit timeout: 0, detail generateOutputFiles Effect conversion, document eager total computation, and add updateExportTime/PostHog tracking to the client flow.
11-task plan covering: event types, Effect-based generateOutputFiles, pipeline queue integration, SSE route handler, toast loading variant, ExportProgressProvider, dialog refactor, and cleanup.
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR replaces the blocking interview export flow with an SSE-driven export pipeline that streams progress updates to the client, enabling persistent “export in progress” toasts and background exports while users navigate the dashboard.
Changes:
- Added an SSE route (
/api/export-interviews) that runs the export pipeline and streams stage/progress/complete/error events. - Refactored file generation into an Effect-based implementation that emits per-file progress updates.
- Extended the toast system with a
loadingvariant (spinner + cancel support) and introduced a dashboard-levelExportProgressProvider.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
app/api/export-interviews/route.ts |
New SSE endpoint that runs exportPipeline and streams events via Effect Queue/Stream |
lib/export/pipeline.ts |
Pipeline now accepts a progress queue and emits stage events between steps |
lib/network-exporters/formatters/session/generateOutputFiles.ts |
Converted generation step to Effect with progress events and concurrent execution |
lib/export/exportEvents.ts |
Defines export event protocol and SSE formatting helper |
components/ui/Toast.tsx |
Adds loading variant + spinner animation + cancel button support |
components/ExportProgressProvider.tsx |
New client provider that consumes SSE and updates persistent progress toasts |
app/dashboard/layout.tsx |
Wraps dashboard children with ExportProgressProvider |
app/dashboard/interviews/_components/ExportInterviewsDialog.tsx |
Switches export trigger to provider-based background export |
schemas/export.ts |
Adds request-body schema for the SSE route |
lib/export/__tests__/pipeline.test.ts |
Updates tests for new queue argument + basic stage event emission check |
components/ui/Toast.stories.tsx |
Adds Storybook demo for loading toast updates |
lib/interviewer/components/InterviewToast.tsx |
Adds loading arrow variant styling |
actions/interviews.ts |
Removes blocking exportInterviews server action |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Fix pipeline fiber getting interrupted when parent Effect completes by using forkDaemon instead of fork. Add tests for formatSSE utility and full pipeline stage event ordering.
Remove the dedicated loading variant from the toast system. Export progress toasts now use the default variant with a spinner rendered inline in the description content.
The interviews query was fetching full network JSON (all node/edge attributes) and full protocol data for every interview, causing 75MB+ downloads with 2000 interviews. Now uses select to fetch only the fields the table needs, and computes node/edge type counts server-side instead of sending raw network data to the client.
Move codebook lookups (node/edge names and colors) into the server-side network summary. Replace protocol.stages array with a computed stageCount number. The interview list query now sends only the minimal data the table needs — no more duplicated codebook or stage definitions per interview.
Replace Prisma findMany with $queryRaw to compute network summaries and stage counts entirely in PostgreSQL. The full network JSON and codebook are never transferred from DB to app server — PostgreSQL aggregates node/edge counts by type and looks up names/colors from the codebook JSON directly. Also bypasses Prisma client extensions that were parsing every network through NcNetworkSchema.parse() and every protocol through VersionedProtocolSchema.parse().
Thread icon prop through toast data so custom icons (like the export spinner) render in ToastItem. Use the Button component for the cancel action. Clear onCancel when transitioning to success state.
Wrap request.json() in try/catch in both route handlers to return 400 instead of 500 on malformed JSON. Remove unused exports flagged by knip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ExportProgressProvidercontext in the dashboard layout so exports run in the background — users can navigate freely while exportingloadingvariant to the toast system with spinner icon and cancel buttonTechnical Details
generateOutputFilesconverted to an Effect-returning function usingEffect.forEachwith unbounded concurrency andRef-based progress trackingEffect.Queueand emits stage/progress events between stepsQueue → Stream.fromQueue → Stream.map (SSE format) → Stream.toReadableStream → ResponseAbortControllerTest Plan
pnpm typecheck)pnpm lint)pnpm test) — 1842 passing