Skip to content

Implement two-phase project loading#655

Open
ulrikandersen wants to merge 12 commits intodevelopfrom
feature/two-phase-project-loading
Open

Implement two-phase project loading#655
ulrikandersen wants to merge 12 commits intodevelopfrom
feature/two-phase-project-loading

Conversation

@ulrikandersen
Copy link
Contributor

@ulrikandersen ulrikandersen commented Jan 15, 2026

Description

Split project loading into two phases for faster load and more up-to-date data:

  1. Project List - Fast, lightweight query for sidebar (name, owner, config only)
  2. Project Details - On-demand loading when viewing a project (branches, tags, specs)

Project list and project details now load in parallel and independently. This improves performance on both the "cold start" of Framna Docs and when deeplinking directly to a spec.

Project list is still cached for 30 days, and refreshed in the background after the initial load and when Framna Docs regains focus.

With this change the user can do a simple refresh of the page and get the latest project data (versions, specs) right away instead of waiting for a background refresh, which is currently the case.

Changes:

  • Add ProjectSummary type for lightweight project list
  • Add GET /api/projects endpoint for fast list query
  • Add GET /api/projects/[owner]/[repo] endpoint for on-demand details
  • Add ProjectListContext and ProjectDetailsContext for state management
  • Update sidebar to use lightweight project list
  • Update project view to fetch details on-demand
  • Remove refresh spinner from header (loading projects is now much faster)

Motivation and Context

The current implementation fetches everything upfront: all repos, all branches (100), all tags (100), all file trees, and all PR data. This creates massive GraphQL payloads even though users typically only view one project at a time.

Project refresh takes ~15s with ~30 projects. Most users don't have more than 5, but still the loading of the entire project tree is inefficient for keeping the data fresh.

This change reduces initial load time by only fetching the minimal data needed for the sidebar, then loading full project details on-demand when a user selects a project. This also ensures the user sees up to date versions (GitHub branches) right away.

Screenshots (if appropriate):

Skaermoptagelse.2026-01-16.kl.08.14.00.mov

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Split project loading into fast list query (ProjectSummary) for sidebar
and on-demand details query (full Project) when viewing a project.

Changes:
- Add ProjectSummary type for lightweight project list
- Add IProjectListDataSource and GitHubProjectListDataSource
- Add IProjectDetailsDataSource and GitHubProjectDetailsDataSource
- Add GET /api/projects and GET /api/projects/[owner]/[repo] endpoints
- Add ProjectListContext and ProjectDetailsContext
- Update sidebar to use ProjectListContext
- Update project view to fetch details on-demand
- Remove deprecated refresh-projects endpoint
- Remove refresh spinner from header (loading is now fast)
- Remove unused caching infrastructure (CachingProjectDataSource,
  ProjectRepository, GitHubProjectDataSource, GitHubRepositoryDataSource,
  FilteringGitHubRepositoryDataSource)
Copilot AI review requested due to automatic review settings January 15, 2026 05:15
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request implements a two-phase project loading strategy to significantly improve initial load times by splitting project data fetching into a lightweight list view and on-demand detail loading.

Changes:

  • Introduced ProjectSummary type for lightweight project list display with new API endpoints (GET /api/projects and GET /api/projects/[owner]/[repo])
  • Replaced monolithic ProjectsContext with separate ProjectListContext and ProjectDetailsContext for independent state management
  • Removed deprecated caching infrastructure including ProjectRepository, CachingProjectDataSource, FilteringGitHubRepositoryDataSource, and the /api/refresh-projects endpoint (~1,700 lines of code)

Reviewed changes

Copilot reviewed 39 out of 40 changed files in this pull request and generated 13 comments.

Show a summary per file
File Description
src/features/projects/view/ProjectListContextProvider.tsx New context provider for managing lightweight project list state
src/features/projects/view/ProjectDetailsContextProvider.tsx New context provider for on-demand project details with caching
src/features/projects/data/GitHubProjectListDataSource.ts New data source for fetching minimal project metadata
src/features/projects/data/GitHubProjectDetailsDataSource.ts New data source for fetching complete project details including branches, tags, and specs
src/features/projects/data/useProjectSelection.ts Updated to work with on-demand project loading
src/app/(authed)/(project-doc)/[...slug]/page.tsx Modified to trigger project details fetch on page load
src/app/api/projects/route.ts New API endpoint for project list
src/app/api/projects/[owner]/[repo]/route.ts New API endpoint for individual project details
src/features/sidebar/view/internal/sidebar/Header.tsx Removed refresh spinner UI
src/composition.ts Updated dependency injection to use new data sources
jest.config.js Added tsx to module file extensions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…aSource

Coverage includes:
- Project list: filtering, deduplication, pagination, config parsing, image URLs
- Project details: versions, specifications, PRs, remote versions, default spec
Copilot AI review requested due to automatic review settings January 15, 2026 05:34
@ulrikandersen ulrikandersen force-pushed the feature/two-phase-project-loading branch 2 times, most recently from 61cec21 to e971c42 Compare January 15, 2026 05:36
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 44 out of 45 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings January 15, 2026 05:38
@ulrikandersen ulrikandersen force-pushed the feature/two-phase-project-loading branch from c07e278 to 2468e2a Compare January 15, 2026 05:40
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 42 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Add null check for defaultSpec in useProjectSelection to prevent
  potential crash if a version has no specifications
- Add explicit type annotations for 404 handling in fetchProject
- Add comment explaining why cache is in callback dependencies
Cache project list in Redis with 1 minute TTL to eliminate loading
skeleton on subsequent page loads. Returns cached data immediately
and refreshes in background.
- Memoize navigateToSelectionIfNeeded with useCallback to prevent
  unnecessary effect runs on every render
- Wrap projectNavigator in useMemo to stabilize dependencies
- Reset isLoadingRef in cleanup to prevent race conditions
- Restore console.info logging for decryption failures
Copilot AI review requested due to automatic review settings January 16, 2026 06:20
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/features/projects/domain/ProjectListRepository.ts:36

  • The cache TTL of 60 seconds (1 minute) is very short and may cause frequent re-fetching. Combined with the background refresh in CachingProjectListDataSource, this could lead to excessive GitHub API calls. Consider increasing the TTL to at least 5-10 minutes to balance freshness with API rate limits, or remove the TTL entirely if background refresh handles cache updates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings January 16, 2026 06:28
@ulrikandersen ulrikandersen force-pushed the feature/two-phase-project-loading branch from 36911db to c6512d6 Compare January 16, 2026 06:29
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Add refresh option to IProjectListDataSource and CachingProjectListDataSource
- API route passes ?refresh=true to bypass cache
- Remove background refresh (frontend controls refresh)
- TTL back to 30 days
- Frontend uses ?refresh=true on visibility/focus events
Remove fingerprint comparison entirely - it was preventing the UI from
updating when new project data was fetched.
Use owner/name from URL path directly for selection instead of requiring
full project details to be loaded first. This makes selection immediate.
Copilot AI review requested due to automatic review settings January 16, 2026 07:26
Single fetchProjects function used for both initial load and refresh.
Removes duplicate inline fetch logic.
@ulrikandersen ulrikandersen force-pushed the feature/two-phase-project-loading branch from d839270 to ab81447 Compare January 16, 2026 07:29
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants