feat(api): replace api-notion-fetch with api-native fetch jobs

[luandro]

Summary

• API-native fetch-ready/fetch-all job model implemented in api-server/
• deterministic single-process fetch lock + timeout + lifecycle logs
• legacy api-notion-fetch.yml removed after smoke validation
• A1-A4, B1-B5, C1-C4, D1-D3, E1-E3, F1-F2, G1-G3, H1-H3

Testing

Commands run:

# Commit 1 tests
bunx vitest run api-server/fetch-job-runner.test.ts api-server/job-executor-fetch.test.ts api-server/endpoint-schema-validation.test.ts api-server/validation-schemas.test.ts api-server/job-tracker.test.ts

# Commit 2 tests  
bunx vitest run api-server/api-validate-workflow.test.ts api-server/github-actions-secret-handling.test.ts

# Commit 3 tests
bunx vitest run api-server/deployment-runbook.test.ts

# Final verification
bunx vitest run api-server/fetch-job-runner.test.ts api-server/job-executor-fetch.test.ts api-server/api-validate-workflow.test.ts api-server/deployment-runbook.test.ts api-server/github-actions-secret-handling.test.ts

All tests passed (92 tests total).

Scripts run:

• scripts/ci-validation/vps-fetch-smoke.sh - Created for VPS dry-run smoke validation

Reviewer notes

• reviewer subagent loop unavailable due agent-thread limit; compensated with targeted tests and manual gate checks

Residual risks

• recommend one live GitHub Actions run of api-validate.yml post-PR should merge to feat/notion-api-service not main

Greptile Summary

Replaces the GitHub Actions workflow api-notion-fetch.yml with API-native fetch-ready and fetch-all job types running inside the API server. This refactoring moves Notion fetch orchestration from CI workflows into the API service itself, providing deterministic single-process execution with proper timeout handling, git branch management, and status transitions.

Major changes:

• New fetch-job-runner.ts implements the core fetch-ready/fetch-all execution flow with timeout protection, exponential retry for Notion API calls, and automatic status transitions from "Ready to publish" to "Draft published"
• fetch-job-lock.ts provides in-memory lock with TTL to prevent concurrent fetch jobs
• content-repo.ts adds extensive git operations for content branch preparation, merge conflict handling, staging, committing, and push retry logic
• job-tracker.ts now tracks terminal state for fetch jobs and handles in-flight job recovery on server restart
• routes/jobs.ts integrates lock checking during job creation and returns simplified response schemas for fetch jobs
• Legacy api-notion-fetch.yml workflow deleted
• New api-validate.yml workflow added for CI smoke testing

Design highlights:

• Single-writer model for origin/content branch prevents race conditions
• Merge origin/main into content before every fetch to ensure consistency
• Status transitions only occur after content is successfully pushed to remote
• Timeout protection throughout with AbortSignal and withTimeout wrapper
• Comprehensive test coverage (92 tests) validates timeout, retry, and edge case behavior

Confidence Score: 4/5

• This PR is safe to merge with minor timeout-related issues already identified
• The implementation is well-tested with 92 passing tests and comprehensive coverage of edge cases. The architectural changes are sound, with proper git operations, lock management, and error handling. Two timeout-related issues were previously identified in fetch-job-runner.ts that should be addressed, but they don't block merging to the feature branch. The extensive test coverage and careful handling of git operations, retries, and status transitions demonstrate thorough engineering.
• Pay close attention to api-server/fetch-job-runner.ts for the two timeout-related issues already noted in previous comments

Important Files Changed

Filename	Overview
api-server/fetch-job-runner.ts	New fetch job runner with timeout handling, retry logic, and Notion status transitions - two timeout-related issues already noted in previous comments
api-server/fetch-job-lock.ts	Simple in-memory lock mechanism with TTL for single-process fetch job coordination
api-server/job-executor.ts	Integrates new fetch job runner, handles timeout and abort signals, manages fetch lock lifecycle
api-server/job-tracker.ts	Adds terminal state tracking for fetch jobs and handles in-flight job recovery on server restart
api-server/content-repo.ts	Extensive new git operations for content branch management, merging, staging, and push retry logic
api-server/routes/jobs.ts	Integrates fetch job lock checking, returns simplified response schemas for fetch jobs

Sequence Diagram

sequenceDiagram
    participant Client
    participant API as API Server
    participant Lock as Fetch Job Lock
    participant Tracker as Job Tracker
    participant Runner as Fetch Job Runner
    participant Notion as Notion API
    participant Git as Content Repo

    Client->>API: POST /jobs (fetch-ready)
    API->>Lock: isFetchJobLockHeld()?
    Lock-->>API: false
    API->>Tracker: createJob(fetch-ready)
    Tracker-->>API: jobId
    API->>Lock: tryAcquireFetchJobLock(jobId)
    Lock-->>API: true (acquired)
    API-->>Client: 202 {jobId, status: pending}
    
    API->>Runner: runFetchJob (async)
    Runner->>Notion: fetchAllNotionData (Ready to publish)
    Notion-->>Runner: pages + candidateIds
    Runner->>Git: prepareContentBranch (merge main->content)
    Git-->>Runner: remoteRef
    Runner->>Runner: runGenerationScript (spawn bun)
    Runner->>Git: copyGeneratedContent + stage
    Runner->>Git: hasStagedChanges?
    Git-->>Runner: true
    Runner->>Git: commit + push
    Git-->>Runner: commitHash
    Runner->>Notion: updatePageStatus (Draft published)
    Notion-->>Runner: success
    Runner->>Tracker: setTerminalState
    Runner->>Lock: releaseFetchJobLock(jobId)
    
    Client->>API: GET /jobs/{jobId}
    API->>Tracker: getJob(jobId)
    Tracker-->>API: job with terminal state
    API-->>Client: 200 {status: completed, pagesTransitioned, commitHash}

Loading

_{Last reviewed commit: 32209e9}

[chatgpt-codex-connector]

💡 Codex Review

comapeo-docs/api-server/request-handler.ts

Line 46 in c5a09bb

: requireAuth(authHeader);

Skip auth before handling CORS preflight

Protected routes are authenticated before routing, but OPTIONS handling lives in routeRequest, so unauthenticated browser preflight requests to endpoints like /jobs receive 401 instead of the expected 204 CORS response. This blocks cross-origin API usage even when the actual request would include a valid API key, because browsers do not send credentials in the preflight by default.

---

comapeo-docs/api-server/fetch-job-runner.ts

Lines 361 to 364 in c5a09bb

	.filter(
	(page) => page.status === NOTION_PROPERTIES.READY_TO_PUBLISH
	)
	.map((page) => page.id)

Keep ready-page transition targets when selecting children

The fetch-ready flow derives transitionCandidates only from returned pages whose status is Ready to publish, but the new status-filter behavior can replace matching parents with their Sub-item children; in that common parent/child setup, this list becomes empty and no page status is transitioned to Draft published after a successful fetch-ready run. This breaks the post-publish status advancement for databases that use parent pages with related children.

---

comapeo-docs/Dockerfile

Line 16 in c5a09bb

COPY package.json bun.lockb* ./

Copy the tracked Bun lockfile into Docker builds

The image build copies bun.lockb*, but this repository tracks bun.lock; as a result the lockfile is not included in the dependency install layer, so bun install --frozen-lockfile runs without the committed lock state and can resolve different dependency trees over time. That makes production images non-reproducible and can diverge from tested dependency versions.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Overview

The PR implements API-native fetch jobs (fetch-ready and fetch-all) to replace the GitHub Actions workflow-based approach. The implementation follows the detailed plan in implementation-plan-fetch-ready.md and includes:

• New job types with proper concurrency control via in-memory lock
• Timeout handling (20 min default) with proper cleanup
• Status transition logic with retry and warning handling
• Terminal state persistence for job restart recovery
• New CI validation workflow (api-validate.yml)
• Updated docker-compose with API key environment variable

Files Reviewed (43 files)

• .github/workflows/api-notion-fetch.yml - Removed (replaced by API-native jobs)
• .github/workflows/api-validate.yml - New CI validation workflow
• .github/workflows/clean-content.yml - Updated with content branch sync
• api-server/content-repo.ts - Extended with fetch-specific functions
• api-server/fetch-job-runner.ts - New file for fetch job execution
• api-server/fetch-job-lock.ts - New file for concurrency control
• api-server/routes/jobs.ts - Updated with fetch job handling
• api-server/validation-schemas.ts - Added new job type validation
• docker-compose.yml - Added API_KEY_GITHUB_ACTIONS env var
• scripts/notion-fetch-all/fetchAll.ts - Added candidateIds tracking
• scripts/ci-validation/vps-fetch-smoke.sh - New VPS smoke test script
• Context and documentation files

Prior Issues Status

All previously reported issues have been addressed:

• ✅ content-repo.ts now uses config.contentBranch
• ✅ fetch-job-runner.ts has timeout wrapper for Notion API calls
• ✅ Concurrency lock properly releases in finally block
• ✅ Error classification handles ContentRepoError with proper codes

[github-actions]

🚀 Preview Deployment

Your documentation preview is ready!

Preview URL: https://pr-143.comapeo-docs.pages.dev

🔄 Content: Regenerated 5 pages from Notion (script changes detected)

💡 Tip: Add label fetch-all-pages to test with full content, or fetch-10-pages for broader coverage.

This preview will update automatically when you push new commits to this PR.

---

_{Built with commit 32209e9}

[luandro]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6ce70263f2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[greptile-apps]

_{42 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[github-actions]

🧹 Preview Deployment Cleanup

The preview deployment for this PR has been cleaned up.

Preview URL was: https://pr-143.comapeo-docs.pages.dev

---

_{Note: Cloudflare Pages deployments follow automatic retention policies. Old previews are cleaned up automatically.}