Skip to content

Move entity transformation to build time#5

Open
OAGr wants to merge 2 commits intomainfrom
claude/simplify-build-data-UUWWk
Open

Move entity transformation to build time#5
OAGr wants to merge 2 commits intomainfrom
claude/simplify-build-data-UUWWk

Conversation

@OAGr
Copy link
Contributor

@OAGr OAGr commented Feb 9, 2026

Summary

This PR moves entity transformation (type mapping, expert/org merging, risk categorization) from runtime to build time. Entities are now pre-transformed into typed entities during the build process and stored in database.json as typedEntities, eliminating runtime transformation overhead and simplifying the data loading layer.

Key Changes

  • Build-time transformation: Created apps/longterm/scripts/lib/entity-transform.mjs to handle all entity transformation logic (type mapping, expert/org merging, risk categories) during build
  • Simplified runtime loading: Removed ~230 lines of transformation code from apps/longterm-next/src/data/index.ts; now just validates pre-transformed entities via Zod
  • Removed entity overrides system: Deleted applyEntityOverrides() and related override logic; type remapping now happens at build time
  • Extracted utilities: Moved computed facts logic to apps/longterm/scripts/lib/computed-facts.mjs and unconverted links detection to apps/longterm/scripts/lib/unconverted-links.mjs for better modularity
  • Updated database schema: Added optional typedEntities field to DatabaseShape; runtime now requires this field and throws helpful error if missing
  • Updated test fixtures: Added sample typedEntities to mock database in tests

Implementation Details

  • Entity transformation now happens in build-data.mjs via transformEntities() which handles:
    • Old type name mapping to canonical entity types
    • Lab-* type flattening to organization with orgType
    • Expert/organization data merging
    • Custom field extraction for typed fields (role, affiliation, etc.)
    • Risk category assignment
  • Runtime validation is now simpler: parse pre-transformed entities and fall back to generic schema if needed
  • Build process logs count of transformed entities for visibility
  • Maintains backward compatibility: generic fallback still handles unknown types

Benefits

  • Performance: Eliminates runtime transformation overhead
  • Clarity: Build-time transformation is easier to understand and debug
  • Maintainability: Transformation logic is now in one place (build script) rather than split between build and runtime
  • Type safety: Pre-validation at build time catches issues earlier

https://claude.ai/code/session_01DL2zdVVyQUfB3UXYjA7Sj7

Phase A: Extract 4 inline function groups from build-data.mjs into separate
modules in apps/longterm/scripts/lib/:
- computed-facts.mjs: parseNumericValue, resolveComputedFacts
- statistics.mjs: computeStats
- unconverted-links.mjs: buildUrlToResourceMap, findUnconvertedLinks, countConvertedLinks
- mdx-generator.mjs: generateMdxFromYaml

Phase B: Move entity transformation from runtime (index.ts) to build time:
- Create entity-transform.mjs with transformEntities orchestrator that handles
  type mapping, expert/org merging, risk categories, and entity overrides
- Add typedEntities to database.json output from build-data.mjs
- Simplify getTypedEntities() in index.ts to read pre-transformed entities
- Remove ~287 lines of dead runtime transform code from index.ts

Results: build-data.mjs 1124→592 lines, index.ts 1246→959 lines.

https://claude.ai/code/session_01DL2zdVVyQUfB3UXYjA7Sj7
@vercel
Copy link

vercel bot commented Feb 9, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
cairn Ready Ready Preview Feb 9, 2026 3:47pm
cairn-meta Ready Ready Preview Feb 9, 2026 3:47pm
longtermwiki Ready Ready Preview Feb 9, 2026 3:47pm

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants