Consider dropping baked-in embedding model in favor of startup warmup

[legreffier]

Context

Surfaced during #1223 (Docker image pipeline refactor).

apps/rest-api currently bakes Xenova/e5-small-v2 (q8) into the production image at build time:

• A host-side Nx target (download-model) runs tools/download-embedding-model.mjs --cache-dir apps/rest-api/dist/models
• The Dockerfile COPY apps/rest-api/dist/models/ ./models/
• Runtime config sets EMBEDDING_CACHE_DIR=/app/models and EMBEDDING_ALLOW_REMOTE_MODELS=false so @huggingface/transformers runs in local_files_only mode

Question

Should we drop the bake-in and let the model download at startup instead?

Pros of dropping:

• ~30–50 MB smaller image
• Removes the host-side download-model Nx target, the dist/models shipping path, and the env-var coupling
• Dockerfile loses one COPY (closer to pure packaging)
• Drops dependency on the host being able to dlopen onnxruntime-node during nx build (currently a non-issue, but tightens the contract)

Cons of dropping:

• First cold boot is ~5–30s slower (model fetch from huggingface.co)
• Production becomes dependent on HF reachability — outages, egress policies, or air-gapped deploys break it
• Embedding service initialization needs to happen at startup, not lazily, otherwise the first user request takes the hit instead of the boot

Proposed approach (if we proceed)

1. Remove download-model Nx target + dist/models COPY in apps/rest-api/Dockerfile
2. Default EMBEDDING_ALLOW_REMOTE_MODELS=true in production
3. Add an explicit startup warmup in apps/rest-api/src/bootstrap.ts — call embeddingService.embed(['warmup']) after fastify ready, before declaring healthy
4. Keep EMBEDDING_CACHE_DIR=/app/models so the downloaded files persist between requests within the same container lifetime (but not across pod restarts, which is fine)

Decision deferred

For now, #1223 keeps the bake-in behavior to scope the refactor strictly to "where does build happen" without changing what ships. This issue tracks the follow-up evaluation.