Add VCF loading HTTP endpoint

[jsstevenson]

Wraps the VCF loading function. Presumably there will be some questions to answer about how to handle long file uploads.

[quinnwai]

Overview

HTTP endpoint for uploading and ingesting VCF files with allele frequency data into AnyVLM.

Design Decisions

Endpoint

• Path: POST /ingest_vcf
• Processing: Synchronous (MVP)
• Response: Simple JSON status message

Parameters

• file: Required - Uploaded VCF file (multipart/form-data)
• assembly: Required - Query parameter (GRCh37 or GRCh38)

Dependencies

FastAPI with python-multipart w/pysam to parse VCF and anyvar client for business logic

File Requirements

• Format: .vcf.gz only (gzip compressed VCF)
• Max Size: 5GB
• Required INFO Fields: AC, AN, AC_Het, AC_Hom, AC_Hemi
• Well-formed: Valid VCF format with proper headers

Validation

1. Filename extension check (.vcf.gz)
2. Content-Type header validation
3. Gzip magic bytes verification (\x1f\x8b)
4. File size limit check (≤5GB)
5. VCF header format validation
6. Required INFO fields verification
7. Runtime record validation (via pysam)

File Handling

• Upload Strategy: Stream to disk (1MB chunks) - memory efficient (some assumptions about UploadFile here)
• Storage Location: System temp directory (tempfile)
• Cleanup: Immediate delete in finally block
• Unique Naming: anyvlm_{uuid4()}.vcf.gz

Timeout & Performance

• Server Timeout: 30 minutes (Uvicorn config)
• Rationale: Fixed timeout sufficient for 5GB limit
• No per-request dynamic timeouts (defer until performance data available)

Error Handling

400 - Invalid file format/extension
413 - File too large (>5GB)
422 - VCF validation failed (missing fields, malformed)
408 - Processing timeout
500 - Server/processing error

Response Schema

Success: {"status": "success", "message": "..."}
Error:   {"status": "error", "message": "...", "details": "..."}

Testing (pytest)

• Test successful upload (small .vcf.gz)
• Test file size limit (>5GB)
• Test invalid formats (.vcf, .txt, .gz)
• Test missing INFO fields
• Test malformed VCF
• Test magic bytes validation

Documentation

• Add endpoint to API docs (auto-generated via FastAPI)
• Update README with endpoint usage example

Future Work / Discussion Required

• Async Processing: do we prefer a set it + forget it, more workflow-style endpoint
  • Needs a correspond status query endpoint
  • Is this a separate endpoint or dynamically inferred based on file [size]?
• Monitoring: how do we monitor VCF ingestion over time? Goes hand it hand w/ async
• Error Handling: also relatedm do we retain VCFs on server-side to debug failed uploads?
• Misc Configuration: maybe no file size limits? Dynamic timeout calc based on file size
• Security: who gets to write to db?

Success Criteria

• Accept .vcf.gz uploads up to 5GB
• Process valid VCFs and register variants
• Reject invalid files with clear errors
• No memory issues with large files
• Clean up temporary files reliably