Code quality: Replace magic numbers with named constants

[Evrard-Nil]

Problem

The codebase contains magic numbers without explanation:

1. Channel buffer size (src/proxy.rs:126)

   let (tx, rx) = tokio::sync::mpsc::channel::<Result<Bytes, std::io::Error>>(64);

   Why 64?

2. Cache capacity (src/cache.rs:14)

   .max_capacity(10_000)

   Why 10,000 entries?

3. Cache TTL (src/config.rs:107)

   chat_cache_expiration_secs: env_int("CHAT_CACHE_EXPIRATION", 1200) as u64,

   Why 1200 seconds (20 minutes)?

4. UUID truncation (src/proxy.rs:64)

   &uuid::Uuid::new_v4().to_string().replace('-', "")[..24]

   Why 24 characters?

5. Timeout values (src/config.rs:111-112)

   timeout_secs: 600,
   timeout_tokenize_secs: 10,

   Why these specific values?

Impact

Medium - Makes code harder to understand and maintain. If these values need to change, it's unclear where they're used or what their rationale was.

Solution

Define named constants with documentation:

// src/proxy.rs
/// Buffer size for streaming response channels. Set to 64 to balance memory usage
/// with flow control - allows upstream to get ahead by at most 64 chunks (~64KB
/// assuming 1KB average chunk size) before backpressure kicks in.
const STREAMING_CHANNEL_BUFFER_SIZE: usize = 64;

// src/cache.rs
/// Maximum number of signature entries to cache. Set to 10,000 to accommodate
/// high-throughput deployments while limiting memory usage to ~10MB
/// (assuming ~1KB per cached signature).
const MAX_CACHE_ENTRIES: u64 = 10_000;

/// Default TTL for cached signatures (20 minutes). Chosen to balance:
/// - Signature freshness (shorter is better)
/// - Cache hit rate (longer is better)
/// - Memory pressure (shorter is better)
const DEFAULT_CACHE_TTL_SECS: u64 = 20 * 60;

// src/proxy.rs
/// Length of auto-generated response IDs. OpenAI IDs are typically 24-29 chars,
/// we use 24 for compatibility while ensuring uniqueness (128 bits of entropy).
const GENERATED_ID_LENGTH: usize = 24;

// src/config.rs
/// Default timeout for backend requests (10 minutes). Set high to accommodate
/// long-running inference tasks. Individual endpoints can override.
const DEFAULT_REQUEST_TIMEOUT_SECS: u64 = 10 * 60;

/// Timeout for tokenize endpoint (10 seconds). Tokenization should be fast,
/// so use a shorter timeout to fail fast.
const TOKENIZE_TIMEOUT_SECS: u64 = 10;

Then use these constants:

// Before:
let (tx, rx) = tokio::sync::mpsc::channel::<Result<Bytes, std::io::Error>>(64);

// After:
let (tx, rx) = tokio::sync::mpsc::channel::<Result<Bytes, std::io::Error>>(STREAMING_CHANNEL_BUFFER_SIZE);

File Locations

• src/proxy.rs:126, 379 - Channel buffer size
• src/proxy.rs:64, 236, 337 - ID truncation length
• src/cache.rs:14, 15 - Cache capacity and TTL
• src/config.rs:103-112 - Timeouts and limits

Additional Benefits

• Easier to make values configurable later via env vars
• Self-documenting code
• Centralized location for tuning parameters