Add continuous batching support for concurrent LLM inference#150

Open

ronaldmannak wants to merge 101 commits intoml-explore:mainfrom

PicoMLX:batching3

Commits on Mar 27, 2026

Add mission infrastructure for continuous batching

ronaldmannak
and
factory-droid[bot]
committed
Implement BatchKVCache with left-padding strategy for continuous batching

ronaldmannak
and
factory-droid[bot]
committed
Add batch-aware masking, BatchPositionedKVCache protocol, and applyRotaryPosition helper

ronaldmannak
and
factory-droid[bot]
committed
Implement BatchRotatingKVCache for sliding-window batch attention

ronaldmannak
and
factory-droid[bot]
committed
Add Metal availability guard to skip MLX-dependent tests in SPM builds

ronaldmannak
and
factory-droid[bot]
committed
Fix swift-format lint violations in batch files

ronaldmannak
and
factory-droid[bot]
committed
Record batch-kv-cache scrutiny findings

ronaldmannak
and
factory-droid[bot]
committed
Update worker skill with MLX Metal TDD guidance

ronaldmannak
and
factory-droid[bot]
committed
Fix BatchKVCache state serialization, makeMask offset timing, and Sendable warning

ronaldmannak
and
factory-droid[bot]
committed
Add prepare/finalize for cached-prompt prefill and preserve keep in BatchRotatingKVCache

ronaldmannak
and
factory-droid[bot]
committed
Record batch-kv-cache scrutiny rerun findings

ronaldmannak
and
factory-droid[bot]
committed
Fix BatchRotatingKVCache to preserve keep prefix tokens during sliding-window overflow

ronaldmannak
and
factory-droid[bot]
committed
Record batch-kv-cache keep-semantics scrutiny rerun

ronaldmannak
and
factory-droid[bot]
committed
Fix BatchRotatingKVCache.extract() negative leftPadding after overflow

ronaldmannak
and
factory-droid[bot]
committed
Record batch-kv-cache scrutiny pass after extract fix

ronaldmannak
and
factory-droid[bot]
committed
Record batch-kv-cache user-testing findings

ronaldmannak
and
factory-droid[bot]
committed
Fix BatchKVCache.makeMask() key-width to equal _idx

ronaldmannak
and
factory-droid[bot]
committed
Fix BatchRotatingKVCache keep-prefix corruption for padded sequences during overflow

ronaldmannak
and
factory-droid[bot]
committed
Record batch-kv-cache user-testing rerun pass

ronaldmannak
and
factory-droid[bot]
committed
Implement BatchTokenIterator core batch generation engine

ronaldmannak
and
factory-droid[bot]
committed
Add per-request sampler/processor support and correctness tests for BatchTokenIterator

ronaldmannak
and
factory-droid[bot]
committed
Record batch-engine scrutiny findings

ronaldmannak
and
factory-droid[bot]
committed
Fix batch admission scheduling and add concurrency safety to BatchTokenIterator

ronaldmannak
and
factory-droid[bot]
committed
Record batch-engine scrutiny rerun pass

ronaldmannak
and
factory-droid[bot]
committed
Record batch-engine user-testing findings

ronaldmannak
and
factory-droid[bot]
committed
Fix per-request sampler concatenate crash in BatchTokenIterator

ronaldmannak
and
factory-droid[bot]
committed
Record batch-engine user-testing rerun pass

ronaldmannak
and
factory-droid[bot]
committed
Implement InferenceScheduler actor with single-first upgrade strategy

ronaldmannak
and
factory-droid[bot]
committed
Integrate InferenceScheduler with ModelContainer for transparent batching

ronaldmannak
and
factory-droid[bot]
committed
Record scheduler scrutiny findings

ronaldmannak
and
factory-droid[bot]
committed
Fix scheduler upgrade stream continuity, KV cache migration, and ChatSession history

ronaldmannak
and
factory-droid[bot]
committed
Record scheduler scrutiny rerun findings

ronaldmannak
and
factory-droid[bot]
committed
Fix scheduler upgrade to use live TokenIterator state and rebind cancellation

ronaldmannak
and
factory-droid[bot]
committed
Record scheduler scrutiny rerun findings

ronaldmannak
and
factory-droid[bot]
committed
Fix scheduler upgrade tensor shape, token boundary drop, and actor reentrancy

ronaldmannak
and
factory-droid[bot]
committed
Record scheduler scrutiny rerun findings

ronaldmannak
and
factory-droid[bot]
committed
Fix maxTokens off-by-one in upgradeToBatch() and Sendable warnings

ronaldmannak
and
factory-droid[bot]
committed
Record scheduler scrutiny rerun findings

ronaldmannak
and
factory-droid[bot]
committed
Override scheduler scrutiny: all xcodebuild tests pass, code correct

ronaldmannak
and
factory-droid[bot]
committed
Record scheduler user-testing findings

ronaldmannak
and
factory-droid[bot]
committed
Defer 6 scheduler assertions to cross-area, add Sendable fix feature, override user testing

ronaldmannak
and
factory-droid[bot]
committed
Fix Sendable/data-race warnings in scheduler integration

ronaldmannak
and
factory-droid[bot]
committed
Implement LRUPromptCache with trie-based prefix matching and LRU eviction

ronaldmannak
and
factory-droid[bot]
committed
Integrate LRUPromptCache with batch generation for cached prompt prefill

ronaldmannak
and
factory-droid[bot]
committed
Record prompt-cache scrutiny findings

ronaldmannak
and
factory-droid[bot]
committed
Fix 4 correctness bugs in LRUPromptCache with regression tests

ronaldmannak
and
factory-droid[bot]
committed
Fix 2 correctness bugs in prompt-cache batch integration

ronaldmannak
and
factory-droid[bot]
committed
Record prompt-cache scrutiny rerun findings

ronaldmannak
and
factory-droid[bot]
committed
Fix mixed-depth cached-prefill holes and RotatingKVCache support in batch path

ronaldmannak
and
factory-droid[bot]
committed
Record prompt-cache scrutiny round 3 findings
ronaldmannak
committed
Fix mixed-depth cached-prefill with prepare/finalize lifecycle

ronaldmannak
and
factory-droid[bot]
committed
Record prompt-cache scrutiny round 4 findings

ronaldmannak
and
factory-droid[bot]
committed
Record prompt-cache user-testing findings

ronaldmannak
and
factory-droid[bot]
committed
Fix trimPromptCache to trim all layers and correct exact-hit test expectations

ronaldmannak
and
factory-droid[bot]
committed
Record prompt-cache user-testing rerun findings

ronaldmannak
and
factory-droid[bot]
committed
Migrate all MLXLLM model RoPE calls to applyRotaryPosition

ronaldmannak
and
factory-droid[bot]
committed
Add comprehensive cross-area integration tests for batching

ronaldmannak
and
factory-droid[bot]
committed
Record example-app scrutiny findings
ronaldmannak
committed
Strengthen cross-area integration test assertions and fix compile warnings

ronaldmannak
and
factory-droid[bot]
committed
Record example-app scrutiny rerun findings
ronaldmannak
committed
Override example-app scrutiny: all tests pass, dead code paths dismissed

ronaldmannak
and
factory-droid[bot]
committed
Record example-app user-testing findings

ronaldmannak
and
factory-droid[bot]
committed
Record example-app user-testing rerun findings

ronaldmannak
and
factory-droid[bot]
committed
Complete example-app milestone: all 86 assertions passed

ronaldmannak
and
factory-droid[bot]
committed
Add continuous batching section to README

ronaldmannak
and
factory-droid[bot]
committed
Fix 3rd+ requests missing streaming events in batch mode

ronaldmannak
and
factory-droid[bot]
committed
Fix rotating/sliding-window caches silently dropped during batch creation and upgrade

ronaldmannak
and
factory-droid[bot]
committed
Fix batched .info events: report correct promptTokenCount and preserve timing through upgrade

ronaldmannak
and
factory-droid[bot]
committed
Wire LRUPromptCache into scheduler path for upstream parity

ronaldmannak
and
factory-droid[bot]
committed
Record post-review scrutiny findings

ronaldmannak
and
factory-droid[bot]
committed
Fix vacuous rotating cache preservation test with actual content verification

ronaldmannak
and
factory-droid[bot]
committed
Fix incorrect promptTime for 3rd+ requests joining existing batch

ronaldmannak
and
factory-droid[bot]
committed
Fix prompt cache wiring completeness: write-back, single-path cache use, and correct token keys

ronaldmannak
and
factory-droid[bot]
committed
Record post-review scrutiny rerun findings

ronaldmannak
and
factory-droid[bot]
committed
Make testUpgradePreservesRotatingKVCacheState deterministic

ronaldmannak
and
factory-droid[bot]
committed
Fix prompt cache write-back to use full token sequence key

ronaldmannak
and
factory-droid[bot]
committed
Fix flaky testUpgradePreservesRotatingKVCacheState timing

ronaldmannak
and
factory-droid[bot]
committed
Record post-review scrutiny round 3 findings

ronaldmannak
and
factory-droid[bot]
committed
Fix RotatingCacheMockModel to never produce EOS token 0

ronaldmannak
and
factory-droid[bot]
committed
Fix prompt cache write-back to include pre-upgrade generated tokens

ronaldmannak
and
factory-droid[bot]
committed
Record post-review scrutiny round 4 findings

ronaldmannak
and
factory-droid[bot]
committed
Record post-review user testing results

ronaldmannak
and
factory-droid[bot]
committed
Fix mixed-layer cached partial-hit to use per-layer type check

ronaldmannak
and
factory-droid[bot]
committed
Fix BatchKVCache masks for post-update attention width

ronaldmannak
and
factory-droid[bot]
committed
Fix mixed-depth cached-prefill final cache extraction

ronaldmannak
and
factory-droid[bot]
committed
Record post-review-followup scrutiny findings

ronaldmannak
and
factory-droid[bot]
committed
Record post-review-followup user testing results

ronaldmannak
and
factory-droid[bot]
committed
Fix scheduler fallback prompt-cache propagation

ronaldmannak
and
factory-droid[bot]
committed
Record post-review-followup-2 scrutiny findings

ronaldmannak
and
factory-droid[bot]
committed
Record post-review-followup-2 user testing results

ronaldmannak
and
factory-droid[bot]
committed
Update gitignore, remove .factory from git repo
ronaldmannak
committed
Add batching support for NanoChat and Phi3
ronaldmannak
committed
Batch-aware attention masks for FalconH1 and Gemma
ronaldmannak
committed
swift lint
ronaldmannak
committed
Add wired memory support
ronaldmannak
committed
swift lint
ronaldmannak
committed
improve dual path routing
ronaldmannak
committed
Add raw token batching
ronaldmannak
committed
Add raw token batching
ronaldmannak
committed
Update SKILL.md
ronaldmannak
committed

Commits on Mar 29, 2026

Revert order Model Factories (VLM is first again)
ronaldmannak
committed