Homepage: https://www.usenix.org/conference/nsdi26
Paper list: https://www.usenix.org/conference/nsdi26/technical-sessions
- Spring: 24.2% (= 50 / 207)
- Fall: 22.1% (= 100 / 452)
- LLM Training
- MoE Training
- SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling [Paper] [arXiv]
- Stanford & NVIDIA & OpenAI
- Propose SYMI, an adaptive MoE training system that decouples the placement of expert parameters from their large optimizer states.
- Statically partition optimizer states across training nodes while dynamically adjusting expert parameter placement using existing weight updates, avoiding frequent state migration overheads.
- Improve time-to-convergence by 30.5% over DeepSpeed and 25.9% over FlexMoE.
- Checkpoint Lite, Recover Right: Efficient Fault Tolerant Training of Mixture-of-Experts Models Using Sparse Checkpoints [Paper]
- Stanford & NVIDIA
- SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling [Paper] [arXiv]
- Cross-Cluster Training
- Di-PS: System-Algorithm Co-Design for Asynchronous and Heterogeneous Cross-Cluster LLM Training at Scale [Paper]
- NUDT & Shanghai AI Lab & NTU
- Di-PS: System-Algorithm Co-Design for Asynchronous and Heterogeneous Cross-Cluster LLM Training at Scale [Paper]
- Fault Tolerance
- Attack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training [Paper] [arXiv]
- HKUST & Alibaba
- Present a straggler-resilient hybrid-parallel training system for pipeline-parallel large-model training under communication stragglers.
- Adapt the pipeline schedule with an analytical model to absorb slow communication without cascading bubbles, and offload delayed communication to host memory with CPU-side RDMA to avoid GPU head-of-line blocking.
- Reduce training iteration time by 1.2x to 3.5x under various straggler settings.
- Flare: Anomaly Diagnostics for Divergent LLM Training in GPU Clusters of Thousand-Plus Scale [Paper] [arXiv]
- SJTU & Ant Group & NUS
- Introduce Flare, a diagnostic framework for distributed LLM training at scale.
- Combine a lightweight tracing daemon for full-stack, backend-extensible tracing with a diagnostic engine that automatically diagnoses anomalies, especially performance regressions.
- Demonstrate deployment across 6,000 GPUs with continuous operation for over eight months in production scenarios.
- EROICA: Online Performance Troubleshooting for Large-scale Model Training in Production [Paper] [arXiv]
- Alibaba
- Present EROICA, an online troubleshooting system for large-scale model training that combines fine-grained profiling with full-cluster coverage.
- Summarize runtime execution patterns through online profiling and use differential observability to localize hardware, software, and mixed root causes with minimal production impact.
- Report deployment on production GPU clusters of about 100,000 GPUs for 1.5 years, diagnosing difficult performance issues with 97.5% success.
- Attack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training [Paper] [arXiv]
- RL Post-Training
- RollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching [Paper] [arXiv]
- HKUST & Alibaba
- Introduce tail batching, a rollout scheduling strategy for synchronous RL post-training that packs prompts with long-tail responses into a small number of long rounds while keeping most rounds balanced and short.
- Combine tail batching with elastic rollout parallelism, dynamic reward-stage resource scheduling, and stream-based training to reduce rollout bubbles without relaxing synchronization.
- Cut end-to-end training time by 2.03x to 2.56x over veRL and by up to 2.24x over RLHFuse on Qwen2.5 models.
- RollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching [Paper] [arXiv]
- Performance Modeling and Simulation
- Supercharging Packet-Level Network Simulation of Large Model Training via Memoization and Fast-Forwarding [Paper] [arXiv]
- THU & Zhongguancun Laboratory & BUPT & Huawei
- Propose Wormhole, a user-transparent packet-level discrete-event simulation kernel for large-model training that reduces redundant simulation work without simplifying the network model.
- Memoize unsteady states and fast-forward steady states through network partitioning, state reuse, and rate-based steady-state identification while preserving simulation consistency.
- Achieve 744x speedup over ns-3 with less than 1% error; combined with multithreading, the speedup reaches 1012x.
- GPUSynth: Maximizing Code Reuse in Simulation-Based Machine Learning System Performance Estimation [Paper] [arXiv]
- Duke & UC Berkeley & Meta
- Introduce Phantora, a hybrid GPU-cluster simulator for ML training performance estimation that runs unmodified ML frameworks in a distributed containerized environment.
- Intercept GPU and communication operations during execution, enabling direct reuse of ML framework source code instead of reimplementing frameworks inside a simulator.
- Match the accuracy of static workload simulation while supporting three state-of-the-art LLM training frameworks out of the box on a single GPU.
- Supercharging Packet-Level Network Simulation of Large Model Training via Memoization and Fast-Forwarding [Paper] [arXiv]
- MoE Training
- LLM Inference
- Request Scheduling
- FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference [Paper] [arXiv] [Code]
- PKU
- Propose FastServe, an LLM serving system that enables iteration-level preemptive scheduling for autoregressive decoding instead of request-level FIFO execution.
- Combine a skip-join multi-level feedback queue scheduler with a parallelism-aware suspension strategy that downscales preempted models into CPU/NVMe storage.
- Improve multi-model serving throughput and latency; the abstract reports up to 5.1x higher throughput and 13.9x to 111.8x lower latency than prior systems.
- JITServe: SLO-aware LLM Serving with Imprecise Request Information [Paper] [arXiv]
- UIUC & NJU & Google Labs & Cisco Research
- Propose JITServe, an LLM serving system for workloads where request information is imprecise at arrival time.
- Forecast future decoding lengths from near-future token generation and construct just-in-time execution schedules that balance prediction accuracy against rapidly changing system dynamics.
- Increase throughput while improving latency SLO satisfaction; the abstract reports 1.8x to 7.5x higher throughput and 2.2x to 8.7x better SLO attainment than prior systems.
- Libra: Flexible Request Partitioning and Scheduling for Serving Unbalanced and Dynamic LLM Workloads [Paper]
- NUS & USTC & UC Berkeley
- FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference [Paper] [arXiv] [Code]
- KV Cache Management
- DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants [Paper] [arXiv]
- UChicago & Microsoft
- Introduce DroidSpeak, the first distributed LLM inference system that reuses prefix KV caches across different LLMs with the same architecture, including across distributed nodes.
- Selectively recompute a small subset of layers from another model's KV cache and reuse the remaining layers, then pipeline recomputation with reused-cache loading to improve performance while preserving quality.
- Improve throughput by up to 4x and prefill latency by about 3.1x with negligible quality loss.
- SYMPHONY: Improving Memory Management for LLM Inference Workloads [Paper] [arXiv]
- UT Austin & UW-Madison
- Exploit hints from multi-turn workloads to migrate KV caches off the critical serving path instead of recomputing them or pinning serving to specific machines through host-memory offload.
- Dynamically migrate KV caches to enable fine-grained scheduling of inference requests across the cluster.
- Handle more than 8x as many requests as state-of-the-art baselines while preserving a similar latency profile.
- DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants [Paper] [arXiv]
- Workload Characterization
- Seshat: Workload Characterization and Generation of Large Language Model Serving in Production [Paper] [arXiv]
- PKU & Alibaba
- Characterize production LLM serving workloads from a worldwide cloud inference service, covering language, multimodal, and reasoning models.
- Build a per-client workload generation framework that composes realistic serving workloads from the observed production characteristics.
- Avoid 50% under-provisioning compared with naive workload generation in a production validation case.
- Seshat: Workload Characterization and Generation of Large Language Model Serving in Production [Paper] [arXiv]
- Serverless Computing
- HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds [Paper] [arXiv] [Code]
- PKU & Alibaba Cloud
- Present HydraServe, a serverless LLM serving system for public clouds that minimizes cold-start latency.
- Proactively distribute models across servers, overlap cold-start stages within workers, place workers to avoid GPU-network contention, and consolidate pipelines to reduce cold-start resource usage.
- Reduce cold-start latency by 1.7x to 4.7x and improve SLO attainment by 1.43x to 1.74x over baselines.
- HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds [Paper] [arXiv] [Code]
- Multiplexing
- FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees [Paper] [arXiv]
- CMU & Purdue & Anthropic & Mistral AI & Stanford
- Introduce FlexLLM, the first system to co-serve LLM inference and parameter-efficient fine-tuning on shared GPUs by fusing computation at the token level.
- Use dependent parallelization and graph pruning to shrink activation memory, then interleave inference and training tokens with token-level finetuning and a hybrid token scheduler to meet latency SLOs.
- Save up to 80% of GPU memory and improve finetuning throughput by 1.9x to 4.8x under heavy inference load and 2.5x to 6.8x under light load.
- FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees [Paper] [arXiv]
- LLM Agents
- Agentix: An Efficient Serving Engine for LLM Agents as General Programs [Paper] [arXiv]
- UC Berkeley & Google DeepMind & SJTU
- Treat agent programs as first-class scheduling entities in LLM serving to reduce end-to-end latency for agentic workloads.
- Intercept program-issued LLM calls to expose program-level context and preemptively prioritize calls based on previously completed work for both single-threaded and distributed programs.
- Improve program throughput by 4x to 15x at the same latency compared with systems such as vLLM.
- Cortex: Achieving Low-Latency, Cost-Efficient Remote Data Access For LLM via Semantic-Aware Knowledge Caching [Paper] [arXiv]
- NUS & USTC & UofT & Sea AI Lab
- Introduce Cortex, a cross-region knowledge caching architecture for LLM agents that targets semantic reuse rather than exact-match query reuse.
- Build semantic-aware retrieval on Semantic Element and Semantic Retrieval Index abstractions, then add semantic-aware cache hits, eviction, prefetching, and a colocated lightweight LLM judger.
- Increase throughput by up to 3.6x on search workloads while preserving accuracy close to uncached baselines, and improve coding-task throughput by 20%.
- Agentix: An Efficient Serving Engine for LLM Agents as General Programs [Paper] [arXiv]
- MoE Inference
- SwiftEP: Accelerating MoE Inference with Buffer Fusion and TMA Offloading [Paper]
- Tencent & Nanjing University & Zhongguancun Laboratory
- SwiftEP: Accelerating MoE Inference with Buffer Fusion and TMA Offloading [Paper]
- Request Scheduling
- LLM Fine-Tuning
- MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexing [Paper] [arXiv]
- SJTU & NUS
- Present MuxTune, a multi-tenant fine-tuning system that spatially and temporally multiplexes the shared backbone across concurrent parameter-efficient fine-tuning tasks.
- Build on unified fine-tuning representations with hierarchical co-scheduling across tasks, operators, and data, including hybrid spatial-temporal multiplexing, two-tier hybrid parallelism, and chunk-based data alignment.
- Achieve up to 2.33x higher throughput and 5.29x lower memory usage than prior baselines.
- MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexing [Paper] [arXiv]
- LLM Storage
- ZipLLM: Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression [Paper] [arXiv] [Homepage]
- UVA & Harvard
- Characterize all publicly available Hugging Face LLM repositories and identify structured sparse deltas within model families, bitwise-similarity-based family clustering, and tensor-level deduplication as the right storage granularity.
- Design BitX, a lossless delta compression algorithm for XORed differences between fine-tuned and base models, and build ZipLLM to unify tensor-level deduplication with BitX compression.
- Reduce model storage consumption by 54%, over 20% better than prior deduplication and compression approaches.
- ZipLLM: Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression [Paper] [arXiv] [Homepage]
- Checkpointing
- Checkmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication [Paper] [arXiv]
- Tufts & MIT
- Introduce Checkmate, a model checkpointing system that reuses network-replicated gradients to avoid additional network transfer and disk I/O overhead on the normal checkpointing path.
- Use dynamically reconfigurable in-network checkpoint replica placement and a failure-resilient accelerator pipeline to preserve resilience under failures while overlapping checkpoint creation with gradient computation and propagation.
- Report nearly 100% throughput improvement over GPU-optimized checkpointing and up to 13.7% over asynchronous checkpointing on a 32-node GPU cluster.
- Checkmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication [Paper] [arXiv]
- Collective Communication
- FAST: An Efficient Scheduler for All-to-All GPU Communication [Paper] [arXiv]
- CMU & MangoBoost & UW & UPenn
- Present FAST, an efficient All-to-All(v) scheduler for modern ML workloads, especially MoE models on heterogeneous two-tier GPU fabrics.
- Address workload skew through intra-server rebalancing and enforce balanced one-to-one scale-out transfers to avoid incast congestion.
- Outperform prior schedulers on skewed workloads while reducing schedule synthesis time by orders of magnitude.
- HeteCCL: Synthesizing Near-Optimal Collective Communication Schedules for Heterogeneous GPU Clusters [Paper]
- Northeastern University & Alibaba Cloud & SIAT, CAS
- ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics [Paper] [arXiv]
- UW & THU & Microsoft
- Present ForestColl, a schedule generation tool that constructs broadcast and aggregation spanning trees to produce throughput-optimal collective communication schedules for arbitrary network topologies.
- Achieve theoretical optimality with polynomial-time schedule generation while supporting both switching fabrics and direct accelerator interconnects.
- Outperform vendor communication libraries and prior schedule generation methods on AMD MI250 and NVIDIA DGX A100 and H100 clusters, including LLM training workloads.
- FAST: An Efficient Scheduler for All-to-All GPU Communication [Paper] [arXiv]
- KV: Key-Value
- LLM: Large Language Model
- MoE: Mixture-of-Experts
- RL: Reinforcement Learning
- SLO: Service Level Objective