vllm-stack-0.1.10

The stack deployment of vLLM

What's Changed

• Add servingEngineSpec environment variable by @shernshiou in #799
• [Fix] Handle missing max_tokens in disaggregated prefill requests by @keyuchen21 in #797
• [Router]: add routes for Image and Audio API by @nmiguel in #820
• [Router][Fix]: fixed name of images/edits endpoint by @nmiguel in #822
• Update contact information in README.md by @ruizhang0101 in #821
• Fix OCI OKE deployment script (entry_point.sh) — end-to-end tested by @fede-kamel in #811
• mention resources at the values.yaml as valid option by @eladmotola in #806
• [Doc] Update README for global env on servingEngineSpec by @shernshiou in #814
• feat(helm): add standard Kubernetes labels to deployments and services by @keyuchen21 in #810
• [BugFix][Feat]: fix serviceEngineSpec probe field and improve probe management in helm template by @emanuelecassese in #809
• [Bugfix] Increase router default memory size by @ruizhang0101 in #804
• [FEAT] Add per-model token and error Prometheus metrics (part of #699) by @ardecode in #813
• [CI/CD] Add stable router image by @ruizhang0101 in #823
• [Feat] Add toleration for vllmRunTimes by @mahmoudk1000 in #825
• [Feat] Operator : add GPUType for resources to replace "nvidia.com/gpu" in vllmruntime by @dotmobo in #829
• [Bugfix] Update aiohttp and python-multipart by @shernshiou in #831
• fix: make --log-level CLI argument actually control router log levels by @keyuchen21 in #832
• fix: Exclude content-length from response headers in route_general_transcriptions by @fidoriel in #733
• [Feat] Reorder hfTokenSecret for vllmRunTimes by @mahmoudk1000 in #826
• feat(router): add initial support for anthropic messages endpoint by @nejch in #775
• [Feat] Add token redaction for logger debug by @shernshiou in #824
• refactor: replace logging.getLogger() with init_logger() across codebase by @keyuchen21 in #835
• [CI/CD] add ci/cd for production stack operator by @ruizhang0101 in #843
• fix: filter hop-by-hop headers from streaming responses by @keyuchen21 in #836
• fix: upgrade h11 to 0.16.0 to resolve GHSA-vqfr-h8mv-ghfj by @keyuchen21 in #837
• Increase timeout values in e2e test workflow by @ruizhang0101 in #848
• [Feat][Router] Add request migration with configurable failover reroute attempts by @ikaadil in #839
• feat(helm) add support for extra manifests and annotation on pvc by @enneitex in #847
• feat: add --root-path CLI option for hosting router under a subpath by @keyuchen21 in #844
• [Misc] Expose LMCache log level as configurable Helm value and default to INFO. by @NargiT in #846
• [Feat] Add --log-format json option for structured logging by @keyuchen21 in #849
• [Router]: image edit routes multi-part form request by @nmiguel in #850
• [Docs] Update readme by @ruizhang0101 in #856
• Bump chart version to 0.1.10 by @ruizhang0101 in #859

New Contributors

• @nmiguel made their first contribution in #820
• @emanuelecassese made their first contribution in #809
• @dotmobo made their first contribution in #829
• @fidoriel made their first contribution in #733
• @nejch made their first contribution in #775
• @enneitex made their first contribution in #847

Full Changelog: vllm-stack-0.1.9...vllm-stack-0.1.10

vllm-stack-0.1.9

The stack deployment of vLLM

What's Changed

• [Feat] Add imagePullSecrets support for router and cache-server deplo… by @HanFa in #762
• [Feat] Add production-ready vLLM Nebius MK8s terraform tutorial by @brokedba in #748
• [Feat] Allow declaring modelSpec resources directly by @danhubern in #729
• [Router] Introduction of /v1/responses endpoint by @sebastiaanvduijn in #691
• [Bugfix][Router] Fix router startup race when using multiple replicas by @bcdonadio in #768
• [Docs] Correct parameter in transcription API tutorial by @davidgao7 in #685
• [Bugfix] Concurrent requests to model are currently limited to 100 due to aiohttp default by @dermodmaster in #767
• Update nixlPeerHost to pd-llama-decode-engine-service by @Xunzhuo in #771
• [Feat] Production Stack Router: Add OpenTelemetry tracing support with W3C context propagation by @HanFa in #772
• [Feat]: Add support for chatTemplates by @mahmoudk1000 in #779
• [Build][Router] Update vllm to v0.13.0 by @shernshiou in #770
• [Feat] Add nodeSelectorTerms for vllmRunTimes by @mahmoudk1000 in #778
• Update calendar link for community meetings by @ruizhang0101 in #783
• Update the documentation of the semantic router deployment to use helm by @szedan-rh in #786
• Fix incorrect import path in batch processor initialization by @keyuchen21 in #784
• [Build][Router] Update aiohttp by @shernshiou in #793
• Update Slack channel link in README by @keyuchen21 in #798
• [Doc] Remove official email link from README by @ruizhang0101 in #805
• feat(oci): Add Oracle Cloud Infrastructure (OKE) deployment support by @fede-kamel in #794
• [Feat] add keda support by @eladmotola in #781
• [CI/Build] Add stable version tags to Docker images during release by @ardecode in #801
• Fix score payload typo and add regression test by @keyuchen21 in #769
• [Feat] Include runner and convert flag by @shernshiou in #803

New Contributors

• @danhubern made their first contribution in #729
• @sebastiaanvduijn made their first contribution in #691
• @bcdonadio made their first contribution in #768
• @dermodmaster made their first contribution in #767
• @szedan-rh made their first contribution in #786
• @fede-kamel made their first contribution in #794
• @eladmotola made their first contribution in #781
• @ardecode made their first contribution in #801

Full Changelog: vllm-stack-0.1.8...vllm-stack-0.1.9

vllm-stack-0.1.8

The stack deployment of vLLM

What's Changed

• [Feat] Add GKE example for lmcache cpu ram + local disk offloading by @dannawang0221 in #678
• [Feat] Use the lmcache 0.3.5 for kvaware routing by @zerofishnoodles in #673
• [Feat]: add pull policy option to to ray-cluster.yaml (helm chart) by @moriabs88 in #686
• [Feat] Add support for scaling down to zero in KEDA by @Romero027 in #679
• [bugfix] Small fix to observability tutorial by @Romero027 in #695
• [Feat][Router]: add vision model type by @max-wittig in #603
• Adding Support for Sleep Mode for vLLM Container without Command Args by @dumb0002 in #696
• [Bugfix] Increase liveness failure threshold for crd by @zerofishnoodles in #688
• [bugfix] Add close method for static discovery by @zerofishnoodles in #692
• [Bugfix][Router]: loop through model_names by @max-wittig in #694
• [Misc] bump up otel col version and use a simplified image by @JaredTan95 in #698
• [vllm-router] fall back to remote tokenizer as 2nd path by @panpan0000 in #702
• [Bugfix][Router]: do not filter by model label in transcription by @max-wittig in #712
• [CI] move e2e machine to self hosted by @zerofishnoodles in #716
• [Feat] Add Production-ready vLLM EKS terraform stack tutorial by @brokedba in #704
• [bugfix] Add annotation to pod after loading the lora adapter to trigger the modify event by @zerofishnoodles in #703
• [Feat] [Router] [Misc] [Doc] increased configurability of affinity and probes by @Garrukh in #715
• [Bugfix] fix pd client initialization issue by @zerofishnoodles in #717
• [Bugfix] Update aiohttp to resolve CVE-2024-23334 vulnerability by @ikaadil in #722
• [Bugfix/Feature] Support extraPorts in service-vllm by @NargiT in #725
• Update gateway-inference-extension.rst by @linsun in #728
• feat(helm): Use emptyDir as pvcStorage by @Jimmy-Newtron in #616
• [Bugfix] Support service discovery by service name: add missing role and rolebinding for #586 by @NargiT in #724
• Update doc 04-GCP-GKE-lmcache-local-disk.md by @dannawang0221 in #727
• [Feat] Enable MIG support for Ray Head Node using chart.resources helper by @shima8823 in #732
• [feat] Enable session key in request body by @zerofishnoodles in #741
• [Feat] Add basic integration path for semantic router by @zerofishnoodles in #740
• [Bugfix] Pod rolebinding are requiered even with k8s_discovery_mode=serivce-name by @NargiT in #744
• [Feat] allow annotation on router pod by @NargiT in #743
• [Integration]: Add Intelligent Semantic Routing with vLLM-SR by @Xunzhuo in #750
• [Integration]: Update Docs with vLLM-SR by @Xunzhuo in #752
• [Bugfix] kv aware routing for lmcache 0.3.9 by @zerofishnoodles in #697
• [Feat] Ability to add labels to model pvc by @NargiT in #754
• [Bugfix] Helm: Add security context support, fix #756 by @aplufr in #757
• [Bugfix] lmcache server points to wrong file in entrypoint by @Senne-Mennes in #730
• [Feat] Add per-model runtimeClassName configuration support by @HanFa in #755
• Bumping version to 0.1.8 by @YuhanLiu11 in #738

New Contributors

• @dannawang0221 made their first contribution in #678
• @moriabs88 made their first contribution in #686
• @JaredTan95 made their first contribution in #698
• @panpan0000 made their first contribution in #702
• @brokedba made their first contribution in #704
• @Garrukh made their first contribution in #715
• @NargiT made their first contribution in #725
• @linsun made their first contribution in #728
• @Jimmy-Newtron made their first contribution in #616
• @shima8823 made their first contribution in #732
• @aplufr made their first contribution in #757
• @Senne-Mennes made their first contribution in #730
• @HanFa made their first contribution in #755

Full Changelog: vllm-stack-0.1.7...vllm-stack-0.1.8

vllm-stack-0.1.7

The stack deployment of vLLM

What's Changed

• [Feat] Added option to specify priority class by @Fabhiahn in #557
• [CI/Build] Change CI runner to L4 by @Shaoting-Feng in #595
• [Bugfix] fix dynamic config by @zerofishnoodles in #598
• [refactor] redesign RST documentation by @kobe0938 in #592
• [Misc] revert uv.lock by @kobe0938 in #604
• [CI/Build] Specify transformers version in router end to end test by @Shaoting-Feng in #607
• [Feat] allow service discovery by service names by @learner0810 in #586
• feat: add hpa for router by @BrianPark314 in #568
• [Misc] add helm configuration values table by @zerofishnoodles in #599
• [Feat] Use sidecar to download lora model for helm deployment by @zerofishnoodles in #618
• [Router] Improve performance of round-robin router by @zhouwfang in #584
• [Feat] Use sidecar to download lora model for operator deployment by @zerofishnoodles in #622
• [Feature] Add method to check Pod termination status and update Pod readiness logic by @KevinCheung2259 in #602
• [Feat] Add Sentry Continuous Profiling Support to vLLM Router by @ikaadil in #624
• [Feat][Helm] Add HTTPRoute template for Gateway API support by @Hexoplon in #610
• [Feat] Init Container "extraVolumeMount" by @cm-enfuse in #600
• [Bugfix][Router]: simplify test payload by @max-wittig in #613
• [Feat] Add Support HAMi resources variables by @andresd95 in #579
• feature/KV-cache-aware-routing by @BrianPark314 in #550
• [Router] Replace httpx with aiohttp in vllm_router for enhanced high-concurrency performance by @ikaadil in #589
• feature/prefix-aware-routing by @BrianPark314 in #546
• [Feat][Router]: add extra support for YAML config file by @antoineauger in #621
• [CI] Add stress testing for router by @kobe0938 in #633
• [Misc] Auto-size Minikube memory via calculate_safe_memory by @fulvius31 in #637
• [Bugfix][Router]: reconfigure callbacks with dynamic config by @antoineauger in #642
• [Doc] Add a missing word in the description by @JiangJiaWei1103 in #645
• [Bugfix] Correct the routing logic for KV cache aware routing by @JiangJiaWei1103 in #648
• [Router][CI/CD and misc.] Add RoundRobinRouter logic testing by @lucas-tucker in #639
• [Docs] Modify the kvaware routing doc by @zerofishnoodles in #652
• [Router] Optimize request parsing by removing duplicate await calls by @ikaadil in #629
• [Feat][Router] Add configurable timeout_seconds for Kubernetes watchers by @ikaadil in #654
• [Misc] Change community meeting time by @zerofishnoodles in #662
• [Bugfix] Fix install script path prefix by @nicolasj92 in #665
• [Feat] Env from secret by @redno2 in #641
• [Bugfix] Fix routing to delete endpoint by @zerofishnoodles in #668
• Bugfix(vllm-operator): add missing RBAC permissions for PVCs and Ingresses by @mahmoudk1000 in #647
• [feat]: add transcription API endpoint using OpenAI Whisper-small by @davidgao7 in #469
• Bump helm chart version by @philandstuff in #674

New Contributors

• @Fabhiahn made their first contribution in #557
• @ikaadil made their first contribution in #624
• @cm-enfuse made their first contribution in #600
• @andresd95 made their first contribution in #579
• @antoineauger made their first contribution in #621
• @fulvius31 made their first contribution in #637
• @JiangJiaWei1103 made their first contribution in #645
• @nicolasj92 made their first contribution in #665
• @redno2 made their first contribution in #641
• @mahmoudk1000 made their first contribution in #647
• @davidgao7 made their first contribution in #469
• @philandstuff made their first contribution in #674

Full Changelog: vllm-stack-0.1.6...vllm-stack-0.1.7

vllm-stack-0.1.6

The stack deployment of vLLM

What's changed

• [CI]: change the entrypoint of nightly docker images (#514) (by @sammshen )
• Add support for sleep and wake_up endpoints (#498) (by @dumb0002 )
• [Bugfix] add health probe for lmcache server (#520) (by @zerofishnoodles )
• [Doc, Feat] basic KEDA support and tutorials (#487) (by @Romero027 )
• [Misc] Delete Unnecessary file (#521) (by @zerofishnoodles )
• change keda name (#529) (by @zerofishnoodles )
• [CI/CD] Add roundrobin router e2e test (#525) (by @zerofishnoodles )
• [Doc] Add CRD deployment docs (#530) (by @kobe0938 )
• [Doc] Kubernetes in Docker (kind) tutorial (#534) (by @lucas-tucker )
• FEAT introduce ruff to project 1 - tests (#527) (by @BrianPark314 )
• [CI/CD] Add static e2e test for prefixaware (#532) (by @zerofishnoodles )
• fix(request): make sure to extend full_response (#536) (by @max-wittig )
• [CI/CD] Add prefix aware routing test (#523) (by @zerofishnoodles )
• [Bugfix][Helm] prevent duplicate securitycontext entry for containers (#544) (by @Hexoplon )
• feature/gateway-inference-extension (#537) (by @BrianPark314 )
• Add Artifact Hub metadata for verified publisher (#540) (by @kobe0938 )
• [CI/CD] Add multiple routing logic test (#547) (by @zerofishnoodles )
• [Doc] Adding security context for disaggregated prefill (#555) (by @YuhanLiu11 )
• [CI/CD] Add checkov security check for infomation (by @zerofishnoodles )
• fix(reconciler): trigger update when image or replicas are changed (#554) (by @googs1025 )
• [Feat] Terraform Quickstart Tutorials for MS Azure (#552) (by @falconlee236 )
• [Router] Expose /tokenize and /detokenize endpoints (#541) (by @Exchioz )
• feature/ruff-router (#553) (by @BrianPark314 )
• [Doc] Adding tutorial for Gateway Inference Extension support (#570) (@YuhanLiu11 )
• fix: race condition in trie insert (by @zhouwfang )
• [Feature] Moving default vLLM version from v0 to v1 (#580) (@YuhanLiu11 )
• feat(helm): make imagePullPolicy configurable & fix router service annotation for LoadBalancer (#573) (by @lonelygo )
• perf: minimize lock contention (#581) (by @zhouwfang )
• [BugFix] fix lora controller reconcile logic (#565) (by @zerofishnoodles )
• [FEAT] Add LoRA helm deployment (#563) (by @zerofishnoodles )

vllm-stack-0.1.5

The stack deployment of vLLM

vllm-stack-0.1.4

The stack deployment of vLLM

What's changed

• Adding support to route a request to a specific engine instance (#438) @dumb0002
• [Perf] Improve disaggregated prefill router performance (#440) @YuhanLiu11
• [Fix] Only the default namespace service monitor namespace (#447) @nicole-lihui
• update install script kubectl command to find kuberay-operator pod globally (#460) @googs1025
• [Doc] Adding documentation for disaggregated prefill (#477) @YuhanLiu11
• Optimize port conversion (#466) @learner0810
• [Misc] Making KV aware routing compatible with latest LMCache (#475) @YuhanLiu11
• fix(operator): fix cr status base on deployment replicas (#443) @googs1025
• [Misc] Update the request_id handling logic to align with vLLM (#473) @KevinCheung2259
• [CI/Build] Add env clean up before run (#486) @Shaoting-Feng
• [BugFix] Fix v1/models in static discovery (#492) @zerofishnoodles
• Bugfix/482 helm rayspec fix (#483) @insukim1994

vllm-stack-0.1.3

The stack deployment of vLLM

Changes made

• [Feat] add extraVolumes and extraVolumeMounts options @BrianPark314 (#396 )
• [Bugfix] fix(services): make post_request callback not dependent on semantic_cache @ant-ms (#399 )
• [Feat] Support for manual scheduling of a engine pod @dumb0002 (#400 )
• [Bugfix] add miss argument type set @googs1025 (#401)
• [Feat] add sentry sdk and cli args @pwuersch (#395 )
• [Doc] Added documentation about uninstalling previous minikube instal @insukim1994 (#405 )
• [Feat] KV cache aware routing @YuhanLiu11 (#403 )
• [Feat] add event when Reconciling configmap failed @googs1025 (#402 )
• [Misc] Update helm chart for v1 @YuhanLiu11 (#412 )
• [Bugfix] fix(parser): fix dynamic config not working @max-wittig (#413 )
• [feat] add model aliases @max-wittig (#397 )
• [Misc] use schema https://json-schema.org/draft/2020-12/schema @sh1ng (#423 )
• [Feat] Add initial CRD support for production stack @royyhuang (#415 )
• [Feat] Prefix aware routing implementation based on hash trie @KuntaiDu (#432)
• [Feat] Simple Gateway inference extension integration @YuhanLiu11 (#436)
• [Feat] Adding support for disaggregated prefill based on vLLM v1 @YuhanLiu11 (#435)
• refactor: Replace services list with a single service object @googs1025 (#409)
• [Feat][Router] add static-model-types argument @max-wittig (#430 )
• [CI/CD] Adding CI/CD tests for CRDs @YuhanLiu11 (#452 )
• Switch context in CI @Shaoting-Feng (#451)
• chore: add unittest coverage @max-wittig (#449)
• Feat/basic pipeline parallelism @insukim1994 (#422)
• feat: add endpoint health checks to static router @max-wittig (#428)
• [Feat][lora] add lora operator and modify vllm router to support @zerofishnoodles (#446)

vllm-stack-0.1.2

The stack deployment of vLLM

What's Changed

• [Feat] Adding support to turn on/off engine deployment by @dumb0002 #311
• [Feat] Add nodeSelectorTerms for router & cacher servers by @kinoute #314
• [Bugfix] Update logger handler to handle stdout/stderr properly @corona10 #320
• [CI] Always upload logs of Helm functionality checks @pwuersch #321
• [CI/Build] Remove sudo requirements in CI/CD @Shaoting-Feng #325
• [Feat] Multiple service creation when multiple models specified @lucas-tucker #326
• [CI] Add coverage tracking @zhuohangu #330
• [CLI/Doc]Update on gke deployment with gpu quota @EaminC #334
• [Bugfix] Fix thread creation to pass parameters properly. @corona10 #336
• [Feat] OpenTelemetry Support Example @lucas-tucker #346
• [Feat] Tool calling support for MCP client integration @YuhanLiu11 #352
• [Benchmark] Add api key option @Kimdongui #354
• [Bugfix] fix init container pvc volume mount @zerofishnoodles #359
• [Feat] Enabled latency monitor and added average latency computation logic @insukim1994 #362
• [Feat] Added a tutorial document for deploying production stack on amd gpus @insukim1994 #364
• [Bugfix] Deprecated least loaded routing logic @insukim1994 #366
• [Bugfix] added model name to deployment selector @TamKej #367
• [Feat] helm: add routerSpec.serviceType value @marquiz #368
• [Feat] Support Multi-Model Deployment with Enhanced vLLM Configurations @haitwang-cloud #371
• [Bugfix] Fixing issues on the engine svc labels @dumb0002 #376
• [Bugfix] Declare logger properly for protocols.py @corona10 #381
• [Feat] Adding a tutorial for using vLLM v1 in production stack @YuhanLiu11 #390

vllm-stack-0.1.1

The stack deployment of vLLM

What's Changed

• [CI/Build][Router] Make semantic caching optional by @Shaoting-Feng in #218
• [Benchmark] Add router config in tutorial by @Shaoting-Feng in #223
• refactor: standard fastapi project structure for better main… by @BrianPark314 in #217
• Added lora support proposal by @wangchen615 in #216
• [Feat] Added initContainer to modelSpec by @AbelHristodor in #221
• [Router] Fix semantic cache check in chat completion url by @Shaoting-Feng in #224
• [Doc] Change repo in tutorial 08 naive k8s by @Shaoting-Feng in #225
• [Doc] Update community meeting calendar invite by @YuhanLiu11 in #231
• [Doc] Fix startupProbe indentation in values-07 tutorial file by @AbelHristodor in #226
• [Doc] Initial docs structure by @Siddhant-Ray in #234
• [Doc] Update endpoint in 01 tutorial by @Shaoting-Feng in #236
• [Doc] add example page and readme by @Siddhant-Ray in #241
• [Doc] Fix typo of model name and output len in AIBrix by @Shaoting-Feng in #242
• [Doc] Add doc page for benchmark qa by @Siddhant-Ray in #243
• [Doc] add doc on gcp.rst by @EaminC in #249
• [Feat] add vllm-api-key by @JustinDuy in #194
• [CI/Build] Add concurrency to functionality test by @Shaoting-Feng in #219
• [Doc] update tutorial and user manual docs by @Siddhant-Ray in #257
• [Doc] Add docs for router CRD config and dev, some small tweaks by @Siddhant-Ray in #259
• [FEAT] Terraform Quickstart Tutorials for Google GKE by @falconlee236 in #250
• [Feat] add requestGPUType to modelSpec by @Hexoplon in #253
• [Doc][CI/Build] Minor fix by @Shaoting-Feng in #258
• [Doc] dev api docs, bug fixes by @Siddhant-Ray in #266
• [Feat] add explicit resource limit values by @Hexoplon in #255
• [DOC] format unified gcp.rst adding trouble shooting by @EaminC in #263
• [Doc] Minor fix in tutorial by @YuhanLiu11 in #272
• [Doc] Minor fix in benchmarking scripts by @YuhanLiu11 in #273
• [Tutorial] Deployment on Azure AKS by @surajssd in #247
• [Feat] add model label on engine deployments by @Hexoplon in #269
• [Misc] Add schedulerName in servingEngineSpec by @hongkunyoo in #275
• [Feat] Remove sudo requirement for kubectl and helm by @Romero027 in #256
• [Benchmark] Minor fix in benchmark script by @YuhanLiu11 in #284
• [Benchmark] Minor updates to benchmark script by @YuhanLiu11 in #286
• [Doc] Minor fix in tutorials by @YuhanLiu11 in #288
• [Feat] add extraVolume and extraVolumeMount helm variables by @Hexoplon in #280
• Update 09-lora-enabled-installation.md by @wangchen615 in #287
• chore: use extra deps to optionally install additional pkg by @rootfs in #289
• [Feat] Request rewriter interface in router by @ApostaC in #230
• [Feat] add security context to servingEngineSpec by @Hexoplon in #282
• [Doc] add docs link to readme by @Siddhant-Ray in #290
• chore: update e2e test to use python 3.12 to match setup.py requirements by @rootfs in #295
• [CI/Build] Github action for building docs pipeline by @Siddhant-Ray in #291
• Add .readthedocs.yaml by @hmellor in #296
• Hotfix readthedocs build by @hmellor in #298
• Update docs link in README by @hmellor in #299
• feat: support PII detection in http request by @rootfs in #235
• [Bugfix]: add missing v1 prefix by @Xunzhuo in #302
• [Misc] Bumping version to 0.1.1 by @YuhanLiu11 in #308

New Contributors

• @AbelHristodor made their first contribution in #221
• @JustinDuy made their first contribution in #194
• @falconlee236 made their first contribution in #250
• @Hexoplon made their first contribution in #253
• @surajssd made their first contribution in #247
• @hongkunyoo made their first contribution in #275
• @Romero027 made their first contribution in #256
• @Xunzhuo made their first contribution in #302

Full Changelog: vllm-stack-0.1.0...vllm-stack-0.1.1