Repositories list

• llm-inference-handbook

  Public
  Everything you need to know about LLM inference

  inference-optimizationllmllm-inference+ 2

  inference-optimizationllmllm-inferenceinference-handbookinference-infrastructure

  TypeScript

  •

  Apache License 2.0

  •22•269•3•1•Updated Mar 10, 2026Mar 10, 2026

• OpenLLM

  Public
  Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

  llamamistralfine-tuning+ 15

  llamamistralfine-tuningmlopsbentomlvicunallmmodel-inferencellmopsllm-serving+ 8

  Python

  •

  Apache License 2.0

  •802•12k•4•7•Updated Mar 9, 2026Mar 9, 2026

• BentoML

  Public
  The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

  pythonmachine-learningdeep-learning+ 12

  pythonmachine-learningdeep-learningmodel-servingmultimodalmlopsml-engineeringai-inferencellmgenerative-ai+ 5

  Python

  •

  Apache License 2.0

  •919•8.5k•137•7•Updated Mar 9, 2026Mar 9, 2026

• yatai-common

  Public
  Go

  •6•1•0•2•Updated Mar 7, 2026Mar 7, 2026

• sglang

  Public
  SGLang is a fast serving framework for large language models and vision language models.

  Python

  •

  Apache License 2.0

  •4.8k•0•0•1•Updated Mar 5, 2026Mar 5, 2026

• helm-charts

  Public
  0•3•0•0•Updated Mar 5, 2026Mar 5, 2026

• yatai-image-builder

  Public
  🐳 Build OCI images for Bentos in k8s

  dockerkubernetesdocker-image+ 3

  dockerkubernetesdocker-imagecontainerk8scontainer-image

  Go

  •12•20•6•13•Updated Mar 5, 2026Mar 5, 2026

• openai_emulator

  Public

  bentoml/openai_emulator’s past year of commit activity
  Python

  •

  Apache License 2.0

  •0•0•0•0•Updated Mar 5, 2026Mar 5, 2026

• .github

  Public
  ✨🍱🦄️

  5•1•0•0•Updated Mar 5, 2026Mar 5, 2026

• ai-gateway

  Public

  bentoml/ai-gateway’s past year of commit activity
  Manages Unified Access to Generative AI Services built on Envoy Gateway

  Go

  •

  Apache License 2.0

  •184•0•0•2•Updated Mar 4, 2026Mar 4, 2026

• BentoSGLang

  Public
  Python

  •1•2•0•1•Updated Mar 3, 2026Mar 3, 2026

• BentoVLLM

  Public
  Self-host LLMs with vLLM and BentoML

  Python

  •

  Apache License 2.0

  •22•170•3•5•Updated Mar 3, 2026Mar 3, 2026

• bentocloud-homepage-news

  Public
  2•2•0•0•Updated Feb 10, 2026Feb 10, 2026

• BentoSpacy

  Public
  Python

  •0•0•0•2•Updated Jan 26, 2026Jan 26, 2026

• BentoTRTLLM

  Public
  Python

  •1•10•2•2•Updated Jan 22, 2026Jan 22, 2026

• mise-nebius

  Public
  Shell

  •

  MIT License

  •0•0•0•0•Updated Jan 22, 2026Jan 22, 2026

• BentoLMDeploy

  Public
  Self-host LLMs with LMDeploy and BentoML

  Python

  •2•22•1•1•Updated Dec 26, 2025Dec 26, 2025

• BentoFunctionCalling

  Public
  Python

  •1•9•0•2•Updated Dec 26, 2025Dec 26, 2025

• lago

  Public
  Open Source Metering and Usage Based Billing API ⭐️ Consumption tracking, Subscription management, Pricing iterations, Payment orchestration & Revenue analytics

  Go

  •

  GNU Affero General Public License v3.0

  •543•0•0•1•Updated Nov 20, 2025Nov 20, 2025

• comfy-pack

  Public
  A comprehensive toolkit for reliably locking, packing and deploying environments for ComfyUI workflows.

  ai-artcomfyuigenai+ 4

  ai-artcomfyuigenaicomfyui-managercomfyui-workflowcomfyui-nodescomfyui-api

  Python

  •

  Apache License 2.0

  •29•209•11•2•Updated Nov 10, 2025Nov 10, 2025

• BentoOCR

  Public
  Turn any OCR models into online inference API endpoint 🚀 🌖

  ocrmodel-deploymentmodel-serving+ 2

  ocrmodel-deploymentmodel-servingocr-pythonai-applications

  Python

  •4•59•1•0•Updated Oct 29, 2025Oct 29, 2025

• pipelines-workflows

  Public
  Gruntwork Pipelines for BentoML

  25•0•0•0•Updated Oct 23, 2025Oct 23, 2025

• lago-go-client

  Public
  Lago Go Client

  Go

  •

  MIT License

  •23•0•0•0•Updated Oct 15, 2025Oct 15, 2025

• BentoYolo

  Public
  BentoML service of YOLO v8

  Python

  •

  Apache License 2.0

  •4•12•1•1•Updated Sep 17, 2025Sep 17, 2025

• llm-optimizer

  Public
  Benchmark and optimize LLM inference across frameworks with ease

  Python

  •

  Apache License 2.0

  •17•171•10•0•Updated Sep 12, 2025Sep 12, 2025

• kantoku

  Public
  A Process & Socket Manager built with zmq

  Python

  •

  Other

  •261•1•0•0•Updated Aug 28, 2025Aug 28, 2025

• byoc

  Public
  Shell

  •1•0•0•2•Updated Aug 28, 2025Aug 28, 2025

• BentoProphet

  Public
  BentoML with Facebook Prophet

  Python

  •0•0•0•0•Updated Aug 27, 2025Aug 27, 2025

• XGBoostDemo

  Public
  Python

  •0•2•0•0•Updated Aug 19, 2025Aug 19, 2025

• asynq

  Public
  Simple, reliable, and efficient distributed task queue in Go

  Go

  •

  MIT License

  •909•0•0•0•Updated Aug 5, 2025Aug 5, 2025