Llama Stack “critical path”

[reluctantfuturist]

Goal: identify a subset of Llama Stack features needed for our key partners

Features

Core APIs

All APIs below should be:

• OpenAI-compatible
• Well-documented
• Tested (see below for the provider testing matrix)

[P0] Core APIs:

• Inference
  • vllm
  • ollama
  • TGI
• Safety
  • llama-guard3
  • llama-guard4
• Telemetry
  • Meta reference (with jaeger)
• Vector stores
  • Faiss
  • sqlite
  • Pgvector
  • Chroma
  • Qdrant
  • Milvus

[P0] API conformance tests

• test: introduce api conformance test #3257

[P0] Responses API with support for:

• [P0] File search with the following formats:
  • Text
  • PDF
• [P0] Streaming (response + tool calling)
• [P0] Document which tools are available
• [P0] Non-OpenAI-compatible API deprecation
• [P1] Log usage metrics
• [P1] Reasoning tokens

[P0] Conversations API

• Conversations API #3310
• [P0] Agents API deprecation

Auth

• [P0] Support authentication (Support authentication #2373)
• [P0] Allow updating resource attributes in the Auth API / ABAC structure (Allow updating resource attributes in the Auth API / ABAC structure #2374)

Developer-facing UI

• [P0] Playground
  • Chat completions
  • Responses
  • RAG

Observability

• [P0] Auditing: All API calls must be logged via Telemetry and be queryable efficiently (Auditing functionality #2377)

CI/ID

• [P0] Enable fast nightly runs
• [P0] Enable nightly builds
• [P0] Add tests for all P0 APIs that do not have tests
• [P0] Integration test for run.yaml #3311
• [P1] Update our k8s cluster nightly

[P0] Provider combos tested E2E:

• (k8s) vllm + postgres + ChromaDB + Jaeger (OCI distro)
• (local machine) Ollama + sqlite + faiss

Performance measurement

• [P0] Benchmarking the following performance metrics: Performance testing #2384
  • Latency
  • Throughput
  • TTFT

Documentation

• [P0] Frameworks: Langchain feat: Added llama stack-langchain integration example scripts #3211
• [P0] Frameworks: Crew feat: add llamastack + CrewAI integration example notebook #3275
• [P0] llamastack.github.io update
• [P0] Document stable APIs
• [P0] Document how-to for using the starter Docker container
• [P0] Document how to configure/start a Llama Stack server

P1 -- below the cut line

• [P1] Vector stores: keyword/hybrid search
• [P1] Unified tool API
• [P1] MCP support for resources and prompts (feat: Add apis and support for Model Context Protocol resources and prompts #1567)

UI

• [P1] Developer dashboard
  • Provider configuration
  • API key management

Admin-facing UI

• [P0] API key management (API key management for partners #2376)

• [P0] Admin dashboard

  • Manage users, API keys and access controls
  • Configure app deployment and monitoring
  • Auditing UI

• [P1] Additional metrics for telemetry (Observability: Add Additional Metrics to Llama Stack Telemetry #2596)

• [P1] Test docker containers for 1-2 popular setups