-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy path.env.example
More file actions
160 lines (143 loc) · 8.58 KB
/
Copy path.env.example
File metadata and controls
160 lines (143 loc) · 8.58 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
# Copy to deploy/.env and adjust. Consumed by deploy/docker-compose.yaml.
# HuggingFace token for gated/private models (leave blank for public models).
HF_TOKEN=
# Host directory for the HF weight cache (bind-mounted into the engine at /hf).
# Leave unset to use the standard host HF cache (~/.cache/huggingface) so weights
# are browsable locally and shared with host-side tools. To override, give an
# ABSOLUTE path (e.g. a bigger disk) — a literal ~ or ${HOME} won't expand here.
# HF_CACHE_DIR=/mnt/models/hf-cache
# Host directory for the ModelScope cache (benchmark datasets: ShareGPT, openqa…),
# bind-mounted at /modelscope. Same rules as HF_CACHE_DIR (absolute path only).
# MODELSCOPE_CACHE_DIR=/mnt/models/modelscope-cache
# Which GPUs the engine container may use: "all", or a comma list e.g. "0,1".
NVIDIA_VISIBLE_DEVICES=all
# Host ports (the browser only needs FRONTEND_PORT; the others are for direct
# API access and can be remapped if 5000/8887/9090 are already taken on the host).
FRONTEND_PORT=8884
BACKEND_PORT=5000
ROUTER_PORT=8887
# Router concurrency: number of gunicorn workers in the router container. The
# router only proxies (IO-bound), so raising this is the simplest way to scale
# routing throughput on one host — each worker is a stateless replica sharing the
# same store. 1 is fine for a single GPU (the GPU, not the router, is the limit);
# bump it when fronting many backends. (True multi-container/multi-host routers
# additionally need vLLM bound to a routable address — see ha-phase3 design.)
ROUTER_WORKERS=1
# Prometheus UI / API (Phase 1 monitoring; scrapes the vLLM fleet's /metrics).
PROMETHEUS_PORT=9090
# ---- Authentication --------------------------------------------------------
# Shared admin token. Gates all control/write operations (start/stop/add/edit/
# remove + API-key management) on the backend, and is accepted by the router as
# an inference credential so the dashboard keeps working. Leave BLANK to disable
# auth (dev only: control endpoints are then fully open). Use a long random value.
LLMOPS_ADMIN_TOKEN=
# When true, the router rejects inference (/v1/*) without a valid bearer token —
# either the admin token above or an API key minted on the dashboard's "API 金鑰"
# page. Leave false/blank to keep inference open.
LLMOPS_REQUIRE_API_KEY=false
# ---- Lifecycle alerting (backend Notifier) --------------------------------
# Discrete model-lifecycle alerts (crash, restart-budget exhausted, recovered),
# complementing Grafana's metric alerts. Any subset of sinks; all blank ->
# alerting disabled. Slack/Discord get formatted messages, the generic webhook
# gets raw JSON. More destinations can be added at runtime in the admin
# Notifications page. See docs/alerting-design_zh-CN.md.
LLMOPS_ALERT_WEBHOOK=
LLMOPS_ALERT_SLACK_WEBHOOK=
LLMOPS_ALERT_DISCORD_WEBHOOK=
# Default severity floor (info|warning|error|critical) — applied to the env
# sinks above and offered as the default for sinks added in the UI. It is NOT a
# global gate: each sink decides by its own floor, so a UI sink set to `info`
# really does receive info events (e.g. model_recovered). Plus a per-(model,event)
# cooldown (seconds) so a crash-loop can't spam the channel.
LLMOPS_ALERT_MIN_SEVERITY=error
LLMOPS_ALERT_COOLDOWN_S=300
# ---- Shared store backend (HA) ---------------------------------------------
# Where the shared store (telemetry, keys, audit, versions, cost…) lives.
# Leave blank for the zero-config default: a single SQLite file shared by the
# backend + router. Set a Postgres DSN to use a networked DB instead — required
# once you run more than one backend replica (HA). Must be identical for every
# backend/router process. See docs/ha-phase2-design_zh-CN.md.
# LLMOPS_DB_URL=postgresql://user:pass@postgres:5432/llmops
# Leader election (only with Postgres): only the lease holder runs the singleton
# control loops (reconcile/autoscale/prune); a peer steals an expired lease within
# ~TTL seconds. INSTANCE_ID identifies this replica (default hostname:pid).
LLMOPS_LEADER_LEASE_TTL=15
# LLMOPS_INSTANCE_ID=backend-1
# In multi-node deploys give each host a STABLE id (the default hostname:pid
# changes every restart, which churns node/assignment bookkeeping).
# ---- Multi-node / split deploys (HA Phase 3) -------------------------------
# Single-host collapsed deploys leave these blank — vLLM binds localhost in the
# backend's netns and routers reach it there, exactly as today. To run routers /
# node-agents in a different netns or on another host, set BOTH:
# - VLLM_BIND_HOST: the interface vLLM listens on (0.0.0.0 = all). Without this
# vLLM only answers on localhost and an off-host router can't connect.
# - NODE_HOST: the routable address this node advertises for its instances in
# the shared store (what routers actually connect to — a hostname/IP, NOT
# 0.0.0.0). Leave blank to advertise the instance's configured host.
# See docs/ha-phase3-design_zh-CN.md. (0.0.0.0 exposes vLLM beyond localhost —
# only use it on a trusted/internal network.)
# LLMOPS_VLLM_BIND_HOST=0.0.0.0
# LLMOPS_NODE_HOST=10.0.0.21
# On boot, restart models whose last desired state was "running" but which aren't
# (after a backend restart, or an HA replica taking over). Persisted across
# restarts via the shared store. Set false on a dev box that shouldn't auto-start
# models on every restart.
LLMOPS_REPLAY_DESIRED=true
# ---- Graceful drain (HA) ---------------------------------------------------
# Before killing an instance on stop / scale-down, the backend tells the router
# to send it no new requests and waits for in-flight ones to finish — so a stop
# or rolling update doesn't cut live requests. Returns early once in-flight hits
# 0; capped at DRAIN_TIMEOUT seconds (0 disables draining).
LLMOPS_DRAIN_TIMEOUT=30
LLMOPS_DRAIN_POLL_INTERVAL=1
# ---- SSO / OIDC login (optional) -------------------------------------------
# Let people sign in with a corporate IdP (Google / Entra / Okta / any OIDC
# provider) instead of pasting a token. Humans get a session cookie; machines /
# CI keep using operator tokens & API keys. Leave LLMOPS_OIDC_ISSUER blank to
# disable SSO entirely (behaviour unchanged). Setting it up CLOSES the open
# local-dev backdoor, so auth becomes required. See docs/sso-design_zh-CN.md.
#
# Register an app with your IdP; set the redirect/callback URL to
# https://<your-host>/api/auth/sso/callback
LLMOPS_OIDC_ISSUER=
LLMOPS_OIDC_CLIENT_ID=
LLMOPS_OIDC_CLIENT_SECRET=
# Optional: override the auto-derived callback (only if behind an odd proxy).
# LLMOPS_OIDC_REDIRECT_URL=https://host/api/auth/sso/callback
# Scopes + which claim holds the user's groups (Entra often uses 'roles').
LLMOPS_OIDC_SCOPES=openid email profile
LLMOPS_OIDC_GROUPS_CLAIM=groups
# Role mapping: admin emails win; then group membership; else the default role.
# Comma-separated. Set LLMOPS_OIDC_DEFAULT_ROLE="" to DENY anyone unmatched.
LLMOPS_OIDC_ADMIN_EMAILS=
LLMOPS_OIDC_ADMIN_GROUPS=
LLMOPS_OIDC_OPERATOR_GROUPS=
LLMOPS_OIDC_VIEWER_GROUPS=
LLMOPS_OIDC_DEFAULT_ROLE=viewer
# Session cookie signing key (REQUIRED in prod — a random key means every restart
# logs everyone out). Use a long random value; shared across backend replicas.
LLMOPS_SESSION_SECRET=
LLMOPS_SESSION_TTL=28800
# ---- Cost dashboard --------------------------------------------------------
# The Cost page prices token usage with a per-model price table (set in the UI,
# admin-only). These are the FALLBACK per-1M-token prices for models without an
# explicit row, plus the display currency. Leave at 0 to show cost as 0 until
# you set prices. Prices are per 1,000,000 tokens, split input / output.
LLMOPS_DEFAULT_INPUT_PRICE=0
LLMOPS_DEFAULT_OUTPUT_PRICE=0
LLMOPS_PRICE_CURRENCY=USD
# ---- Config versioning -----------------------------------------------------
# Every runtime config change (the dynamic-model overlay) is snapshotted so it
# can be exported/imported and rolled back from the admin "Config Versions" page.
# Cap on retained snapshots; the oldest are pruned hourly past this.
LLMOPS_CONFIG_VERSIONS_MAX=500
# ---- Monitoring (Grafana) --------------------------------------------------
# Grafana is served at http://<host>:FRONTEND_PORT/grafana (single origin, via
# nginx). Anonymous access is read-only; this password logs in the `admin` user
# for editing dashboards. Change it for any non-local deployment.
GRAFANA_ADMIN_PASSWORD=admin
# Webhook URL the provisioned vLLM alert rules notify (generic JSON POST; works
# with Slack/Discord incoming webhooks or any endpoint). Leave blank to keep a
# placeholder (alerts still show in Grafana's UI but send nowhere); set a real
# URL and restart grafana to receive notifications.
GRAFANA_ALERT_WEBHOOK=