File tree Expand file tree Collapse file tree 1 file changed +3
-2
lines changed Expand file tree Collapse file tree 1 file changed +3
-2
lines changed Original file line number Diff line number Diff line change @@ -272,6 +272,7 @@ See list of parameters at [llm-d-kv-cache-manager/docs/configuration.md](https:/
272
272
273
273
Note that in most cases you will only need to set :
274
274
- Hugging Face token for the `tokenizersPoolConfig` or the `tokenizersCacheDir` to a mounted directory containing the tokenizers.
275
+ - For the HuggingFace token, the inference-scheduler also accepts the environment variable `HF_TOKEN` - this is the practical option for security.
275
276
- IMPORTANT : Token processor's block-size and hash-seed to match those used in the vLLM deployment.
276
277
- KVBlockIndex metrics to true if you wish to enable metrics for the KV-Block Index (admissions, evictions, lookups and hits).
277
278
@@ -285,7 +286,7 @@ plugins:
285
286
blockSize: 64
286
287
hashSeed: "12345"
287
288
tokenizersPoolConfig:
288
- huggingFaceToken: your_hf_token_here
289
+ huggingFaceToken: your_hf_token_here # automatically set by ` HF_TOKEN` environment variable
289
290
kvBlockIndexConfig :
290
291
enableMetrics : true
291
292
` ` `
@@ -313,7 +314,7 @@ plugins:
313
314
enableMetrics : true
314
315
tokenizersPoolConfig :
315
316
workersCount : 8
316
- huggingFaceToken: your_hf_token_here
317
+ huggingFaceToken : your_hf_token_here # automatically set by `HF_TOKEN` environment variable
317
318
tokenizersCacheDir : /tmp/tokenizers
318
319
` ` `
319
320
You can’t perform that action at this time.
0 commit comments