Skip to content

Commit c97e2ea

Browse files
vMaroonnirrozenbaum
authored andcommitted
Update architecture.md (#264)
1 parent 7598e42 commit c97e2ea

File tree

1 file changed

+3
-2
lines changed

1 file changed

+3
-2
lines changed

docs/architecture.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,7 @@ See list of parameters at [llm-d-kv-cache-manager/docs/configuration.md](https:/
272272

273273
Note that in most cases you will only need to set:
274274
- Hugging Face token for the `tokenizersPoolConfig` or the `tokenizersCacheDir` to a mounted directory containing the tokenizers.
275+
- For the HuggingFace token, the inference-scheduler also accepts the environment variable `HF_TOKEN` - this is the practical option for security.
275276
- IMPORTANT: Token processor's block-size and hash-seed to match those used in the vLLM deployment.
276277
- KVBlockIndex metrics to true if you wish to enable metrics for the KV-Block Index (admissions, evictions, lookups and hits).
277278

@@ -285,7 +286,7 @@ plugins:
285286
blockSize: 64
286287
hashSeed: "12345"
287288
tokenizersPoolConfig:
288-
huggingFaceToken: your_hf_token_here
289+
huggingFaceToken: your_hf_token_here # automatically set by `HF_TOKEN` environment variable
289290
kvBlockIndexConfig:
290291
enableMetrics: true
291292
```
@@ -313,7 +314,7 @@ plugins:
313314
enableMetrics: true
314315
tokenizersPoolConfig:
315316
workersCount: 8
316-
huggingFaceToken: your_hf_token_here
317+
huggingFaceToken: your_hf_token_here # automatically set by `HF_TOKEN` environment variable
317318
tokenizersCacheDir: /tmp/tokenizers
318319
```
319320

0 commit comments

Comments
 (0)