Auth issue with Example 06_gpu_and_ml/llm-serving/vllm_inference.py

[tranhoangnguyen03]

The interactive Swagger UI docs at the /docs route of that URL, i.e. https://your-workspace-name--example-vllm-openai-compatible-serve.modal.run/docs does not work due to authorization issue.

Removing "--api-key", API_KEY, from the vllm serve command doesn't solve the authorization issue either.

[charlesfrye]

Thanks for reporting. What is the error you're seeing?

[tranhoangnguyen03]

Simply trying to get the model name results in error. But there's no place in the SwaggerUI to input the api_key or bearer token.

curl -X 'GET' \
  'https://tranhoangnguyen03--example-vllm-openai-compatible-serve.modal.run/v1/models' \
  -H 'accept: application/json'

{
  "error": "Unauthorized"
}

[charlesfrye]

Ah I believe that's an underlying vLLM issue -- they aren't exposing the auth in the OpenAPI spec. I will raise it with them. If you look at old versions of the code, you can see how to properly expose that information if you manually construct the ASGI/Fastapi app in your code. There's a lot more code and it changes per vLLM version, but it's more flexible.

…

On Thu, Mar 20, 2025, 9:49 AM Tran Hoang Nguyen ***@***.***> wrote: Simply trying to get the model name results in error. But there's no place in the SwaggerUI to input the api_key or bearer token. curl -X 'GET' \ 'https://tranhoangnguyen03--example-vllm-openai-compatible-serve.modal.run/v1/models' \ -H 'accept: application/json' { "error": "Unauthorized" } — Reply to this email directly, view it on GitHub <#1105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPVRX5Z757X2YJ4ZSQBMUL2VLWS7AVCNFSM6AAAAABZN5NOTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBRGA4TCMZWGY> . You are receiving this because you commented.Message ID: ***@***.***> [image: tranhoangnguyen03]*tranhoangnguyen03* left a comment (modal-labs/modal-examples#1105) <#1105 (comment)> Simply trying to get the model name results in error. But there's no place in the SwaggerUI to input the api_key or bearer token. curl -X 'GET' \ 'https://tranhoangnguyen03--example-vllm-openai-compatible-serve.modal.run/v1/models' \ -H 'accept: application/json' { "error": "Unauthorized" } — Reply to this email directly, view it on GitHub <#1105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPVRX5Z757X2YJ4ZSQBMUL2VLWS7AVCNFSM6AAAAABZN5NOTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBRGA4TCMZWGY> . You are receiving this because you commented.Message ID: ***@***.***>

[tranhoangnguyen03]

also, your example doesn't work unless secrets=[modal.Secret.from_name("my-huggingface-secret")] is added to the @app.function decorator.

[charlesfrye]

Did you change the model? You should only need a Secret to access gated models, but the default model is not gated.