Skip to content

Simplify and revise vLLM example#1076

Merged
charlesfrye merged 1 commit intomainfrom
cweld/revise-vllm-example
Feb 20, 2025
Merged

Simplify and revise vLLM example#1076
charlesfrye merged 1 commit intomainfrom
cweld/revise-vllm-example

Conversation

@cweld510
Copy link
Copy Markdown
Contributor

@cweld510 cweld510 commented Feb 19, 2025

Updates and drastically simplifies vLLM example to use @web_server and spawn vLLM as a subprocess.

To test this: modal serve followed by python 06_gpu_and_ml/llm-serving/openai_compatible/client.py --app example-vllm-openai-compatible --function serve-dev --api-key super-secret-key

Type of Change

  • New example
  • Example updates (Bug fixes, new features, etc.)
  • Other (changes to the codebase, but not to examples)

Checklist

  • Example is testable in synthetic monitoring system, or lambda-test: false is added to example frontmatter (---)
    • Example is tested by executing with modal run or an alternative cmd is provided in the example frontmatter (e.g. cmd: ["modal", "deploy"])
    • Example is tested by running with no arguments or the args are provided in the example frontmatter (e.g. args: ["--prompt", "Formula for room temperature superconductor:"]
  • Example is documented with comments throughout, in a Literate Programming style.
  • Example does not require third-party dependencies to be installed locally
  • Example pins its dependencies
    • Example pins container images to a stable tag, not a dynamic tag like latest
    • Example specifies a python_version for the base image, if it is used
    • Example pins all dependencies to at least minor version, ~=x.y.z or ==x.y
    • Example dependencies with version < 1 are pinned to patch version, ==0.y.z

@cweld510 cweld510 force-pushed the cweld/revise-vllm-example branch from 628c1bf to 8452d64 Compare February 19, 2025 22:59
@charlesfrye
Copy link
Copy Markdown
Collaborator

🚀 The docs preview is ready! Check it out here: https://modal-labs-examples--frontend-preview-32a7d71.modal.run

@cweld510 cweld510 force-pushed the cweld/revise-vllm-example branch from 78af299 to d068e64 Compare February 19, 2025 23:04
@charlesfrye
Copy link
Copy Markdown
Collaborator

🚀 The docs preview is ready! Check it out here: https://modal-labs-examples--frontend-preview-c2ce8a4.modal.run

@charlesfrye
Copy link
Copy Markdown
Collaborator

🚀 The docs preview is ready! Check it out here: https://modal-labs-examples--frontend-preview-d60df30.modal.run

@charlesfrye
Copy link
Copy Markdown
Collaborator

🚀 The docs preview is ready! Check it out here: https://modal-labs-examples--frontend-preview-c08a2ca.modal.run

@charlesfrye
Copy link
Copy Markdown
Collaborator

nice work!

@charlesfrye charlesfrye merged commit dc662fd into main Feb 20, 2025
7 checks passed
@charlesfrye charlesfrye deleted the cweld/revise-vllm-example branch February 20, 2025 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants