Skip to content

feat(tests): e2e #279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: features/scheduler_refactor
Choose a base branch
from

Conversation

AlonKellner-RedHat
Copy link

Summary

These are E2E tests that worked on a previous fork of mine, but with minor modification should work for any version.
Since these are copied from a previous PR, I will copy-paste that description now:

By default, these will check if the vLLM simulator is available in the local environment, if not - they are skipped and log a warning with the command required to build the vLLM simulator.
The vLLM simulator is built from source in the tests/e2e/vllm-sim.Dockerfile and extracted to bin/llm-d-inference-sim.
The tests use a class VllmSimServer that wraps the vLLM simulator with start and stop methods, then they call the guidellm command and assert the values generated in the output report.
The 2 test modules have varying run times:

test_successful_benchmark (2 tests) - 14.12s
test_max_error_benchmark (1 test) - 15.98s
The max error test takes longer since it allows guidellm to start sending requests before killing the vLLM sim, and then waits for the guidellm command to stop gracefully.
The successful tests share a single vLLM sim instance, hence are more efficient.

Details

  • A vllm-simulator binary builder (dockerfile)
  • A vllm-simulator-server class for tests
  • 2 happy flow tests (the simulator server is always up)
  • 1 faulty flow test (the simulator server crashes mid-test)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant