[CI Failure]: Entrypoints Integration Test (API Server) flake in entrypoints/openai/test_transcription_validation.py::test_basic_audio_gemma

[bbrowning]

Name of failing test

entrypoints/openai/test_transcription_validation.py::test_basic_audio_gemma

Basic information

• Flaky test
• Can reproduce locally
• Caused by external libraries (e.g. bug in transformers)

🧪 Describe the failing test

On an unrelated PR, I caught a flake of this test with:

FAILED entrypoints/openai/test_transcription_validation.py::test_basic_audio_gemma - RuntimeError: Server failed to start in time.

It looks like the server timed out with the default 240s timeout while attempting to load this model. The logs indicate forward progress was being made, albeit slower than expected. Perhaps this test just needs a longer max_wait_seconds to be passed in as an additional arg to the RemoteOpenAIServer initializer?

📝 History of failing test

I cannot see the Buildkit Test Suites feature - perhaps it is not public? But, here's a link to one place this flaked on a PR I had open - https://buildkite.com/vllm/ci/builds/36258/steps/canvas?jid=019a1a38-3f40-423e-9f36-7248c8d7b84e#019a1a38-3f40-423e-9f36-7248c8d7b84e/7-13138

This test was added in #23735

CC List.

cc @NickLucche as the original author of the test in question