Update vLLM GPU Utilization

[dsikka]

SUMMARY:

• Appers as if oneshot memory is not freed resulting in gpu_memory_utilization errors with vLLM which requires 90% of the gpu to launch the engine
• We can set this value to something lower but likely requires some investigation as was not previously needed
  cc @kylesayrs

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[gemini-code-assist]

Summary of Changes

Hello @dsikka, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on refining configuration settings across various model evaluation and vLLM test configurations. It includes a fix for an incorrect save_dir path, introduces explicit GPU memory utilization settings for several vLLM tests, and adjusts recovery thresholds in an lmeval configuration, aiming to improve the accuracy and resource management of model evaluations.

Highlights

• Configuration Correction: A typo in the save_dir path for the w4a16_actorder_none vLLM configuration has been corrected, changing 'group' to 'none'.
• GPU Memory Utilization: New gpu_memory_utilization parameters have been added to several vLLM configurations, specifically qwen3_fp8_dynamic_per_token.yaml, w4a16_grouped_quant_asym_awq.yaml, and w4a16_grouped_quant_sym_awq.yaml, to better manage resource allocation.
• Evaluation Threshold Adjustment: The recovery_threshold values for exact_match,strict-match and exact_match,flexible-extract in the w4a16_awq_sym lmeval configuration have been updated from 0.92 and 0.93 respectively, to 0.91.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates several e2e test configuration files. The main change corrects a save_dir path to align with its corresponding quantization scheme. Additionally, gpu_memory_utilization settings are introduced in some configurations, and recovery thresholds are adjusted. The changes appear to be routine updates and fixes. I've provided a few suggestions to add a final newline to some of the modified YAML files to align with best practices and improve consistency.

I am having trouble creating individual review comments. Click here to see my feedback.

tests/e2e/vLLM/configs/qwen3_fp8_dynamic_per_token.yaml (5)

It's a good practice to end files with a newline character. Some tools and standards (like POSIX) expect it, and it improves consistency across the codebase.

gpu_memory_utilization: 0.55

tests/e2e/vLLM/configs/w4a16_actorder_none.yaml (9)

Please add a newline at the end of the file. It's a common convention that helps prevent issues with file processing tools and git.

save_dir: TinyLlama-1.1B-Chat-v1.0-actorder-none

tests/e2e/vLLM/configs/w4a16_grouped_quant_asym_awq.yaml (9)

This file is missing a final newline. Adding one is a standard practice that ensures file integrity and compatibility with various development tools.

gpu_memory_utilization: 0.85

[kylesayrs]

I believe this should be fixed by #2375