Skip to content

Refactor evaluation module and add unit tests#1325

Closed
Copilot wants to merge 1 commit intoxinhe/evalfrom
copilot/sub-pr-1324
Closed

Refactor evaluation module and add unit tests#1325
Copilot wants to merge 1 commit intoxinhe/evalfrom
copilot/sub-pr-1324

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Jan 23, 2026

Refactored evaluation functionality into a dedicated auto_round/eval module and added comprehensive unit tests for CPU and GPU backends.

Changes

Module Structure

  • auto_round/eval/eval_cli.py (486 lines): CLI argument parsing, vLLM/HF backend orchestration, custom argument parsing
  • auto_round/eval/evaluation.py (439 lines): Core evaluation wrappers for lm_eval library
  • auto_round/eval/__init__.py: Module exports

Unit Tests

  • test/test_cpu/advanced/test_evaluation_functions.py (121 lines): Tests for parse_vllm_args() with various types (int, float, bool, string, mixed)
  • test/test_cuda/advanced/test_evaluation.py (115 lines): Integration tests for vLLM backend with custom args, quantization workflows

Compatibility & Optimization

  • Added allow_deprecated_quantization: True for vLLM 0.14.0 compatibility
  • Removed redundant test cases to reduce CI execution time
  • Added max_model_len=2048 parameter to vLLM tests for faster execution

Example Usage

from auto_round.eval.eval_cli import parse_vllm_args

# Parse custom vLLM arguments
args = parse_vllm_args("--tensor_parallel_size=2,--gpu_memory_utilization=0.9")
# Returns: {'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.9}

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI mentioned this pull request Jan 23, 2026
18 tasks
Copilot AI changed the title [WIP] Refactor eval function and add unit tests Refactor evaluation module and add unit tests Jan 23, 2026
Copilot AI requested a review from xin3he January 23, 2026 02:30
@xin3he xin3he closed this Jan 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants