Skip to content

Conversation

@xin3he
Copy link
Contributor

@xin3he xin3he commented Jan 5, 2026

User description

We observe high accuracy improvement when lm_eval upgrades 0.4.9.1 -> 0.4.9.2


PR Type

Enhancement


Description

  • Update dependencies for improved accuracy

  • Upgrade lm_eval from 0.4.9.1 to 0.4.9.2

  • Update transformers, torch, torchvision, datasets, auto-round, and neural-compressor-pt to latest versions


Diagram Walkthrough

flowchart LR
  A["Update lm_eval"] -- "0.4.9.1 -> 0.4.9.2" --> B["Improved Accuracy"]
  C["Upgrade transformers"] -- "4.56.2 -> 4.57.3" --> B
  D["Upgrade torch"] -- "2.7.0 -> 2.9.0" --> B
  E["Upgrade torchvision"] -- "0.22.0 -> 0.24.0" --> B
  F["Upgrade datasets"] -- "3.6.0 -> 4.4.2" --> B
  G["Upgrade auto-round"] -- ">=0.8.0 -> >=0.9.3" --> B
  H["Upgrade neural-compressor-pt"] -- ">=3.6 -> >=3.7" --> B
Loading

File Walkthrough

Relevant files
Enhancement
requirements.txt
Update dependencies for improved accuracy                               

examples/pytorch/nlp/huggingface_models/language-modeling/quantization/auto_round/llama3/requirements.txt

  • Upgrade transformers to 4.57.3
  • Upgrade torch to 2.9.0
  • Upgrade torchvision to 0.24.0
  • Upgrade lm_eval to 0.4.9.2
  • Upgrade datasets to 4.4.2
  • Upgrade auto-round to >=0.9.3
  • Upgrade neural-compressor-pt to >=3.7
+7/-8     

@PRAgent4INC
Copy link
Collaborator

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Dependency

The dependency deepspeed is missing in the updated requirements.txt. Ensure that all necessary dependencies are included.

transformers==4.57.3
torch==2.9.0
torchvision==0.24.0
lm_eval==0.4.9.2
datasets==4.4.2
auto-round>=0.9.3
neural-compressor-pt>=3.7

@PRAgent4INC
Copy link
Collaborator

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
General
Check package compatibility

Verify compatibility of updated packages with the existing codebase.

examples/pytorch/nlp/huggingface_models/language-modeling/quantization/auto_round/llama3/requirements.txt [1-7]

-transformers==4.56.2
-torch==2.7.0
-torchvision==0.22.0
-lm_eval==0.4.9.1
-datasets==3.6.0
-auto-round>=0.8.0
-neural-compressor-pt>=3.6
+transformers==4.57.3
+torch==2.9.0
+torchvision==0.24.0
+lm_eval==0.4.9.2
+datasets==4.4.2
+auto-round>=0.9.3
+neural-compressor-pt>=3.7
Suggestion importance[1-10]: 5

__

Why: The suggestion asks to verify the compatibility of updated packages, which is important but not a direct code change. It does not address any specific bug or error.

Low

@xin3he xin3he requested a review from chensuyue January 5, 2026 02:53
xin3he added 2 commits January 4, 2026 22:18
Signed-off-by: He, Xin3 <[email protected]>
@xin3he xin3he added this to the 3.7.1 milestone Jan 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants