Skip to content

[Bench] Add bench for MMLU eval #2584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[Bench] Add bench for MMLU eval #2584

wants to merge 1 commit into from

Conversation

Hzfengsy
Copy link
Member

Usage:

python benchmark/eval/mmlu.py --model dist/Meta-Llama-3-8B-Instruct-q4f16_1-MLC --dataset /path/to/dataset --device cuda --log-dir debug/mmlu

Note that chat mode is problematic, which needs to be fixed.

@Hzfengsy Hzfengsy marked this pull request as ready for review June 16, 2024 06:58
@Hzfengsy Hzfengsy force-pushed the mmlu_eval branch 4 times, most recently from 3d8232d to d015821 Compare June 18, 2024 03:14
@Hzfengsy
[Bench] Add bench for MMLU eval
d9e37ca 
Usage:
```
python python/bench/eval/mmlu.py --model dist/Meta-Llama-3-8B-Instruct-q4f16_1-MLC --dataset /path/to/dataset --device cuda --log-dir debug/mmlu
```

Note that chat mode is problematic, which needs to be fixed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Hzfengsy