scripts : exit compare-llama-bench.py gracefully when there's nothing to compare #13451
+5
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
Author
|
I wonder though, sometimes you want to compare apples to oranges (like two different models, or at the very least various settings), so it might make sense to broaden the options for comparison in the future? |
JohannesGaessler
approved these changes
May 11, 2025
Collaborator
|
The script could definitely use more work in general; I wrote the script for myself according to my own needs and while I did contribute it upstream the needs of other users were a secondary concern for me. I think that for people not working on the code the comparisons between commits are maybe not as interesting. So I agree that more options for what to compare would be useful to have. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I stumbled upon this accidentally while testing JSONL (coming soon). I had used two different paths to the same model in consequent tests, causing
get_rowsnot to be able to join the two results.