Revert "Revert "[sharktank] Perplexity refactor + add features (nod-ai#1172)""

This reverts commit 3d8105d.

Author: IanNod

.github/workflows/ci-sharktank-nightly.yml

@@ -136,7 +136,7 @@ jobs:
             -s \
             sharktank/tests/evaluate/perplexity_iree_test.py \
             --run-nightly-llama-tests \
-            --bs=100 \
+            --bs=128 \
             --iree-device=hip://0 \
             --iree-hip-target=gfx942 \
             --iree-hal-target-device=hip \

sharktank/sharktank/evaluate/README.md

@@ -39,7 +39,8 @@ pytest -n 8 -v -s sharktank/tests/evaluate/perplexity_iree_test.py -k test_llama
   --bs=4 \
   --iree-device=hip://0 \
   --iree-hip-target=gfx942 \
-  --iree-hal-target-device=hip
+  --iree-hal-target-device=hip \
+  --run-nightly-llama-tests
 ```
 
 For a new model:
@@ -83,10 +84,10 @@ python -m sharktank.evaluate.perplexity_iree  -h
 
 | CPU            | GPU        | Num of prompts   |
 |:-------------: |:----------:|:----------------:|
-| AMD EPYC 9554  | MI300X     |      100         |
+| AMD EPYC 9554  | MI300X     |      128         |
 
 #### LLaMA 3.1
 
 |Models                          |Torch score   |IREE score    | Model size (GB) |
 |:-------------------------------|:-------------|:-------------|:----------------|
-|8B FP16 Instruct TP1            |20.223236     |19.786807     |16.07            |
+|8B FP16 Instruct TP1            |17.555018       |14.6294       |16.07            |