Update on "add eval for attention sink"

This PR adds the function to evaluate the model's perplexity when AttentionSink is enabled.

This is mostly copied from https://github.com/mit-han-lab/streaming-llm/blob/main/examples/eval_long_ppl.py which is used by the AttentionSink paper to evaluate the model's perplexity when AttentionSink is enabled.

Differential Revision: [D66474732](https://our.internmc.facebook.com/intern/diff/D66474732/)

Perplexity measured for llama 3.2 1B and 1B_Instruct model up to 40k tokens with AttentionSink enabled:

<img width="966" alt="Screenshot 2024-11-25 at 2 46 04 PM" src="https://github.com/user-attachments/assets/ba7118f9-b5d7-4de8-b1fa-7d2ba0646515">


[ghstack-poisoned]

Author: helunwencser

examples/models/llama/eval_llama_lib.py

@@ -326,7 +326,7 @@ def eval_llama_with_attention_sink(model_name: str, args: argparse.ArgumentParse
     """
     assert args.use_attention_sink is not None  # pyre-ignore [16]
     assert args.attention_sink_eval_tokens > 0  # pyre-ignore [16]
-    attention_sink_params = args.use_attention_sink.split(",")  # pyre-ignore [16]
+    attention_sink_params = args.use_attention_sink.split(",")
     assert len(attention_sink_params) == 3
     sink_size = int(attention_sink_params[0])
     window_size = int(attention_sink_params[1])

examples/models/llama/source_transformation/attention_sink.py

@@ -266,7 +266,7 @@ def _replace_attention(
     for _, child_module in module._modules.items():
         if len(list(child_module.children())) > 0:  # pyre-ignore [16]
             _replace_attention(
-                module=child_module, # pyre-ignore [6]
+                module=child_module,  # pyre-ignore [6]
                 rope_with_attention_sink=rope_with_attention_sink,
                 sink_size=sink_size,
                 window_size=window_size,