Merge branch 'main' into main

Author: HamidShojanazeri

README.md

@@ -1,17 +1,18 @@
 # Llama Recipes: Examples to get started using the Llama models from Meta
 <!-- markdown-link-check-disable -->
-The 'llama-recipes' repository is a companion to the [Meta Llama 2](https://github.com/meta-llama/llama) and [Meta Llama 3](https://github.com/meta-llama/llama3) models. The goal of this repository is to provide a scalable library for fine-tuning Meta Llama models, along with some example scripts and notebooks to quickly get started with using the models in a variety of use-cases, including fine-tuning for domain adaptation and building LLM-based applications with Meta Llama and other tools in the LLM ecosystem. The examples here showcase how to run Meta Llama locally, in the cloud, and on-prem.
+The 'llama-recipes' repository is a companion to the [Meta Llama 3](https://github.com/meta-llama/llama3) models. The goal of this repository is to provide a scalable library for fine-tuning Meta Llama models, along with some example scripts and notebooks to quickly get started with using the models in a variety of use-cases, including fine-tuning for domain adaptation and building LLM-based applications with Meta Llama and other tools in the LLM ecosystem. The examples here showcase how to run Meta Llama locally, in the cloud, and on-prem. [Meta Llama 2](https://github.com/meta-llama/llama) is also supported in this repository. We highly recommend everyone to utilize [Meta Llama 3](https://github.com/meta-llama/llama3) due to its enhanced capabilities.
+
 <!-- markdown-link-check-enable -->
 > [!IMPORTANT]
-> Llama 3 has a new prompt template and special tokens (based on the tiktoken tokenizer).
+> Meta Llama 3 has a new prompt template and special tokens (based on the tiktoken tokenizer).
 > | Token | Description |
 > |---|---|
 > `<\|begin_of_text\|>` | This is equivalent to the BOS token. |
 > `<\|end_of_text\|>` | This is equivalent to the EOS token. For multiturn-conversations it's usually unused. Instead, every message is terminated with `<\|eot_id\|>` instead.|
 > `<\|eot_id\|>` | This token signifies the end of the message in a turn i.e. the end of a single message by a system, user or assistant role as shown below.|
 > `<\|start_header_id\|>{role}<\|end_header_id\|>` | These tokens enclose the role for a particular message. The possible roles can be: system, user, assistant. |
 >
-> A multiturn-conversation with Llama 3 follows this prompt template:
+> A multiturn-conversation with Meta Llama 3 follows this prompt template:
 > ```
 > <|begin_of_text|><|start_header_id|>system<|end_header_id|>
 >
@@ -133,7 +134,7 @@ Contains examples are organized in folders by topic:
 [quickstart](./recipes/quickstart) | The "Hello World" of using Llama, start here if you are new to using Llama.
 [finetuning](./recipes/finetuning)|Scripts to finetune Llama on single-GPU and multi-GPU setups
 [inference](./recipes/inference)|Scripts to deploy Llama for inference locally and using model servers
-[use_cases](./recipes/use_cases)|Scripts showing common applications of Llama2
+[use_cases](./recipes/use_cases)|Scripts showing common applications of Meta Llama3
 [responsible_ai](./recipes/responsible_ai)|Scripts to use PurpleLlama for safeguarding model outputs
 [llama_api_providers](./recipes/llama_api_providers)|Scripts to run inference on Llama via hosted endpoints
 [benchmarks](./recipes/benchmarks)|Scripts to benchmark Llama models inference on various backends
@@ -159,7 +160,8 @@ Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduc
 
 ## License
 <!-- markdown-link-check-disable -->
-See the License file for Meta Llama 2 [here](https://llama.meta.com/llama2/license/) and Acceptable Use Policy [here](https://llama.meta.com/llama2/use-policy/)
 
 See the License file for Meta Llama 3 [here](https://llama.meta.com/llama3/license/) and Acceptable Use Policy [here](https://llama.meta.com/llama3/use-policy/)
+
+See the License file for Meta Llama 2 [here](https://llama.meta.com/llama2/license/) and Acceptable Use Policy [here](https://llama.meta.com/llama2/use-policy/)
 <!-- markdown-link-check-enable -->

docs/images/messenger_llama_arch.jpg

@@ -1,6 +1,6 @@
 # Llama Model Evaluation
 
-Llama-Recipe make use of `lm-evaluation-harness` for evaluating our fine-tuned Llama2 model. It also can serve as a tool to evaluate quantized model to ensure the quality in lower precision or other optimization applied to the model that might need evaluation.
+Llama-Recipe make use of `lm-evaluation-harness` for evaluating our fine-tuned Meta Llama3 (or Llama2) model. It also can serve as a tool to evaluate quantized model to ensure the quality in lower precision or other optimization applied to the model that might need evaluation.
 
 
 `lm-evaluation-harness` provide a wide range of [features](https://github.com/EleutherAI/lm-evaluation-harness?tab=readme-ov-file#overview):
@@ -12,7 +12,7 @@ Llama-Recipe make use of `lm-evaluation-harness` for evaluating our fine-tuned L
 - Support for evaluation on adapters (e.g. LoRA) supported in Hugging Face's PEFT library.
 - Support for local models and benchmarks.
 
-The Language Model Evaluation Harness is also the backend for 🤗 [Hugging Face's (HF) popular Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). 
+The Language Model Evaluation Harness is also the backend for 🤗 [Hugging Face's (HF) popular Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
 
 ## Setup
 
@@ -36,35 +36,35 @@ pip install -e .
 
 ### Quick Test
 
-To run evaluation for Hugging Face `Llama2 7B` model  on a single GPU please run the following,
+To run evaluation for Hugging Face `Llama3 8B` model  on a single GPU please run the following,
 
 ```bash
-python eval.py --model hf --model_args pretrained=meta-llama/Llama-2-7b-chat-hf --tasks hellaswag --device cuda:0   --batch_size 8
+python eval.py --model hf --model_args pretrained=meta-llama/Meta-Llama-3-8B --tasks hellaswag --device cuda:0   --batch_size 8
 
 ```
 Tasks can be extended by using `,` between them for example `--tasks hellaswag,arc`.
 
-To set the number of shots you can use `--num_fewshot` to set the number for few shot evaluation. 
+To set the number of shots you can use `--num_fewshot` to set the number for few shot evaluation.
 
-### PEFT Fine-tuned model Evaluation  
+### PEFT Fine-tuned model Evaluation
 
 In case you have fine-tuned your model using PEFT you can set the PATH to the PEFT checkpoints using PEFT as part of model_args as shown below:
 
 ```bash
-python eval.py --model hf --model_args pretrained=meta-llama/Llama-2-7b-hf,dtype="float",peft=../peft_output --tasks hellaswag --num_fewshot 10  --device cuda:0 --batch_size 8 
+python eval.py --model hf --model_args pretrained=meta-llama/Meta-Llama-3-8B, dtype="float",peft=../peft_output --tasks hellaswag --num_fewshot 10  --device cuda:0 --batch_size 8
 ```
 
 ### Limit the number of examples in benchmarks
 
 There has been an study from [IBM on efficient benchmarking of LLMs](https://arxiv.org/pdf/2308.11696.pdf), with main take a way that to identify if a model is performing poorly, benchmarking on wider range of tasks is more important than the number example in each task. This means you could run the evaluation harness with fewer number of example to have initial decision if the performance got worse from the base line. To limit the number of example here, it can be set using `--limit` flag with actual desired number. But for the full assessment you would need to run the full evaluation. Please read more in the paper linked above.
 
 ```bash
-python eval.py --model hf --model_args pretrained=meta-llama/Llama-2-7b-hf,dtype="float",peft=../peft_output --tasks hellaswag --num_fewshot 10  --device cuda:0 --batch_size 8 --limit 100
+python eval.py --model hf --model_args pretrained=meta-llama/Meta-Llama-3-8B,dtype="float",peft=../peft_output --tasks hellaswag --num_fewshot 10  --device cuda:0 --batch_size 8 --limit 100
 ```
 
 ### Reproducing Hugging Face Open-LLM-Leaderboard
 
-Here, we provided a list of tasks from `Open-LLM-Leaderboard` which can be used by passing `--open-llm-leaderboard-tasks` instead of `tasks` to the `eval.py`. 
+Here, we provided a list of tasks from `Open-LLM-Leaderboard` which can be used by passing `--open-llm-leaderboard-tasks` instead of `tasks` to the `eval.py`.
 
 **NOTE** Make sure to run the bash script below, that will set the `include paths` in the [config files](./open_llm_leaderboard/). The script will prompt you to enter the path to the cloned lm-evaluation-harness repo.**You would need this step only for the first time**.
 
@@ -76,7 +76,7 @@ bash open_llm_eval_prep.sh
 Now we can run the eval benchmark:
 
 ```bash
-python eval.py --model hf --model_args pretrained=meta-llama/Llama-2-7b-hf,dtype="float",peft=../peft_output --num_fewshot 10  --device cuda:0 --batch_size 8 --limit 100 --open_llm_leaderboard_tasks
+python eval.py --model hf --model_args pretrained=meta-llama/Meta-Llama-3-8B,dtype="float",peft=../peft_output --num_fewshot 10  --device cuda:0 --batch_size 8 --limit 100 --open_llm_leaderboard_tasks
 ```
 
 In the HF leaderboard, the [LLMs are evaluated on 7 benchmarks](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) from Language Model Evaluation Harness as described below:
@@ -107,7 +107,7 @@ To perform *data-parallel evaluation* (where each GPU loads a **separate full co
 ```bash
 accelerate config
 
-accelerate launch eval.py --model hf --model_args "pretrained=meta-llama/Llama-2-7b-chat-hf" --limit 100 --open-llm-leaderboard-tasks --output_path ./results.json --log_samples 
+accelerate launch eval.py --model hf --model_args "pretrained=meta-llama/Meta-Llama-3-8B" --limit 100 --open-llm-leaderboard-tasks --output_path ./results.json --log_samples
 ```
 
 In case your model can fit on a single GPU, this allows you to evaluate on K GPUs K times faster than on one.
@@ -119,7 +119,7 @@ In case your model is *too large to fit on a single GPU.*
 In this setting, run the library *outside of the `accelerate` launcher*, but passing `parallelize=True` to `--model_args` as follows:
 
 ```bash
-python eval.py --model hf --model_args "pretrained=meta-llama/Llama-2-7b-chat-hf,parallelize=True" --limit 100 --open_llm_leaderboard_tasks --output_path ./results.json --log_samples 
+python eval.py --model hf --model_args "pretrained=meta-llama/Meta-Llama-3-8B,parallelize=True" --limit 100 --open_llm_leaderboard_tasks --output_path ./results.json --log_samples
 ```
 
 
@@ -138,7 +138,7 @@ These two options (`accelerate launch` and `parallelize=True`) are mutually excl
 Also `lm-evaluation-harness` supports vLLM for faster inference on [supported model types](https://docs.vllm.ai/en/latest/models/supported_models.html), especially faster when splitting a model across multiple GPUs. For single-GPU or multi-GPU — tensor parallel, data parallel, or a combination of both — inference, for example:
 
 ```bash
-python eval.py --model vllm --model_args "pretrained=meta-llama/Llama-2-7b-chat-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=2" --limit 100 --open_llm_leaderboard_tasks --output_path ./results.json --log_samples --batch_size auto
+python eval.py --model vllm --model_args "pretrained=meta-llama/Meta-Llama-3-8B,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=2" --limit 100 --open_llm_leaderboard_tasks --output_path ./results.json --log_samples --batch_size auto
 ```
 For a full list of supported vLLM configurations, please to [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/076372ee9ee81e25c4e2061256400570354a8d1a/lm_eval/models/vllm_causallms.py#L44-L62).
 

docs/images/whatsapp_llama_arch.jpg

@@ -251,7 +251,7 @@
     "        get_peft_model,\n",
     "        LoraConfig,\n",
     "        TaskType,\n",
-    "        prepare_model_for_int8_training,\n",
+    "        prepare_model_for_kbit_training,\n",
     "    )\n",
     "\n",
     "    peft_config = LoraConfig(\n",
@@ -264,7 +264,7 @@
     "    )\n",
     "\n",
     "    # prepare int-8 model for training\n",
-    "    model = prepare_model_for_int8_training(model)\n",
+    "    model = prepare_model_for_kbit_training(model)\n",
     "    model = get_peft_model(model, peft_config)\n",
     "    model.print_trainable_parameters()\n",
     "    return model, peft_config\n",

recipes/evaluation/README.md

@@ -31,7 +31,7 @@ def main(
     temperature: float=1.0, # [optional] The value used to modulate the next token probabilities.
     top_k: int=50, # [optional] The number of highest probability vocabulary tokens to keep for top-k-filtering.
     repetition_penalty: float=1.0, #The parameter for repetition penalty. 1.0 means no penalty.
-    length_penalty: int=1, #[optional] Exponential penalty to the length that is used with beam-based generation. 
+    length_penalty: int=1, #[optional] Exponential penalty to the length that is used with beam-based generation.
     enable_azure_content_safety: bool=False, # Enable safety check with Azure content safety api
     enable_sensitive_topics: bool=False, # Enable check for sensitive topics using AuditNLG APIs
     enable_salesforce_content_safety: bool=True, # Enable safety check with Salesforce safety flan t5
@@ -98,12 +98,12 @@ def inference(user_prompt, temperature, top_p, top_k, max_new_tokens, **kwargs,)
             top_k=top_k,
             repetition_penalty=repetition_penalty,
             length_penalty=length_penalty,
-            **kwargs 
+            **kwargs
         )
     e2e_inference_time = (time.perf_counter()-start)*1000
     print(f"the inference time is {e2e_inference_time} ms")
     output_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
-    
+
     # Safety check of the model output
     safety_results = [check(output_text, agent_type=AgentType.AGENT, user_prompt=user_prompt) for check in safety_checker]
     are_safe = all([r[1] for r in safety_results])
@@ -156,7 +156,7 @@ def inference(user_prompt, temperature, top_p, top_k, max_new_tokens, **kwargs,)
                 label="Output",
             )
         ],
-        title="Llama2 Playground",
+        title="Meta Llama3 Playground",
         description="https://github.com/facebookresearch/llama-recipes",
       ).queue().launch(server_name="0.0.0.0", share=True)