Skip to content

Commit d7c2559

Browse files
philschmidosansevieropcuencalvwerra
authored and
kashif
committed
Codellama (huggingface#1419)
* init * add demo * Update codellama.md Co-authored-by: Omar Sanseviero <[email protected]> * Update codellama.md Co-authored-by: Omar Sanseviero <[email protected]> * Update codellama.md Co-authored-by: Omar Sanseviero <[email protected]> * 4-bit section, tweaks * adding image * Clarify FIM availability. * Move image to dataset. * update table * FIM typo Thanks ArthurZ! * update * Update codellama.md Co-authored-by: Omar Sanseviero <[email protected]> * Update codellama.md Co-authored-by: Omar Sanseviero <[email protected]> * Apply suggestions from code review * Update codellama.md Co-authored-by: Pedro Cuenca <[email protected]> * Update codellama.md Co-authored-by: Pedro Cuenca <[email protected]> * Update codellama.md Co-authored-by: Pedro Cuenca <[email protected]> * Update codellama.md Co-authored-by: Leandro von Werra <[email protected]> * Update codellama.md Co-authored-by: Pedro Cuenca <[email protected]> --------- Co-authored-by: Omar Sanseviero <[email protected]> Co-authored-by: Pedro Cuenca <[email protected]> Co-authored-by: Leandro von Werra <[email protected]>
1 parent dec6964 commit d7c2559

File tree

3 files changed

+351
-1
lines changed

3 files changed

+351
-1
lines changed

‎_blog.yml

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2731,4 +2731,15 @@
27312731
date: August 25, 2023
27322732
tags:
27332733
- announcement
2734-
- security
2734+
- security
2735+
2736+
- local: codellama
2737+
title: "Llama 2 learns to code"
2738+
author: philschmid
2739+
thumbnail: /blog/assets/160_codellama/thumbnail.jpg
2740+
date: August 25, 2023
2741+
tags:
2742+
- nlp
2743+
- community
2744+
- research
2745+
- LLM

‎assets/160_codellama/thumbnail.jpg

148 KB
Loading

‎codellama.md

Lines changed: 339 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,339 @@
1+
---
2+
title: "Llama 2 learns to code"
3+
thumbnail: /blog/assets/160_codellama/thumbnail.jpg
4+
authors:
5+
- user: philschmid
6+
- user: osanseviero
7+
- user: pcuenq
8+
- user: lewtun
9+
- user: lvwerra
10+
- user: loubnabnl
11+
- user: ArthurZ
12+
- user: joaogante
13+
---
14+
15+
# Llama 2 learns to code
16+
17+
<!-- {blog_metadata} -->
18+
<!-- {authors} -->
19+
20+
## Introduction
21+
22+
Code Llama is a family of state-of-the-art, open-access versions of [Llama 2](https://huggingface.co/blog/llama2) specialized on code tasks, and we’re excited to release integration in the Hugging Face ecosystem! Code Llama has been released with the same permissive community license as Llama 2 and is available for commercial use.
23+
24+
Today, we’re excited to release:
25+
26+
- Models on the Hub with their model cards and license
27+
- Transformers integration
28+
- Integration with Text Generation Inference for fast and efficient production-ready inference
29+
- Integration with Inference Endpoints
30+
- Code benchmarks
31+
32+
Code LLMs are an exciting development for software engineers because they can boost productivity through code completion in IDEs, take care of repetitive or annoying tasks like writing docstrings, or create unit tests.
33+
34+
## Table of Contents
35+
36+
- [Introduction](#introduction)
37+
- [Table of Contents](#table-of-contents)
38+
- [What’s Code Llama?](#whats-code-llama)
39+
- [How to use Code Llama?](#how-to-use-code-llama)
40+
- [Demo](#demo)
41+
- [Transformers](#transformers)
42+
- [Code Completion](#code-completion)
43+
- [Code Infilling](#code-infilling)
44+
- [Conversational Instructions](#conversational-instructions)
45+
- [4-bit Loading](#4-bit-loading)
46+
- [Using text-generation-inference and Inference Endpoints](#using-text-generation-inference-and-inference-endpoints)
47+
- [Evaluation](#evaluation)
48+
- [Additional Resources](#additional-resources)
49+
50+
## What’s Code Llama?
51+
52+
The Code Llama release introduces a family of models of 7, 13, and 34 billion parameters. The base models are initialized from Llama 2 and then trained on 500 billion tokens of code data. Meta fine-tuned those base models for two different flavors: a Python specialist (100 billion additional tokens) and an instruction fine-tuned version, which can understand natural language instructions.
53+
54+
The models show state-of-the-art performance in Python, C++, Java, PHP, C#, TypeScript, and Bash. The 7B and 13B base and instruct variants support infilling based on surrounding content, making them ideal for use as code assistants.
55+
56+
Code Llama was trained on a 16k context window. In addition, the three model variants had additional long-context fine-tuning, allowing them to manage a context window of up to 100,000 tokens.
57+
58+
Increasing Llama 2’s 4k context window to Code Llama’s 16k (that can extrapolate up to 100k) was possible due to recent developments in RoPE scaling. The community found that Llama’s position embeddings can be interpolated linearly or in the frequency domain, which eases the transition to a larger context window through fine-tuning. In the case of Code Llama, the frequency domain scaling is done with a slack: the fine-tuning length is a fraction of the scaled pretrained length, giving the model powerful extrapolation capabilities.
59+
60+
![Training Process](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/160_codellama/training-process.jpg "Training Process")
61+
62+
All models were initially trained with 500 billion tokens on a near-deduplicated dataset of publicly available code. The dataset also contains some natural language datasets, such as discussions about code and code snippets. Unfortunately, there is not more information about the dataset.
63+
64+
For the instruction model, they used two datasets: the instruction tuning dataset collected for Llama 2 Chat and a self-instruct dataset. The self-instruct dataset was created by using Llama 2 to create interview programming questions and then using Code Llama to generate unit tests and solutions, which are later evaluated by executing the tests.
65+
66+
## How to use Code Llama?
67+
68+
Code Llama is available in the Hugging Face ecosystem, starting with `transformers` version 4.33. Until `transformers` 4.33 is released, please install it from the main branch.
69+
70+
### Demo
71+
72+
You can easily try the Code Llama Model (13 billion parameters!) in **[this Space](https://huggingface.co/spaces/codellama/codellama-playground)** or in the playground embedded below:
73+
74+
<script type="module" src="https://gradio.s3-us-west-2.amazonaws.com/3.28.3/gradio.js"> </script>
75+
<gradio-app theme_mode="light" space="codellama/codellama-playground"></gradio-app>
76+
77+
Under the hood, this playground uses Hugging Face's [Text Generation Inference](https://github.com/huggingface/text-generation-inference), the same technology that powers [HuggingChat](https://huggingface.co/chat/), and we'll share more in the following sections.
78+
79+
### Transformers
80+
81+
With the upcoming release of `transformers` 4.33, you can use Code Llama and leverage all the tools within the HF ecosystem, such as:
82+
83+
- training and inference scripts and examples
84+
- safe file format (`safetensors`)
85+
- integrations with tools such as `bitsandbytes` (4-bit quantization) and PEFT (parameter efficient fine-tuning)
86+
- utilities and helpers to run generation with the model
87+
- mechanisms to export the models to deploy
88+
89+
Until `transformers` 4.33 is released, please install it from the main branch.
90+
91+
```bash
92+
!pip install git+https://github.com/huggingface/transformers.git@main accelerate
93+
```
94+
95+
#### Code Completion
96+
97+
The 7B and 13B models can be used for text/code completion or infilling. The following code snippet uses the `pipeline` interface to demonstrate text completion. It runs on the free tier of Colab, as long as you select a GPU runtime.
98+
99+
```python
100+
from transformers import AutoTokenizer
101+
import transformers
102+
import torch
103+
104+
tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf")
105+
pipeline = transformers.pipeline(
106+
"text-generation",
107+
model="codellama/CodeLlama-7b-hf",
108+
torch_dtype=torch.float16,
109+
device_map="auto",
110+
)
111+
112+
sequences = pipeline(
113+
'def fibonacci(',
114+
do_sample=True,
115+
temperature=0.2,
116+
top_p=0.9,
117+
num_return_sequences=1,
118+
eos_token_id=tokenizer.eos_token_id,
119+
max_length=100,
120+
)
121+
for seq in sequences:
122+
print(f"Result: {seq['generated_text']}")
123+
```
124+
125+
This may produce output like the following:
126+
127+
```python
128+
Result: def fibonacci(n):
129+
if n == 0:
130+
return 0
131+
elif n == 1:
132+
return 1
133+
else:
134+
return fibonacci(n-1) + fibonacci(n-2)
135+
136+
def fibonacci_memo(n, memo={}):
137+
if n == 0:
138+
return 0
139+
elif n == 1:
140+
return
141+
```
142+
143+
Code Llama is specialized in code understanding, but it's a language model in its own right. You can use the same generation strategy to autocomplete comments or general text.
144+
145+
#### Code Infilling
146+
147+
This is a specialized task particular to code models. The model is trained to generate the code (including comments) that best matches an existing prefix and suffix. This is the strategy typically used by code assistants: they are asked to fill the current cursor position, considering the contents that appear before and after it.
148+
149+
This task is available in the **base** and **instruction** variants of the 7B and 13B models. It is _not_ available for any of the 34B models or the Python versions.
150+
151+
To use this feature successfully, you need to pay close attention to the format used to train the model for this task, as it uses special separators to identify the different parts of the prompt. Let's see an example:
152+
153+
```python
154+
from transformers import AutoTokenizer, AutoModelForCausalLM
155+
import transformers
156+
import torch
157+
158+
model_id = "codellama/CodeLlama-7b-hf"
159+
tokenizer = AutoTokenizer.from_pretrained(model_id)
160+
model = AutoModelForCausalLM.from_pretrained(
161+
model_id,
162+
torch_dtype=torch.float16
163+
).to("cuda")
164+
165+
prefix = 'def remove_non_ascii(s: str) -> str:\n """ '
166+
suffix = "\n return result\n"
167+
168+
prompt = f"<PRE> {prefix} <SUF>{suffix} <MID>"
169+
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
170+
171+
output = model.generate(
172+
inputs["input_ids"],
173+
max_new_tokens=200,
174+
do_sample=False,
175+
)
176+
output = output[0].to("cpu")
177+
print(tokenizer.decode(output))
178+
```
179+
180+
```
181+
<s> <PRE> def remove_non_ascii(s: str) -> str:
182+
""" <SUF>
183+
return result
184+
<MID>
185+
Remove non-ASCII characters from a string.
186+
187+
:param s: The string to remove non-ASCII characters from.
188+
:return: The string with non-ASCII characters removed.
189+
"""
190+
result = ""
191+
for c in s:
192+
if ord(c) < 128:
193+
result += c <EOT></s>
194+
```
195+
196+
In order to use the completion, you’ll need to process the output to cut the text between the `<MID>` and `<EOT>` tokens – that’s what goes between the prefix and suffix we supplied.
197+
198+
#### Conversational Instructions
199+
200+
The base model can be used for both completion and infilling, as described. The Code Llama release also includes an instruction fine-tuned model that can be used in conversational interfaces.
201+
202+
To prepare inputs for this task we have to use a prompt template like the one described in our [Llama 2 blog post](https://huggingface.co/blog/llama2#how-to-prompt-llama-2), which we reproduce again here:
203+
204+
```
205+
<s>[INST] <<SYS>>
206+
{{ system_prompt }}
207+
<</SYS>>
208+
209+
{{ user_msg_1 }} [/INST] {{ model_answer_1 }} </s><s>[INST] {{ user_msg_2 }} [/INST]
210+
```
211+
212+
Note that the system prompt is optional - the model will work without it, but you can use it to further configure its behavior or style. For example, if you'd always like to get answers in JavaScript, you could state that here. After the system prompt, you need to provide all the previous interactions in the conversation: what was asked by the user and what was answered by the model. As in the infilling case, you need to pay attention to the delimiters used. The final component of the input must always be a new user instruction, which will be the signal for the model to provide an answer.
213+
214+
The following code snippets demonstrate how the template works in practice.
215+
216+
1. **First user query, no system prompt**
217+
218+
```python
219+
user = 'In Bash, how do I list all text files in the current directory (excluding subdirectories) that have been modified in the last month?'
220+
221+
prompt = f"<s>[INST] {user.strip()} [/INST]"
222+
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
223+
```
224+
225+
2. **First user query with system prompt**
226+
227+
```python
228+
system = "Provide answers in JavaScript"
229+
user = "Write a function that computes the set of sums of all contiguous sublists of a given list."
230+
231+
prompt = f"<s><<SYS>>\\n{system}\\n<</SYS>>\\n\\n{user}"
232+
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
233+
```
234+
235+
3. **On-going conversation with previous answers**
236+
237+
The process is the same as in [Llama 2](https://huggingface.co/blog/llama2#how-to-prompt-llama-2). We haven’t used loops or generalized this example code for maximum clarity:
238+
239+
```python
240+
system = "System prompt"
241+
user_1 = "user_prompt_1"
242+
answer_1 = "answer_1"
243+
user_2 = "user_prompt_2"
244+
answer_2 = "answer_2"
245+
user_3 = "user_prompt_3"
246+
247+
prompt = f"<<SYS>>\\n{system}\\n<</SYS>>\\n\\n{user_1}"
248+
prompt = f"<s>[INST] {prompt.strip()} [/INST] {answer_1.strip()} </s>"
249+
prompt += f"<s>[INST] {user_2.strip()} [/INST] {answer_2.strip()} </s>"
250+
prompt += f"<s>[INST] {user_3.strip()} [/INST]"
251+
252+
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
253+
```
254+
255+
#### 4-bit Loading
256+
257+
Integration of Code Llama in Transformers means that you get immediate support for advanced features like 4-bit loading. This allows you to run the big 32B parameter models on consumer GPUs like nvidia 3090 cards!
258+
259+
Here's how you can run inference in 4-bit mode:
260+
261+
```Python
262+
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
263+
import torch
264+
265+
model_id = "codellama/CodeLlama-34b-hf"
266+
quantization_config = BitsAndBytesConfig(
267+
load_in_4bit=True,
268+
bnb_4bit_compute_dtype=torch.float16
269+
)
270+
271+
tokenizer = AutoTokenizer.from_pretrained(model_id)
272+
model = AutoModelForCausalLM.from_pretrained(
273+
model_id,
274+
quantization_config=quantization_config,
275+
device_map="auto",
276+
)
277+
278+
prompt = 'def remove_non_ascii(s: str) -> str:\n """ '
279+
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
280+
281+
output = model.generate(
282+
inputs["input_ids"],
283+
max_new_tokens=200,
284+
do_sample=True,
285+
top_p=0.9,
286+
temperature=0.1,
287+
)
288+
output = output[0].to("cpu")
289+
print(tokenizer.decode(output))
290+
```
291+
292+
### Using text-generation-inference and Inference Endpoints
293+
294+
[Text Generation Inference](https://github.com/huggingface/text-generation-inference) is a production-ready inference container developed by Hugging Face to enable easy deployment of large language models. It has features such as continuous batching, token streaming, tensor parallelism for fast inference on multiple GPUs, and production-ready logging and tracing.
295+
296+
You can try out Text Generation Inference on your own infrastructure, or you can use Hugging Face's [Inference Endpoints](https://huggingface.co/inference-endpoints). To deploy a Codellama 2 model, go to the [model page](https://huggingface.co/codellama) and click on the [Deploy -> Inference Endpoints](https://huggingface.co/codellama/CodeLlama-7b-hf) widget.
297+
298+
- For 7B models, we advise you to select "GPU [medium] - 1x Nvidia A10G".
299+
- For 13B models, we advise you to select "GPU [xlarge] - 1x Nvidia A100".
300+
- For 34B models, we advise you to select "GPU [1xlarge] - 1x Nvidia A100" with `bitsandbytes` quantization enabled or "GPU [2xlarge] - 2x Nvidia A100"
301+
302+
*Note: You might need to request a quota upgrade via email to **[[email protected]](mailto:[email protected])** to access A100s*
303+
304+
You can learn more on how to [Deploy LLMs with Hugging Face Inference Endpoints in our blog](https://huggingface.co/blog/inference-endpoints-llm). The [blog](https://huggingface.co/blog/inference-endpoints-llm) includes information about supported hyperparameters and how to stream your response using Python and Javascript.
305+
306+
## Evaluation
307+
308+
Language models for code are typically benchmarked on datatsets such as HumanEval. It consists of programming challenges where the model is presented with a function signature and a docstring and is tasked to complete the function body. The proposed solution is then verified by running a set of predefined unit tests. Finally, a pass rate is reported which describes how many solutions passed all tests. The pass@1 rate describes how often the model generates a passing solution when having one shot whereas pass@10 describes how often at least one solution passes out of 10 proposed candidates.
309+
310+
While HumanEval is a Python benchmark there have been significant efforts to translate it to more programming languages and thus enable a more holistic evaluation. One such approach is [MultiPL-E](https://github.com/nuprl/MultiPL-E) which translates HumanEval to over a dozen languages. We are hosting a [multilingual code leaderboard](https://huggingface.co/spaces/bigcode/multilingual-code-evals) based on it to allow the community to compare models across different languages to evaluate which model fits their use-case best.
311+
312+
| Model | License | Dataset known | Commercial use? | Pretraining length [tokens] | Python | JavaScript | Leaderboard Avg Score |
313+
| ---------------------- | ------------------ | ------------- | --------------- | --------------------------- | ------ | ---------- | --------------------- |
314+
| CodeLlaMa-34B | Llama 2 license | ❌ | ✅ | 2,500B | 45.11 | 41.66 | 33.89 |
315+
| CodeLlaMa-13B | Llama 2 license | ❌ | ✅ | 2,500B | 35.07 | 38.26 | 28.35 |
316+
| CodeLlaMa-7B | Llama 2 license | ❌ | ✅ | 2,500B | 29.98 | 31.8 | 24.36 |
317+
| CodeLlaMa-34B-Python | Llama 2 license | ❌ | ✅ | 2,620B | 53.29 | 44.72 | 33.87 |
318+
| CodeLlaMa-13B-Python | Llama 2 license | ❌ | ✅ | 2,620B | 42.89 | 40.66 | 28.67 |
319+
| CodeLlaMa-7B-Python | Llama 2 license | ❌ | ✅ | 2,620B | 40.48 | 36.34 | 23.5 |
320+
| CodeLlaMa-34B-Instruct | Llama 2 license | ❌ | ✅ | 2,620B | 50.79 | 45.85 | 35.09 |
321+
| CodeLlaMa-13B-Instruct | Llama 2 license | ❌ | ✅ | 2,620B | 50.6 | 40.91 | 31.29 |
322+
| CodeLlaMa-7B-Instruct | Llama 2 license | ❌ | ✅ | 2,620B | 45.65 | 33.11 | 26.45 |
323+
| StarCoder-15B | BigCode-OpenRail-M | ✅ | ✅ | 1,035B | 33.57 | 30.79 | 22.74 |
324+
| StarCoderBase-15B | BigCode-OpenRail-M | ✅ | ✅ | 1,000B | 30.35 | 31.7 | 22.4 |
325+
| WizardCoder-15B | BigCode-OpenRail-M | ❌ | ✅ | 1,035B | 58.12 | 41.91 | 32.07 |
326+
| OctoCoder-15B | BigCode-OpenRail-M | ✅ | ✅ | 1,000B | 45.3 | 32.8 | 24.01 |
327+
| CodeGeeX-2-6B | CodeGeeX License | ❌ | ❌ | 2,000B | 33.49 | 29.9 | 21.23 |
328+
| CodeGen-2.5-7B-Mono | Apache-2.0 | ✅ | ✅ | 1400B | 45.65 | 23.22 | 12.1 |
329+
| CodeGen-2.5-7B-Multi | Apache-2.0 | ✅ | ✅ | 1400B | 28.7 | 26.27 | 20.04 |
330+
331+
**Note:** The scores presented in the table above are sourced from our code leaderboard, where we evaluate all models with the same settings. For more details, please refer to the [leaderboard](https://huggingface.co/spaces/bigcode/multilingual-code-evals).
332+
333+
## Additional Resources
334+
335+
- [Models on the Hub](https://huggingface.co/codellama)
336+
- [Paper Page](https://huggingface.co/papers/2308.12950)
337+
- [Official Meta announcement](https://ai.meta.com/blog/code-llama-large-language-model-coding/)
338+
- [Responsible Use Guide](https://ai.meta.com/llama/responsible-use-guide/)
339+
- [Demo](https://huggingface.co/spaces/codellama/codellama-playground)

0 commit comments

Comments
 (0)