Skip to content

Commit 41ef785

Browse files
committed
add task names of multipl-e
1 parent 7c31799 commit 41ef785

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

docs/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,8 @@ TF_CPP_MIN_LOG_LEVEL=3 accelerate launch main.py \
100100

101101
### MultiPL-E
102102
[MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E): is a benchamrk for evaluating large language models for code generation that supports 18 programming languages. It takes the OpenAI "HumanEval" Python benchmark and uses little compilers to translate them to other languages. We use similar implementation as [the original repository](https://github.com/nuprl/MultiPL-E/tree/main) and evaluation parameters are similar to HumanEval. Although for this benchmark, we strongly recommend using the provided Dockerfile to build the MultiPL-E container with all required dependencies, and for more safety especially when evaluating on languages like `bash`.
103+
Tasks are named `multiple-<LANG>` where `<LANG>` is the language name, e.g. `multiple-py` for python.
104+
103105
```bash
104106
$ sudo make DOCKERFILE=Dockerfile-multiple all
105107
```

0 commit comments

Comments
 (0)