Skip to content

Commit b21b0c8

Browse files
authored
Update README.md
1 parent 33f5548 commit b21b0c8

File tree

1 file changed

+38
-4
lines changed

1 file changed

+38
-4
lines changed

README.md

Lines changed: 38 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,21 @@
1-
<p align="center"><img width=500 src="/docs/images/torchbench.png"></p>
1+
<p align="center"><img width=500 src="/docs/docs/img/sotabencheval.png"></p>
22

3-
Easily benchmark ML models on selected tasks and datasets.
3+
--------------------------------------------------------------------------------
4+
5+
[![PyPI version](https://badge.fury.io/py/sotabencheval.svg)](https://badge.fury.io/py/sotabencheval) [![Generic badge](https://img.shields.io/badge/Documentation-Here-<COLOR>.svg)](https://shields.io/)
6+
7+
`sotabencheval` is a framework-agnostic library that contains a collection of deep learning benchmarks you can use to benchmark your models. It can be used in conjunction with the [sotabench](https://www.sotabench.com) service to record results for models, so the community can compare model performance on different tasks, as well as a continuous integration style service for your repository to benchmark your models on each commit.
8+
9+
## Benchmarks Supported
10+
11+
- [ADE20K](https://paperswithcode.github.io/sotabench-eval/ade20k/) (Semantic Segmentation)
12+
- [COCO](https://paperswithcode.github.io/sotabench-eval/coco/) (Object Detection)
13+
- [ImageNet](https://paperswithcode.github.io/sotabench-eval/imagenet/) (Image Classification)
14+
- [SQuAD](https://paperswithcode.github.io/sotabench-eval/squad/) (Question Answering)
15+
- [WikiText-103](https://paperswithcode.github.io/sotabench-eval/wikitext-103/) (Language Modelling)
16+
- [WMT](https://paperswithcode.github.io/sotabench-eval/wmt/) (Machine Translation)
17+
18+
PRs welcome for further benchmarks!
419

520
## Installation
621

@@ -10,9 +25,28 @@ Requires Python 3.6+.
1025
pip install sotabench-eval
1126
```
1227

13-
## Documentation
28+
## Get Benching! 🏋️
29+
30+
You should read the [full documentation here](https://paperswithcode.github.io/sotabench-eval/index.html), which contains guidance on getting started and connecting to [sotabench](https://www.sotabench.com).
31+
32+
Integration is lightweight. For example, if you are evaluating an ImageNet model, you initialize an Evaluator object and (optionally) link to the paper where the model originated from to compare with published results:
33+
34+
```
35+
from sotabencheval.image_classification import ImageNetEvaluator
36+
evaluator = ImageNetEvaluator(
37+
model_name='ResNeXt-101-32x8d',
38+
paper_arxiv_id='1611.05431')
39+
```
40+
41+
Then for each batch of predictions your model makes on ImageNet, you pass a dictionary of keys as image IDs and values as output predictions to the `evaluator.add` method:
42+
43+
```
44+
evaluator.add(dict(zip(image_ids, batch_output)))
45+
```
46+
47+
This logic just needs to be written in a `sotabench.py` file (which contains whatever evaluation logic you need - e.g loading and processing the data), and sotabench will run it on each commit and record the results:
1448

15-
You can access the [full documentation here](https://paperswithcode.github.io/sotabench-eval/index.html).
49+
<img width=500 src="/docs/docs/img/results.png">
1650

1751
## Contributing
1852

0 commit comments

Comments
 (0)