Skip to content

Commit 434ea53

Browse files
stesMMathisLabjeylau
committed
Add DeepLabCut benchmark 0.0.1
- Create CNAME - Add source files - Deploy page - Add requirements - Update README.rst - Create .nojekyll - Create .nojekyll - Replace placeholder numbers - Update datasets.rst - Update page - Update dataset links - Update results.tsv - Update datasets.rst - Update index.rst - Update submission.rst - Update table - Add first draft of benchmark api - Fix the Parenting dataset - Debugging - Add metadata files - Squashing bugs - All all working benchmarks - Add encrypted groundtruth - Some edits to benchmark structure and API; finish result collection - Implement caching, finalize eval - Build everything automatically - model name correction and iteration # - Fix --nocache - Add --onerror argument - Update pups mAP calculation - Rewrite DLCDataset to handle "unique" body parts - Add links to code - Adapt docs - Update README - Remove non-public files - Remove unused files - remove non-public files - Add missing dependency and update README - Avoid adding empty Joints to Assemblies - Incorporate pose scoring - Add extra funcs - formatting - Update files for multi mouse, and parenting - Rename models to match current website - Fix typos in model names - Add RMSE column - Add missing column - Adjust column ordering - Finalize new benchmark table, add links to models - Switch to using deeplabcut.benchmark - Complete switch to deeplabcut.benchmark - Remove API docs, update page Co-authored-by: Mackenzie Mathis <[email protected]> Co-authored-by: Jessy <[email protected]>
0 parents  commit 434ea53

File tree

109 files changed

+29002
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

109 files changed

+29002
-0
lines changed

‎.gitignore‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
.results
2+
# If DLC is checked out for installation in current folder
3+
DeepLabCut/
4+
# ground truth files will not be added to public repo
5+
/data/
6+
benchmark/data
7+
8+
# Byte-compiled / optimized / DLL files
9+
__pycache__/
10+
*.py[cod]
11+
*$py.class
12+
13+
# C extensions
14+
*.so
15+
16+
# Distribution / packaging
17+
.Python
18+
build/
19+
develop-eggs/
20+
dist/
21+
downloads/
22+
eggs/
23+
.eggs/
24+
lib/
25+
lib64/
26+
parts/
27+
sdist/
28+
var/
29+
wheels/
30+
pip-wheel-metadata/
31+
share/python-wheels/
32+
*.egg-info/
33+
.installed.cfg
34+
*.egg
35+
MANIFEST
36+
37+
# PyInstaller
38+
# Usually these files are written by a python script from a template
39+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
40+
*.manifest
41+
*.spec
42+
43+
# Installer logs
44+
pip-log.txt
45+
pip-delete-this-directory.txt
46+
47+
# Unit test / coverage reports
48+
htmlcov/
49+
.tox/
50+
.nox/
51+
.coverage
52+
.coverage.*
53+
.cache
54+
nosetests.xml
55+
coverage.xml
56+
*.cover
57+
*.py,cover
58+
.hypothesis/
59+
.pytest_cache/
60+
61+
# Translations
62+
*.mo
63+
*.pot
64+
65+
# Django stuff:
66+
*.log
67+
local_settings.py
68+
db.sqlite3
69+
db.sqlite3-journal
70+
71+
# Flask stuff:
72+
instance/
73+
.webassets-cache
74+
75+
# Scrapy stuff:
76+
.scrapy
77+
78+
# Sphinx documentation
79+
docs/_build/
80+
81+
# PyBuilder
82+
target/
83+
84+
# Jupyter Notebook
85+
.ipynb_checkpoints
86+
87+
# IPython
88+
profile_default/
89+
ipython_config.py
90+
91+
# pyenv
92+
.python-version
93+
94+
# pipenv
95+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
96+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
97+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
98+
# install all needed dependencies.
99+
#Pipfile.lock
100+
101+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
102+
__pypackages__/
103+
104+
# Celery stuff
105+
celerybeat-schedule
106+
celerybeat.pid
107+
108+
# SageMath parsed files
109+
*.sage.py
110+
111+
# Environments
112+
.env
113+
.venv
114+
env/
115+
venv/
116+
ENV/
117+
env.bak/
118+
venv.bak/
119+
120+
# Spyder project settings
121+
.spyderproject
122+
.spyproject
123+
124+
# Rope project settings
125+
.ropeproject
126+
127+
# mkdocs documentation
128+
/site
129+
130+
# mypy
131+
.mypy_cache/
132+
.dmypy.json
133+
dmypy.json
134+
135+
# Pyre type checker
136+
.pyre/

‎.nojekyll‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎LICENSE‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
All code in this repo is currently under exclusive copyright.

‎Makefile‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Minimal makefile for Sphinx documentation
2+
#
3+
4+
# You can set these variables from the command line, and also
5+
# from the environment for the first two.
6+
SPHINXOPTS ?=
7+
SPHINXBUILD ?= sphinx-build
8+
SOURCEDIR = source
9+
BUILDDIR = build
10+
DEPLOYDIR = docs
11+
12+
# Put it first so that "make" without argument is like "make help".
13+
help:
14+
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
15+
16+
deploy: html
17+
cp -r $(BUILDDIR)/html/* $(DEPLOYDIR)
18+
19+
.PHONY: help deploy Makefile
20+
21+
# Catch-all target: route all unknown targets to Sphinx using the new
22+
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
23+
%: Makefile
24+
python -m benchmark
25+
(cd $(SOURCEDIR) && python build.py)
26+
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

‎README.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
DeepLabCut Benchmark
2+
====================
3+
4+
[`[[ benchmark.deeplabcut.org ]]` ](https://benchmark.deeplabcut.org)
5+
6+
Welcome to the DeepLabCut benchmark!
7+
This repo hosts all submitted results, which are available at [benchmark.deeplabcut.org](https://benchmark.deeplabcut.org).
8+
If you are interested in submitting to the benchmark, please find detailed instructions on [benchmark.deeplabcut.org/submission](https://benchmark.deeplabcut.org/submission).
9+
10+
11+
Quickstart for developers
12+
-------------------------
13+
14+
The mandatory requirements for *building the benchmark page* can be installed via
15+
16+
``` bash
17+
$ pip install -r requirements.txt
18+
```
19+
20+
The (non-public) ground truth data needs to be present in `benchmark/data`. Check that this is the case by running
21+
22+
``` bash
23+
find benchmark/data -type f
24+
benchmark/data/CollectedData_Mackenzie.h5
25+
benchmark/data/CollectedData_Daniel.h5
26+
benchmark/data/CollectedData_Valentina.h5
27+
benchmark/data/CollectedData_Mostafizur.h5
28+
```
29+
30+
For using all functionalities of this package and re-running evaluations, a [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut/blob/master/docs/installation.md) installation is additionally required.
31+
32+
Check that the package works as expected by running
33+
34+
``` bash
35+
python -m pytest tests
36+
```
37+
38+
which should finish without errors or warnings.
39+
40+
To re-evaluate all available models, run
41+
42+
``` bash
43+
$ python -m benchmark
44+
```
45+
46+
or, if you want to run in debugging mode,
47+
48+
``` bash
49+
python -m benchmark --nocache --onerror raise
50+
```
51+
52+
from the repository root.
53+
54+
To manually build the documentation, run
55+
56+
``` bash
57+
$ make deploy
58+
```

‎benchmark/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
__version__ = '0.0.1'

‎benchmark/__main__.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
from deeplabcut.benchmark.cli import main
2+
import deeplabcut.benchmark.utils as __utils
3+
4+
results = __utils.import_submodules("benchmark", recursive=True)
5+
6+
if __name__ == "__main__":
7+
main()
8+

‎benchmark/baselines/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Baselines from multi-animal DLC paper

‎benchmark/baselines/__init__.py‎

Whitespace-only changes.

‎benchmark/baselines/data/DLC_dlcrnetms5_CrackingParentingMar24shuffle1_30000-snapshot-30000_assemblies.pickle‎

572 KB
Binary file not shown.

0 commit comments

Comments
 (0)