Skip to content

Commit 92a761b

Browse files
dholzmuellerdholzmueller
committed
added missing files
1 parent 7db53a6 commit 92a761b

File tree

7 files changed

+1503
-0
lines changed

7 files changed

+1503
-0
lines changed

‎docs/source/bench/refine_then_calibrate.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Reproducing results of "Rethinking Early Stopping: Refine, Then Calibrate"
2+
3+
Here, we document how to reproduce results from our paper [Rethinking Early Stopping: Refine, Then Calibrate](https://arxiv.org/abs/2501.19195).
4+
For general instructions on how to set data paths and use slurm,
5+
we refer to the installation page.
6+
The following will be the parts specific to this paper.
7+
8+
## Installation
9+
10+
```bash
11+
pip install probmetrics[extra] # to get smECE
12+
pip install pytabkit[bench,dev]
13+
```
14+
15+
### Original environment
16+
17+
The original conda environment for exact reproduction
18+
is stored in `original_requirements/conda_env_2025_01_15.yml`.
19+
20+
## Downloading datasets
21+
22+
Download the zipped datasets (`dataset-1218.zip`) of the TALENT benchmark from
23+
[here](https://drive.google.com/drive/folders/1j1zt3zQIo8dO6vkO-K-WE6pSrl71bf0z).
24+
Extract them into a folder. Then, use
25+
26+
```commandline
27+
python3 scripts/download_data.py --import_talent_class_small --talent_folder=<unzipped data folder>
28+
```
29+
30+
where the provided data folder should be the `data` folder inside the unzipped results.
31+
32+
## Running experiments
33+
34+
Experiments can be run using `python3 scripts/run_probclass_experiments.py`,
35+
then plots can be generated using `python3 scripts/create_probclass_plots.py`.

‎original_requirements/conda_env_2025_01_15.yml‎

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
name: probclass
2+
channels:
3+
- pytorch
4+
- nvidia
5+
- defaults
6+
dependencies:
7+
- _libgcc_mutex=0.1
8+
- _openmp_mutex=5.1
9+
- blas=1.0
10+
- brotli-python=1.0.9
11+
- bzip2=1.0.8
12+
- ca-certificates=2024.12.31
13+
- certifi=2024.12.14
14+
- charset-normalizer=3.3.2
15+
- cuda-cudart=11.8.89
16+
- cuda-cupti=11.8.87
17+
- cuda-libraries=11.8.0
18+
- cuda-nvrtc=11.8.89
19+
- cuda-nvtx=11.8.86
20+
- cuda-runtime=11.8.0
21+
- cuda-version=12.6
22+
- expat=2.6.4
23+
- ffmpeg=4.3
24+
- filelock=3.13.1
25+
- freetype=2.12.1
26+
- giflib=5.2.2
27+
- gmp=6.2.1
28+
- gnutls=3.6.15
29+
- idna=3.7
30+
- intel-openmp=2023.1.0
31+
- jinja2=3.1.4
32+
- jpeg=9e
33+
- lame=3.100
34+
- lcms2=2.16
35+
- ld_impl_linux-64=2.40
36+
- lerc=4.0.0
37+
- libcublas=11.11.3.6
38+
- libcufft=10.9.0.58
39+
- libcufile=1.11.1.6
40+
- libcurand=10.3.7.77
41+
- libcusolver=11.4.1.48
42+
- libcusparse=11.7.5.86
43+
- libdeflate=1.22
44+
- libffi=3.4.4
45+
- libgcc-ng=11.2.0
46+
- libgomp=11.2.0
47+
- libiconv=1.16
48+
- libidn2=2.3.4
49+
- libjpeg-turbo=2.0.0
50+
- libnpp=11.8.0.86
51+
- libnvjpeg=11.9.0.86
52+
- libpng=1.6.39
53+
- libstdcxx-ng=11.2.0
54+
- libtasn1=4.19.0
55+
- libtiff=4.5.1
56+
- libunistring=0.9.10
57+
- libuuid=1.41.5
58+
- libwebp=1.3.2
59+
- libwebp-base=1.3.2
60+
- llvm-openmp=14.0.6
61+
- lz4-c=1.9.4
62+
- markupsafe=2.1.3
63+
- mkl=2023.1.0
64+
- mkl-service=2.4.0
65+
- mkl_fft=1.3.11
66+
- mkl_random=1.2.8
67+
- mpmath=1.3.0
68+
- ncurses=6.4
69+
- nettle=3.7.3
70+
- networkx=3.2.1
71+
- openh264=2.1.1
72+
- openjpeg=2.5.2
73+
- openssl=3.0.15
74+
- pillow=11.0.0
75+
- pip=24.2
76+
- pysocks=1.7.1
77+
- python=3.12.8
78+
- pytorch=2.5.1
79+
- pytorch-cuda=11.8
80+
- pytorch-mutex=1.0
81+
- pyyaml=6.0.2
82+
- readline=8.2
83+
- requests=2.32.3
84+
- setuptools=72.1.0
85+
- sqlite=3.45.3
86+
- tbb=2021.8.0
87+
- tk=8.6.14
88+
- torchtriton=3.1.0
89+
- torchvision=0.20.1
90+
- typing_extensions=4.12.2
91+
- urllib3=2.2.3
92+
- wheel=0.44.0
93+
- xz=5.4.6
94+
- yaml=0.2.5
95+
- zlib=1.2.13
96+
- zstd=1.5.6
97+
- pip:
98+
- absl-py==2.1.0
99+
- adjusttext==1.3.0
100+
- aiohappyeyeballs==2.4.4
101+
- aiohttp==3.11.11
102+
- aiosignal==1.3.2
103+
- alabaster==1.0.0
104+
- argon2-cffi==23.1.0
105+
- argon2-cffi-bindings==21.2.0
106+
- attrs==24.3.0
107+
- autorank==1.2.1
108+
- babel==2.16.0
109+
- baycomp==1.0.3
110+
- catboost==1.2.7
111+
- cffi==1.17.1
112+
- cir-model==0.2.0
113+
- click==8.1.8
114+
- cloudpickle==3.1.0
115+
- contourpy==1.3.1
116+
- coverage==7.6.10
117+
- cycler==0.12.1
118+
- dask==2024.12.1
119+
- dask-expr==1.1.21
120+
- deprecation==2.1.0
121+
- dill==0.3.9
122+
- docutils==0.21.2
123+
- et-xmlfile==2.0.0
124+
- fire==0.7.0
125+
- fonttools==4.55.3
126+
- frozenlist==1.5.0
127+
- fsspec==2024.12.0
128+
- gpytorch==1.13
129+
- grpcio==1.69.0
130+
- imagesize==1.4.1
131+
- iniconfig==2.0.0
132+
- jaxtyping==0.2.19
133+
- joblib==1.4.2
134+
- jsonschema==4.23.0
135+
- jsonschema-specifications==2024.10.1
136+
- kiwisolver==1.4.8
137+
- liac-arff==2.5.0
138+
- lightgbm==4.5.0
139+
- lightning-utilities==0.11.9
140+
- linear-operator==0.5.3
141+
- locket==1.0.0
142+
- markdown==3.7
143+
- markdown-it-py==3.0.0
144+
- matplotlib==3.7.5
145+
- mdit-py-plugins==0.4.2
146+
- mdurl==0.1.2
147+
- minio==7.2.14
148+
- msgpack==1.1.0
149+
- msgpack-numpy==0.4.8
150+
- multidict==6.1.0
151+
- myst-parser==4.0.0
152+
- netcal==1.3.6
153+
- numpy==1.26.4
154+
- nvidia-ml-py==12.560.30
155+
- nvidia-nccl-cu12==2.24.3
156+
- openml==0.15.0
157+
- openpyxl==3.1.5
158+
- opt-einsum==3.4.0
159+
- packaging==24.2
160+
- pandas==2.2.3
161+
- partd==1.4.2
162+
- patool==3.1.0
163+
- patsy==1.0.1
164+
- plotly==5.24.1
165+
- pluggy==1.5.0
166+
- probmetrics==0.0.1
167+
- propcache==0.2.1
168+
- protobuf==5.29.3
169+
- psutil==6.1.1
170+
- pyarrow==18.1.0
171+
- pycparser==2.22
172+
- pycryptodome==3.21.0
173+
- pygments==2.19.1
174+
- pynvml==12.0.0
175+
- pyparsing==3.2.1
176+
- pyro-api==0.1.2
177+
- pyro-ppl==1.9.1
178+
- pytabkit==1.1.3
179+
- pytest==8.3.4
180+
- pytest-cov==6.0.0
181+
- python-dateutil==2.9.0.post0
182+
- python-graphviz==0.20.3
183+
- pytorch-lightning==2.5.0.post0
184+
- pytz==2024.2
185+
- ray==2.40.0
186+
- referencing==0.35.1
187+
- relplot==1.0
188+
- rpds-py==0.22.3
189+
- scikit-learn==1.5.2
190+
- scipy==1.15.1
191+
- seaborn==0.13.2
192+
- six==1.17.0
193+
- skorch==1.1.0
194+
- snowballstemmer==2.2.0
195+
- sphinx==8.1.3
196+
- sphinx-rtd-theme==3.0.2
197+
- sphinxcontrib-applehelp==2.0.0
198+
- sphinxcontrib-devhelp==2.0.0
199+
- sphinxcontrib-htmlhelp==2.1.0
200+
- sphinxcontrib-jquery==4.1
201+
- sphinxcontrib-jsmath==1.0.1
202+
- sphinxcontrib-qthelp==2.0.0
203+
- sphinxcontrib-serializinghtml==2.0.0
204+
- statsmodels==0.14.4
205+
- swig==4.3.0
206+
- sympy==1.13.1
207+
- tabulate==0.9.0
208+
- tenacity==9.0.0
209+
- tensorboard==2.18.0
210+
- tensorboard-data-server==0.7.2
211+
- termcolor==2.5.0
212+
- threadpoolctl==3.5.0
213+
- tikzplotlib==0.9.8
214+
- toolz==1.0.0
215+
- torchmetrics==1.6.1
216+
- tqdm==4.67.1
217+
- tueplots==0.0.17
218+
- typeguard==4.4.1
219+
- tzdata==2024.2
220+
- venn-abers==1.4.6
221+
- werkzeug==3.1.3
222+
- xgboost==2.1.3
223+
- xlrd==2.0.1
224+
- xmltodict==0.14.2
225+
- yarl==1.18.3

0 commit comments

Comments
 (0)