Skip to content

Commit 69cabcb

Browse files
mhmdk0hasan7n
mhmdk0
and
hasan7n
authored
Add Documentation for the WebUI (#661)
* add .md without videos * tmp * update codespaces startup code * update codespaces startup code * update docs * finalize docs --------- Co-authored-by: hasan7n <hasankassim7@hotmail.com>
1 parent 9d49a24 commit 69cabcb

File tree

3 files changed

+280
-0
lines changed

3 files changed

+280
-0
lines changed

‎docs/getting_started/webui_full_tutorial.md‎

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,279 @@
1+
# Federated Evaluation Tutorial (WebUI version)
2+
3+
## Requirements
4+
5+
To run this tutorial, you need a machine with the following requirements:
6+
7+
- Internet access.
8+
- Web browser (to connect to codespaces)
9+
10+
## Running in cloud via Github Codespaces
11+
12+
As the most easy way to play with the tutorials you can launch a preinstalled [Codespace](https://github.com/features/codespaces) cloud environment for MedPerf by clicking this link:
13+
14+
[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/mlcommons/medperf/tree/tutorial?devcontainer_path=.devcontainer%2Fwebui%2Fdevcontainer.json){target="\_blank"}
15+
16+
After opening the link, proceed to creating the codespace without changing any option. It will take around 7 minutes to get the codespace up and running. Please wait until you see the message `Medperf is ready for local usage` printed on the terminal.
17+
18+
## Run Medperf WebUI Local Server
19+
20+
In your codespaces terminal, run the following command to start the local webUI:
21+
22+
```bash
23+
medperf_webui
24+
```
25+
26+
You will see a long URL in the terminal. Click on the URL (CTRL + Click) to open the web UI in the browser.
27+
![Overview](../images/codespaces_terminal.png)
28+
29+
## About the tutorial
30+
31+
You will be using MedPerf to do federated benchmarking for the task of chest X-ray classification. The data consists of chest X-ray images and their labels. You can take a look at the data located in this folder: `medperf_tutorial/sample_raw_data`.
32+
33+
Throughout the tutorial, you will play three roles:
34+
35+
- The benchmark owner who defines and manages the benchmark experiment.
36+
- The model owner who will provide a model to be benchmarked.
37+
- The data owner who will provide their data for benchmarking the model. (Note that the data stays on the data owner machine)
38+
39+
The following outlines the steps involved:
40+
41+
1. Benchmark Owner: Define and register the benchmark.
42+
2. Model Owner: Register a model
43+
3. Model Owner: Request participation in the benchmark
44+
4. Data Owner: Register a dataset
45+
5. Data Owner: Run data preparation.
46+
6. Data Owner: Request participation
47+
7. Benchmark Owner: Approve the participation requests
48+
8. Data Owner: Run the benchmark
49+
9. Data Owner: Submit the results
50+
10. Benchmark Owner: View the results.
51+
52+
## 1. Benchmark Owner: Define and register the benchmark
53+
54+
First, login as the benchmark owner.
55+
56+
- Use the email `testbo@example.com`.
57+
58+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/login_bmk_owner.mp4" width="800px" controls></video>
59+
60+
### Defining the benchmark Components
61+
62+
The implementation of a valid benchmark workflow is accomplished by implementing three Containers:
63+
64+
- **Data Preparator Container:** This container transforms raw data into a format ready to be ingested by AI models. In this tutorial, the data preparation container will transform chest x-ray images and labels into numpy arrays. You can take a look at the implementation in the following folder: `medperf_tutorial/data_preparator`.
65+
66+
- **Reference Model Container:** Provides an example AI model compatible with the benchmark, so that models respect the input data format and write predictions in a format specified by the benchmark owner. For this tutorial, you can take a look at the implementation in the following folder: `medperf_tutorial/model_custom_cnn`.
67+
68+
- **Metrics Container:** Evaluates model performance by comparing models predictions to the data ground truth labels. For this tutorial, the metrics container will calculate Accuracy and AUC metrics. You can take a look at the implementation in the following folder: `medperf_tutorial/metrics`.
69+
70+
Additionally, a demo/toy dataset should be provided as part of the benchmark to be used for testing the compatibility and validity of participating models.
71+
72+
All these four components are already ready to be used for this tutorial. Below, you will learn how to register them to the MedPerf server and then register the benchmark.
73+
74+
### Register the three Containers
75+
76+
#### Data Preparator Container
77+
78+
Navigate to the `Containers` tab at the top, and click on the `Register a new container` button.
79+
80+
In this tutorial, for the Data Preparator container, the registration should include:
81+
82+
- The URL to the hosted container configuration file, which is:
83+
84+
```
85+
https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/data_preparator/container_config.yaml
86+
```
87+
88+
- The URL to the hosted parameters file, which is:
89+
90+
```
91+
https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/data_preparator/workspace/parameters.yaml
92+
```
93+
94+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/register_data_prep.mp4" width="800px" controls></video>
95+
96+
#### Reference Model Container
97+
98+
Navigate back to the `Containers` tab at the top, and click on the `Register a new container` button.
99+
100+
In this tutorial, for the Reference Model container, the registration should include:
101+
102+
- The URL to the hosted container configuration file:
103+
104+
```
105+
https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/model_custom_cnn/container_config.yaml
106+
```
107+
108+
- The URL to the hosted parameters file:
109+
110+
```
111+
https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/model_custom_cnn/workspace/parameters.yaml
112+
```
113+
114+
- The URL to the hosted additional files tarball file, which includes model weights:
115+
116+
```
117+
https://storage.googleapis.com/medperf-storage/chestxray_tutorial/cnn_weights.tar.gz
118+
```
119+
120+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/register_reference_model.mp4" width="800px" controls></video>
121+
122+
#### Metrics Container
123+
124+
Navigate back to the `Containers` tab at the top, and click on the `Register a new container` button.
125+
126+
In this tutorial, for the Metrics container, the registration should include:
127+
128+
- The URL to the hosted container configuration file:
129+
130+
```
131+
https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/metrics/container_config.yaml
132+
```
133+
134+
- The URL to the hosted parameters file:
135+
136+
```
137+
https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/metrics/workspace/parameters.yaml
138+
```
139+
140+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/register_evaluator.mp4" width="800px" controls></video>
141+
142+
Finally, now after having the containers registered, you can register the benchmark to the MedPerf server.
143+
144+
### Register the benchmark
145+
146+
Navigate to the `Benchmarks` tab at the top, and click on the `Register a new benchmark` button.
147+
148+
You need to keep at hand the following information:
149+
150+
- The Demo/Toy Dataset URL. Here, the URL will be:
151+
152+
```
153+
https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo_data.tar.gz
154+
```
155+
156+
- The names you used for the three containers that you have registered in the previous steps.
157+
158+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/bmk_register.mp4" width="800px" controls></video>
159+
160+
## 2. Model Owner: Register a model
161+
162+
- Logout, then use the email `testmo@example.com` to login as the model owner.
163+
164+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/login_model_owner.mp4" width="800px" controls></video>
165+
166+
The implementation of the model container that will we will be registering in this section can be found in the following folder: `medperf_tutorial/model_mobilenetv2`.
167+
168+
### Register your model
169+
170+
Navigate to the `Containers` tab at the top, and click on the `Register a new container` button.
171+
172+
In this tutorial, for the model owner's container, the registration should include:
173+
174+
- The URL to the hosted container configuration file:
175+
176+
```
177+
https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/model_mobilenetv2/container_config.yaml
178+
```
179+
180+
- The URL to the hosted parameters file:
181+
182+
```
183+
https://raw.githubusercontent.com/mlcommons/medperf/main/examples/chestxray_tutorial/model_mobilenetv2/workspace/parameters.yaml
184+
```
185+
186+
- The URL to the hosted additional files tarball file (which contains the model weights):
187+
188+
```
189+
https://storage.googleapis.com/medperf-storage/chestxray_tutorial/mobilenetv2_weights.tar.gz
190+
```
191+
192+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/model_register.mp4" width="800px" controls></video>
193+
194+
## 3. Model Owner: Request participation in the benchmark
195+
196+
You must request to associate your model with the benchmark in order to be part of it.
197+
198+
To initiate an association request, you need to keep at hand the following information:
199+
200+
- The target benchmark name.
201+
- Your container's name.
202+
203+
Then, visit your container's detail page, and start the association request by clicking on the `Associate with a benchmark` button.
204+
205+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/model_request_association.mp4" width="800px" controls></video>
206+
207+
## 4. Data Owner: Register a dataset
208+
209+
- Logout, then use the email `testdo@example.com` to login as the data owner.
210+
211+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/login_dataset_owner.mp4" width="800px" controls></video>
212+
213+
### Register your dataset
214+
215+
Navigate to the `Datasets` tab at the top, and click on the `Register a new dataset` button.
216+
217+
You will be registering the dataset located at `medperf_tutorial/sample_raw_data`.
218+
219+
To register your dataset, you need to keep at hand the following information:
220+
221+
- A name you wish to have for your dataset.
222+
- A small description of the dataset.
223+
- The source location of your data (e.g., hospital name).
224+
- The path to the data records (here, it is `medperf_tutorial/sample_raw_data/images`).
225+
- The path to the labels of the data (here, it is `medperf_tutorial/sample_raw_data/labels`)
226+
- The benchmark that you wish to participate in. This ensures your data in the next step will be prepared using the benchmark's data preparation container.
227+
228+
!!! note
229+
You will be submitting general information about the data, not the data itself. The data never leaves your machine.
230+
231+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/dataset_register.mp4" width="800px" controls></video>
232+
233+
## 5. Data Owner: Run data preparation
234+
235+
Click on the `prepare` button in your dataset page.
236+
237+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/dataset_prepare.mp4" width="800px" controls></video>
238+
239+
### Set dataset into operational mode
240+
241+
After running preparation, click on the `set opertional` button to mark your dataset as ready for benchmarking. This will also upload some statistics calculated on the dataset according to the benchmark owner's requirement. For the benchmark of this tutorial, the statistics will be the number of cases and the label counts.
242+
243+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/dataset_set_operational.mp4" width="800px" controls></video>
244+
245+
## 6. Data Owner: Request participation
246+
247+
Click on the `associate with benchmark` button in your dataset page.
248+
249+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/dataset_request_association.mp4" width="800px" controls></video>
250+
251+
You won't be able to run the benchmark until the benchmark owner approves your association request.
252+
253+
## 7. Benchmark Owner: Approve the participation requests
254+
255+
Logout, then login as the benchmark owner, navigate to your benchmark page, then accept participation requests from the model owner and from the data owner.
256+
257+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/bmk_approve_associations.mp4" width="800px" controls></video>
258+
259+
## 8. Data Owner: Run the benchmark
260+
261+
Now, Logout, then login as the data owner, click on the `Datasets` tab, navigate to your dataset page, and run the benchmark on your data.
262+
263+
You will see that the reference model has been already run during the association. Click on the `Run` button of the other model to run it.
264+
265+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/run_execution.mp4" width="800px" controls></video>
266+
267+
## 9. Data Owner: Submit the results
268+
269+
Now you can click on the `Submit` button for each model to submit the results to the MedPerf server, so that the benchmark owner can view them. You can click `View Results` before submitting as well.
270+
271+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/view_and_submit_results.mp4" width="800px" controls></video>
272+
273+
## 10. Benchmark Owner: View the results
274+
275+
Now, logout, then login as the benchmark owner, navigate to your benchmark page, and you can see the results.
276+
277+
<video src="https://storage.googleapis.com/medperf-storage/webui_snippets/bmk_view_results.mp4" width="800px" controls></video>
278+
279+
This concludes our tutorial!

‎docs/images/codespaces_terminal.png‎

35.9 KB
Loading

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ nav:
1818
- Bechmark Committee: getting_started/benchmark_owner_demo.md
1919
- Model Owners: getting_started/model_owner_demo.md
2020
- Data Owners: getting_started/data_owner_demo.md
21+
- Full Tutorial (WebUI): getting_started/webui_full_tutorial.md
2122
- ADVANCED CONCEPTS:
2223
- Containers:
2324
- MedPerf-compatible Containers: containers/containers.md

0 commit comments

Comments
 (0)