Skip to content

Commit 72725fa

Browse files
hasan7nmhmdk0
hasan7n
and
mhmdk0
authored
Confidential Computing with GCP (#675)
* add asset policy management * server side changes for model assets * client side changes for model assets * update some tests * update server seed * craete database migrations * fix bug in migrations * finalize client side * fix data migration script * add user extension in admin panel * finalize * add examples * fix server permission * finalize * misc updates, add download results * add rano cc example * add rano test, stream gcp logs * add skip compatibility tests, fix rano example * fix serial logging * allow update cc config * update auth tests and encrypted chestxray test * remove local containers compatibility tests * update tests * change model-container design * temporary disable models webui * some bugfixes * remove associations from container webui * tmp: add model html, js, routes * update webui * update ui print message to not confuse users about container vs. model IDs * fix data migration script * fix import issue * fix seed.py for tutorial demo * remove model type from select dropdown - fix typo from model_detail * fix e2e tests * add tooltips, placeholders for asset registration * update unit tests * update server tests * tmp: debug training tests * fix training ci test * fix linter issues * update documentation * use proper defaults * fix todict * update pydantic models design * fix pydantic issues * fix default values for demo dataset * make storage migrations --------- Co-authored-by: mhmdk0 <mohammadkassim54.mk@gmail.com>
1 parent c6650f7 commit 72725fa

File tree

304 files changed

+9999
-3068
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

304 files changed

+9999
-3068
lines changed

‎.github/workflows/train-ci.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@ jobs:
4040
working-directory: ./server
4141
run: python seed.py --cert cert.crt
4242

43+
# - name: Setup tmate session
44+
# uses: mxschmitt/action-tmate@v3
45+
4346
- name: Run client integration tests
4447
working-directory: .
4548
run: sh cli/cli_tests_training.sh -f

‎cli/auth_tests.sh‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -88,9 +88,9 @@ medperf container submit --name prep -m $PREP_MLCUBE -p $PREP_PARAMS --operation
8888
checkFailed "Prep submission failed"
8989
PREP_UID=$(medperf container ls | tail -n 1 | tr -s ' ' | cut -d ' ' -f 2)
9090

91-
medperf container submit --name model1 -m $MODEL_MLCUBE -p $MODEL1_PARAMS -a $MODEL_ADD --operational
91+
medperf model submit --name model1 -m $MODEL_MLCUBE -p $MODEL1_PARAMS -a $MODEL_ADD --operational
9292
checkFailed "Model1 submission failed"
93-
MODEL1_UID=$(medperf container ls | tail -n 1 | tr -s ' ' | cut -d ' ' -f 2)
93+
MODEL1_UID=$(medperf model ls | tail -n 1 | tr -s ' ' | cut -d ' ' -f 2)
9494

9595
# wait 30s to make sure the token will be refreshed under the hood
9696
# (so that this functionality is tested). The chosen place of this is arbitrary
@@ -117,7 +117,7 @@ echo "\n"
117117
echo "====================================="
118118
echo "Submit benchmark"
119119
echo "====================================="
120-
medperf benchmark submit --name bmk --description bmk --demo-url $DEMO_URL --data-preparation-container $PREP_UID --reference-model-container $MODEL1_UID --evaluator-container $METRICS_UID --operational
120+
medperf benchmark submit --name bmk --description bmk --demo-url $DEMO_URL --data-preparation-container $PREP_UID --reference-model $MODEL1_UID --evaluator-container $METRICS_UID --operational
121121
checkFailed "Benchmark submission failed"
122122
BMK_UID=$(medperf benchmark ls | tail -n 1 | tr -s ' ' | cut -d ' ' -f 2)
123123

‎cli/cli_chestxray_tutorial_test.sh‎

Lines changed: 6 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,6 @@ print_eval medperf profile create -n testbenchmark
2525
checkFailed "testbenchmark profile creation failed"
2626
print_eval medperf profile create -n testdata
2727
checkFailed "testdata profile creation failed"
28-
print_eval medperf profile create -n noserver
29-
checkFailed "noserver profile creation failed"
3028
print_eval medperf profile create -n testprivate
3129
checkFailed "testprivate profile creation failed"
3230
print_eval medperf profile set --server https://example.com
@@ -162,11 +160,11 @@ echo "\n"
162160
echo "====================================="
163161
echo "Submit a private model"
164162
echo "====================================="
165-
print_eval medperf container submit --name privmodel \
163+
print_eval medperf model submit --name privmodel \
166164
-m $CHESTXRAY_ENCRYPTED_MODEL -p $CHESTXRAY_ENCRYPTED_MODEL_PARAMS \
167165
-a $CHESTXRAY_ENCRYPTED_MODEL_ADD --decryption_key $PRIVATE_MODEL_LOCAL/key.bin --operational
168-
checkFailed "private container submission failed"
169-
PMODEL_UID=$(medperf container ls | grep privmodel | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
166+
checkFailed "private model submission failed"
167+
PMODEL_UID=$(medperf model ls | grep privmodel | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
170168
##########################################################
171169

172170
echo "\n"
@@ -175,7 +173,7 @@ echo "\n"
175173
echo "====================================="
176174
echo "Running private model association"
177175
echo "====================================="
178-
print_eval medperf container associate -m $PMODEL_UID -b 1 -y
176+
print_eval medperf model associate -m $PMODEL_UID -b 1 -y
179177
checkFailed "private model association failed"
180178
##########################################################
181179

@@ -185,7 +183,7 @@ echo "\n"
185183
echo "====================================="
186184
echo "Give Access to Private Model"
187185
echo "====================================="
188-
print_eval medperf container grant_access --model-id $PMODEL_UID --benchmark-id 1 -y
186+
print_eval medperf model grant_access --model-id $PMODEL_UID --benchmark-id 1 -y
189187
checkFailed "Failed to Give Model Access to Data owner"
190188
##########################################################
191189

@@ -206,7 +204,7 @@ echo "====================================="
206204
echo "Running benchmark execution step - Public"
207205
echo "====================================="
208206
# Create results
209-
print_eval medperf run -b 1 -d $DSET_UID -m 5 -y
207+
print_eval medperf run -b 1 -d $DSET_UID -m 1 -y
210208
checkFailed "Benchmark execution step failed (public)"
211209
##########################################################
212210

@@ -223,60 +221,6 @@ checkFailed "Benchmark execution step failed (private)"
223221

224222
echo "\n"
225223

226-
##########################################################
227-
echo "====================================="
228-
echo " Offline Compatibility Test - Public "
229-
echo "====================================="
230-
231-
# Test offline compatibility test
232-
print_eval wget -P $MODEL_LOCAL/workspace/additional_files "https://storage.googleapis.com/medperf-storage/chestxray_tutorial/cnn_weights.tar.gz"
233-
print_eval tar -xzvf $MODEL_LOCAL/workspace/additional_files/cnn_weights.tar.gz -C $MODEL_LOCAL/workspace/additional_files
234-
235-
## Change the server and logout just to make sure this command will work without connecting to a server
236-
print_eval medperf profile activate noserver
237-
checkFailed "noserver profile activation failed"
238-
239-
240-
print_eval medperf test run --offline --no-cache \
241-
--demo_dataset_url https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo_data.tar.gz \
242-
--demo_dataset_hash "71faabd59139bee698010a0ae3a69e16d97bc4f2dde799d9e187b94ff9157c00" \
243-
-p $PREP_LOCAL/container_config.yaml \
244-
-m $MODEL_LOCAL/container_config.yaml \
245-
-e $METRIC_LOCAL/container_config.yaml \
246-
--data_preparator_parameters $PREP_LOCAL/workspace/parameters.yaml \
247-
--model_parameters $MODEL_LOCAL/workspace/parameters.yaml \
248-
--evaluator_parameters $METRIC_LOCAL/workspace/parameters.yaml \
249-
--model_additional_files $MODEL_LOCAL/workspace/additional_files/
250-
251-
checkFailed "offline compatibility test execution step failed - public model"
252-
##########################################################
253-
254-
echo "\n"
255-
256-
##########################################################
257-
echo "====================================="
258-
echo " Offline Compatibility Test - Private "
259-
echo "====================================="
260-
print_eval medperf test run --offline --no-cache \
261-
--demo_dataset_url https://storage.googleapis.com/medperf-storage/chestxray_tutorial/demo_data.tar.gz \
262-
--demo_dataset_hash "71faabd59139bee698010a0ae3a69e16d97bc4f2dde799d9e187b94ff9157c00" \
263-
-p $PREP_LOCAL/container_config.yaml \
264-
-m $PRIVATE_MODEL_LOCAL/container_config.yaml \
265-
-e $METRIC_LOCAL/container_config.yaml \
266-
-d $PRIVATE_MODEL_LOCAL/key.bin \
267-
--data_preparator_parameters $PREP_LOCAL/workspace/parameters.yaml \
268-
--model_parameters $MODEL_LOCAL/workspace/parameters.yaml \
269-
--evaluator_parameters $METRIC_LOCAL/workspace/parameters.yaml \
270-
--model_additional_files $MODEL_LOCAL/workspace/additional_files/
271-
272-
checkFailed "offline compatibility test execution step failed - private model"
273-
274-
print_eval rm $MODEL_LOCAL/workspace/additional_files/cnn_weights.tar.gz
275-
print_eval rm $MODEL_LOCAL/workspace/additional_files/cnn_weights.pth
276-
##########################################################
277-
278-
echo "\n"
279-
280224
##########################################################
281225
echo "====================================="
282226
echo "Logout users"
@@ -309,9 +253,6 @@ checkFailed "Profile deletion failed"
309253
print_eval medperf profile delete testdata
310254
checkFailed "Profile deletion failed"
311255

312-
print_eval medperf profile delete noserver
313-
checkFailed "Profile deletion failed"
314-
315256
print_eval medperf profile delete testprivate
316257
checkFailed "Profile deletion failed"
317258

‎cli/cli_tests.sh‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -116,25 +116,25 @@ checkFailed "Prep submission failed"
116116
PREP_UID=$(medperf container ls | grep mock-prep | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
117117
echo "PREP_UID=$PREP_UID" >> "$LAST_ENV_FILE"
118118

119-
print_eval medperf container submit --name model1 -m $MODEL_MLCUBE -p $MODEL1_PARAMS -a $MODEL_ADD --operational
119+
print_eval medperf model submit --name model1 -m $MODEL_MLCUBE -p $MODEL1_PARAMS -a $MODEL_ADD --operational
120120
checkFailed "Model1 submission failed"
121-
MODEL1_UID=$(medperf container ls | grep model1 | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
121+
MODEL1_UID=$(medperf model ls | grep model1 | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
122122
echo "MODEL1_UID=$MODEL1_UID" >> "$LAST_ENV_FILE"
123123

124-
print_eval medperf container submit --name model2 -m $MODEL_ARCHIVE_MLCUBE -p $MODEL2_PARAMS -a $MODEL_ADD --operational
124+
print_eval medperf model submit --name model2 -m $MODEL_ARCHIVE_MLCUBE -p $MODEL2_PARAMS -a $MODEL_ADD --operational
125125
checkFailed "Model2 submission failed"
126-
MODEL2_UID=$(medperf container ls | grep model2 | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
126+
MODEL2_UID=$(medperf model ls | grep model2 | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
127127
echo "MODEL2_UID=$MODEL2_UID" >> "$LAST_ENV_FILE"
128128

129129
# Container with singularity section
130-
print_eval medperf --platform singularity container submit --name model3 -m $MODEL_WITH_SINGULARITY -p $MODEL3_PARAMS -a $MODEL_ADD --operational
130+
print_eval medperf --platform singularity model submit --name model3 -m $MODEL_WITH_SINGULARITY -p $MODEL3_PARAMS -a $MODEL_ADD --operational
131131
checkFailed "Model3 submission failed"
132-
MODEL3_UID=$(medperf container ls | grep model3 | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
132+
MODEL3_UID=$(medperf model ls | grep model3 | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
133133
echo "MODEL3_UID=$MODEL3_UID" >> "$LAST_ENV_FILE"
134134

135-
print_eval medperf container submit --name model-fail -m $FAILING_MODEL_MLCUBE -p $MODEL4_PARAMS -a $MODEL_ADD --operational
135+
print_eval medperf model submit --name model-fail -m $FAILING_MODEL_MLCUBE -p $MODEL4_PARAMS -a $MODEL_ADD --operational
136136
checkFailed "failing model submission failed"
137-
FAILING_MODEL_UID=$(medperf container ls | grep model-fail | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
137+
FAILING_MODEL_UID=$(medperf model ls | grep model-fail | head -n 1 | tr -s ' ' | cut -d ' ' -f 2)
138138
echo "FAILING_MODEL_UID=$FAILING_MODEL_UID" >> "$LAST_ENV_FILE"
139139

140140
print_eval medperf container submit --name mock-metrics -m $METRIC_MLCUBE -p $METRIC_PARAMS --operational
@@ -159,7 +159,7 @@ echo "\n"
159159
echo "====================================="
160160
echo "Submit benchmark"
161161
echo "====================================="
162-
print_eval medperf benchmark submit --name bmk --description bmk --demo-url $DEMO_URL --data-preparation-container $PREP_UID --reference-model-container $MODEL1_UID --evaluator-container $METRICS_UID --operational
162+
print_eval medperf benchmark submit --name bmk --description bmk --demo-url $DEMO_URL --data-preparation-container $PREP_UID --reference-model $MODEL1_UID --evaluator-container $METRICS_UID --operational
163163
checkFailed "Benchmark submission failed"
164164
BMK_UID=$(medperf benchmark ls | grep bmk | tail -n 1 | tr -s ' ' | cut -d ' ' -f 2)
165165
echo "BMK_UID=$BMK_UID" >> "$LAST_ENV_FILE"
@@ -363,7 +363,7 @@ echo "\n"
363363
echo "====================================="
364364
echo "Running model2 association"
365365
echo "====================================="
366-
print_eval medperf container associate -m $MODEL2_UID -b $BMK_UID -y
366+
print_eval medperf model associate -m $MODEL2_UID -b $BMK_UID -y
367367
checkFailed "Model2 association failed"
368368
##########################################################
369369

@@ -376,7 +376,7 @@ echo "====================================="
376376
# this will run two types of singularity containers:
377377
# 1) an already built singularity image (model 3)
378378
# 2) a docker image to be converted (metrics)
379-
print_eval medperf --platform singularity container associate -m $MODEL3_UID -b $BMK_UID -y
379+
print_eval medperf --platform singularity model associate -m $MODEL3_UID -b $BMK_UID -y
380380
checkFailed "Model3 association failed"
381381
##########################################################
382382

@@ -386,7 +386,7 @@ echo "\n"
386386
echo "======================================================"
387387
echo "Running failing model association (This will NOT fail)"
388388
echo "======================================================"
389-
print_eval medperf container associate -m $FAILING_MODEL_UID -b $BMK_UID -y
389+
print_eval medperf model associate -m $FAILING_MODEL_UID -b $BMK_UID -y
390390
checkFailed "Failing model association failed"
391391
##########################################################
392392

0 commit comments

Comments
 (0)