about the active learning #1419

spencer03172023 · 2023-04-26T10:59:28Z

spencer03172023
Apr 26, 2023

Hi,

i follow the instructions from https://docs.monai.io/projects/label/en/latest/quickstart.html try to reproduce the active learning process.

But seems it was not run successfully with updated model.pt output and saved.
I post the log as below, anyone can help me?

[2023-04-26 18:49:44,974] [31098] [MainThread] [INFO] (monailabel.interfaces.app:492) - New label saved for: spleen_9 => spleen_9
[2023-04-26 18:49:45,026] [31098] [Thread-1] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-04-26 18:49:56,905] [31098] [MainThread] [INFO] (monailabel.utils.async_tasks.task:41) - Train request: {'model': 'swin_unetr_btcv_segmentation', 'device': 'cuda', 'pretrained': True, 'max_epochs': 50, 'val_split': 0.2, 'multi_gpu': True, 'gpus': 'all', 'tracking': 'None', 'tracking_uri': '', 'tracking_experiment_name': '', 'model_filename': 'model.pt', 'client_id': 'user-xyz', 'bundle_path': '/home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation'}
[2023-04-26 18:49:56,906] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:49) - Before:: /home/ubuntu/qiushenjie/monai:
[2023-04-26 18:49:56,906] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:53) - After:: /home/ubuntu/qiushenjie/monai::/home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation
[2023-04-26 18:49:56,907] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:65) - COMMAND:: /home/ubuntu/qiushenjie/monai/venv/bin/python -m monailabel.interfaces.utils.app -m train -r {"model":"swin_unetr_btcv_segmentation","device":"cuda","pretrained":true,"max_epochs":50,"val_split":0.2,"multi_gpu":true,"gpus":"all","tracking":"None","tracking_uri":"","tracking_experiment_name":"","model_filename":"model.pt","client_id":"user-xyz","bundle_path":"/home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation"}
[2023-04-26 18:49:57,161] [21773] [MainThread] [INFO] (main:37) - Initializing App from: /home/ubuntu/qiushenjie/monai/monaibundle; studies: /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr; conf: {'models': 'swin_unetr_btcv_segmentation'}
[2023-04-26 18:49:59,904] [21773] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.utils.others.generic:303) - +++ Adding Bundle from Zoo: swin_unetr_btcv_segmentation => /home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.utils.others.generic:313) - +++ Using Bundle Models: ['swin_unetr_btcv_segmentation']
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.interfaces.app:134) - Init Datastore for: /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: False; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2023-04-26 18:50:08,925] [21773] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-04-26 18:50:09,267] [21773] [MainThread] [INFO] (main:67) - +++ Adding Inferer:: swin_unetr_btcv_segmentation => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x7f3b323a3490>
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (main:81) - +++ Adding Trainer:: swin_unetr_btcv_segmentation => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x7f3b323a34d0>
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (main:91) - Active Learning Strategies:: ['random', 'first']
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/ubuntu/.cache/monailabel/sessions
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
usage: app.py [-h] [-a APP] [-s STUDIES] -m
{infer,train,info,batch_infer,scoring} [-r REQUEST] [-o OUTPUT]
[-d] [--multi_gpu] [--local_rank LOCAL_RANK]
optional arguments:
-h, --help show this help message and exit
-a APP, --app APP
-s STUDIES, --studies STUDIES
-m {infer,train,info,batch_infer,scoring}, --method {infer,train,info,batch_infer,scoring}
-r REQUEST, --request REQUEST
-o OUTPUT, --output OUTPUT
-d, --debug
--multi_gpu
--local_rank LOCAL_RANK
[2023-04-26 18:50:09,696] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 255

tangy5 · 2023-04-26T15:14:19Z

tangy5
Apr 26, 2023
Maintainer

Hi @spencer03172023 , thanks for trying the app. From above logs, we didn't see errors. Can you describe more or paste complete error logs? Thanks.

1 reply

spencer03172023 Apr 26, 2023
Author

2023-04-26 18:47:26,311 - USING:: version = False
2023-04-26 18:47:26,311 - USING:: app = /home/ubuntu/qiushenjie/monai/monaibundle
2023-04-26 18:47:26,311 - USING:: studies = /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr
2023-04-26 18:47:26,311 - USING:: verbose = INFO
2023-04-26 18:47:26,311 - USING:: conf = [['models', 'swin_unetr_btcv_segmentation']]
2023-04-26 18:47:26,311 - USING:: host = 0.0.0.0
2023-04-26 18:47:26,311 - USING:: port = 8000
2023-04-26 18:47:26,311 - USING:: uvicorn_app = monailabel.app:app
2023-04-26 18:47:26,311 - USING:: ssl_keyfile = None
2023-04-26 18:47:26,311 - USING:: ssl_certfile = None
2023-04-26 18:47:26,311 - USING:: ssl_keyfile_password = None
2023-04-26 18:47:26,311 - USING:: ssl_ca_certs = None
2023-04-26 18:47:26,311 - USING:: workers = None
2023-04-26 18:47:26,311 - USING:: limit_concurrency = None
2023-04-26 18:47:26,311 - USING:: access_log = False
2023-04-26 18:47:26,311 - USING:: log_config = None
2023-04-26 18:47:26,311 - USING:: dryrun = False
2023-04-26 18:47:26,311 - USING:: action = start_server
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_API_STR =
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_APP_DIR =
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_STUDIES =
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}'
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_AUTH_REALM_URI = http://localhost:8080/realms/monailabel
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_AUTH_TIMEOUT = 10
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_AUTH_TOKEN_USERNAME = preferred_username
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_AUTH_TOKEN_EMAIL = email
2023-04-26 18:47:26,311 - ENV SETTINGS:: MONAI_LABEL_AUTH_TOKEN_NAME = name
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_AUTH_TOKEN_ROLES = realm_access#roles
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_AUTH_ROLE_ADMIN = monailabel-admin
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_AUTH_ROLE_RESEARCHER = monailabel-researcher
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_AUTH_ROLE_REVIEWER = monailabel-reviewer
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_AUTH_ROLE_ANNOTATOR = monailabel-annotator
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_AUTH_ROLE_USER = monailabel-user
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_URL =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_USERNAME =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PASSWORD =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_API_KEY =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_CACHE_PATH =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PROJECT =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_ASSET_PATH =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_DSA_ANNOTATION_GROUPS =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX = None
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX = None
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX = None
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CONVERT_TO_NIFTI = True
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_SEARCH_FILTER = '{"Modality": "CT"}'
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_EXPIRY = 180
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PROXY_TIMEOUT = 30.0
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_READ_TIMEOUT = 5.0
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_READ_ONLY = False
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '[".nii.gz", ".nii", ".nrrd", ".jpg", ".png", ".tif", ".svs", ".xml"]'
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]'
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_TRACKING_ENABLED = True
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_TRACKING_URI =
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_ZOO_SOURCE = github
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_ZOO_REPO = Project-MONAI/model-zoo/hosting_storage_v1
2023-04-26 18:47:26,312 - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True
2023-04-26 18:47:26,312 -
Allow Origins: ['']
[2023-04-26 18:47:26,554] [31098] [MainThread] [INFO] (uvicorn.error:75) - Started server process [31098]
[2023-04-26 18:47:26,555] [31098] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2023-04-26 18:47:26,555] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: /home/ubuntu/qiushenjie/monai/monaibundle; studies: /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr; conf: {'models': 'swin_unetr_btcv_segmentation'}
[2023-04-26 18:47:26,598] [31098] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-04-26 18:47:36,451] [31098] [MainThread] [INFO] (monailabel.utils.others.generic:303) - +++ Adding Bundle from Zoo: swin_unetr_btcv_segmentation => /home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation
[2023-04-26 18:47:36,451] [31098] [MainThread] [INFO] (monailabel.utils.others.generic:313) - +++ Using Bundle Models: ['swin_unetr_btcv_segmentation']
[2023-04-26 18:47:36,451] [31098] [MainThread] [INFO] (monailabel.interfaces.app:134) - Init Datastore for: /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr
[2023-04-26 18:47:36,451] [31098] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2023-04-26 18:47:36,455] [31098] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-04-26 18:47:36,455] [31098] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD)
[2023-04-26 18:47:37,319] [31098] [MainThread] [INFO] (main:67) - +++ Adding Inferer:: swin_unetr_btcv_segmentation => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x7fc018d53690>
[2023-04-26 18:47:37,319] [31098] [MainThread] [INFO] (main:81) - +++ Adding Trainer:: swin_unetr_btcv_segmentation => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x7fc010433050>
[2023-04-26 18:47:37,319] [31098] [MainThread] [INFO] (main:91) - Active Learning Strategies:: ['random', 'first']
[2023-04-26 18:47:37,319] [31098] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/ubuntu/.cache/monailabel/sessions
[2023-04-26 18:47:37,319] [31098] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-04-26 18:47:37,320] [31098] [MainThread] [INFO] (monailabel.interfaces.app:468) - App Init - completed
[2023-04-26 18:47:37,320] [timeloop] [INFO] Starting Timeloop..
[2023-04-26 18:47:37,320] [31098] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2023-04-26 18:47:37,320] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7fc00c1f1050>
[2023-04-26 18:47:37,320] [31098] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7fc00c1f1050>
[2023-04-26 18:47:37,320] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2023-04-26 18:47:37,320] [31098] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2023-04-26 18:47:37,320] [31098] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete.
[2023-04-26 18:47:37,320] [31098] [MainThread] [INFO] (uvicorn.error:211) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2023-04-26 18:48:26,007] [31098] [MainThread] [INFO] (monailabel.endpoints.activelearning:43) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
[2023-04-26 18:48:26,008] [31098] [MainThread] [INFO] (monailabel.tasks.activelearning.random:47) - Random: Selected Image: spleen_9; Weight: 1682506106
[2023-04-26 18:48:26,010] [31098] [MainThread] [INFO] (monailabel.endpoints.activelearning:59) - Next sample: {'id': 'spleen_9', 'weight': 1682506106, 'path': '/home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr/spleen_9.nii.gz', 'ts': 1682412961, 'name': 'spleen_9.nii.gz'}
[2023-04-26 18:48:29,655] [31098] [MainThread] [INFO] (monailabel.endpoints.infer:160) - Infer Request: {'model': 'swin_unetr_btcv_segmentation', 'image': 'spleen_9', 'device': 'cuda', 'model_filename': 'model.pt', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'}
[2023-04-26 18:48:29,655] [31098] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:285) - Infer Request (final): {'device': 'cuda', 'model_filename': 'model.pt', 'model': 'swin_unetr_btcv_segmentation', 'image': '/home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr/spleen_9.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'A pre-trained model for volumetric (3D) multi-organ segmentation from CT image'}
[2023-04-26 18:48:29,663] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s)
[2023-04-26 18:48:29,663] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model_filename', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path']
[2023-04-26 18:48:32,319] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImageTensord): Time: 2.6555; image: torch.Size([512, 512, 41])(torch.float32)
[2023-04-26 18:48:32,320] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0005; image: torch.Size([1, 512, 512, 41])(torch.float32)
[2023-04-26 18:48:32,321] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Orientationd): Time: 0.0009; image: torch.Size([1, 512, 512, 41])(torch.float32)
[2023-04-26 18:48:32,607] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.2863; image: torch.Size([1, 255, 255, 101])(torch.float32)
[2023-04-26 18:48:32,622] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.0145; image: torch.Size([1, 255, 255, 101])(torch.float32)
[2023-04-26 18:48:32,622] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.0001; image: torch.Size([1, 255, 255, 101])(torch.float32)
[2023-04-26 18:48:32,624] [31098] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:477) - Inferer:: cuda => SlidingWindowInferer => {'roi_size': [96, 96, 96], 'sw_batch_size': 4, 'overlap': 0.5, 'mode': constant, 'sigma_scale': 0.125, 'padding_mode': constant, 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'cpu_thresh': None, 'roi_weight_map': None}
[2023-04-26 18:48:32,624] [31098] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:426) - Infer model path: /home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation/models/model.pt
[2023-04-26 18:48:47,651] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s)
[2023-04-26 18:48:47,651] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model_filename', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'image_meta_dict', 'latencies', 'pred']
[2023-04-26 18:48:47,653] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.002; image: torch.Size([1, 255, 255, 101])(torch.float32); pred: torch.Size([14, 255, 255, 101])(torch.float32)
[2023-04-26 18:48:47,907] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Invertd): Time: 0.253; image: torch.Size([1, 255, 255, 101])(torch.float32); pred: torch.Size([14, 512, 512, 41])(torch.float32)
[2023-04-26 18:48:47,907] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 0.0002; image: torch.Size([1, 255, 255, 101])(torch.float32); pred: torch.Size([1, 512, 512, 41])(torch.float32)
[2023-04-26 18:48:47,907] [31098] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0001; image: torch.Size([1, 255, 255, 101])(torch.float32); pred: torch.Size([512, 512, 41])(torch.float32)
[2023-04-26 18:48:47,907] [31098] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:577) - Writing Result...
[2023-04-26 18:48:47,907] [31098] [MainThread] [INFO] (monailabel.transform.writer:189) - Result ext: .nrrd; write_to_file: True; dtype: uint8
[2023-04-26 18:48:47,990] [31098] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:348) - ++ Latencies => Total: 18.3354; Pre: 2.9670; Inferer: 15.0247; Invert: 0.0000; Post: 0.2603; Write: 0.0832
[2023-04-26 18:48:47,991] [31098] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:364) - Result File: /tmp/tmph8ot5r61.nrrd
[2023-04-26 18:48:47,991] [31098] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:365) - Result Json Keys: ['label_names', 'latencies']
[2023-04-26 18:49:44,970] [31098] [MainThread] [INFO] (monailabel.endpoints.datastore:100) - Saving Label for spleen_9 for tag: final by admin
[2023-04-26 18:49:44,971] [31098] [MainThread] [INFO] (monailabel.endpoints.datastore:111) - Save Label params: {"label_info": [{"name": "spleen", "idx": 1}, {"name": "right kidney", "idx": 2}, {"name": "left kideny", "idx": 3}, {"name": "gallbladder", "idx": 4}, {"name": "esophagus", "idx": 5}, {"name": "liver", "idx": 6}, {"name": "stomach", "idx": 7}, {"name": "aorta", "idx": 8}, {"name": "ivc", "idx": 9}, {"name": "portal and splenic veins", "idx": 10}, {"name": "pancreas", "idx": 11}, {"name": "right adrenal gland", "idx": 12}, {"name": "left adrenal gland", "idx": 13}], "client_id": "user-xyz"}
[2023-04-26 18:49:44,971] [31098] [MainThread] [INFO] (monailabel.datastore.local:486) - Saving Label for Image: spleen_9; Tag: final; Info: {'label_info': [{'name': 'spleen', 'idx': 1}, {'name': 'right kidney', 'idx': 2}, {'name': 'left kideny', 'idx': 3}, {'name': 'gallbladder', 'idx': 4}, {'name': 'esophagus', 'idx': 5}, {'name': 'liver', 'idx': 6}, {'name': 'stomach', 'idx': 7}, {'name': 'aorta', 'idx': 8}, {'name': 'ivc', 'idx': 9}, {'name': 'portal and splenic veins', 'idx': 10}, {'name': 'pancreas', 'idx': 11}, {'name': 'right adrenal gland', 'idx': 12}, {'name': 'left adrenal gland', 'idx': 13}], 'client_id': 'user-xyz'}
[2023-04-26 18:49:44,971] [31098] [MainThread] [INFO] (monailabel.datastore.local:494) - Adding Label: spleen_9 => final => /tmp/tmp4y4io3ds.nii.gz
[2023-04-26 18:49:44,972] [31098] [MainThread] [INFO] (monailabel.datastore.local:510) - Label Info: {'label_info': [{'name': 'spleen', 'idx': 1}, {'name': 'right kidney', 'idx': 2}, {'name': 'left kideny', 'idx': 3}, {'name': 'gallbladder', 'idx': 4}, {'name': 'esophagus', 'idx': 5}, {'name': 'liver', 'idx': 6}, {'name': 'stomach', 'idx': 7}, {'name': 'aorta', 'idx': 8}, {'name': 'ivc', 'idx': 9}, {'name': 'portal and splenic veins', 'idx': 10}, {'name': 'pancreas', 'idx': 11}, {'name': 'right adrenal gland', 'idx': 12}, {'name': 'left adrenal gland', 'idx': 13}], 'client_id': 'user-xyz', 'ts': 1682506184, 'name': 'spleen_9.nii.gz'}
[2023-04-26 18:49:44,974] [31098] [MainThread] [INFO] (monailabel.interfaces.app:492) - New label saved for: spleen_9 => spleen_9
[2023-04-26 18:49:45,026] [31098] [Thread-1] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-04-26 18:49:56,905] [31098] [MainThread] [INFO] (monailabel.utils.async_tasks.task:41) - Train request: {'model': 'swin_unetr_btcv_segmentation', 'device': 'cuda', 'pretrained': True, 'max_epochs': 50, 'val_split': 0.2, 'multi_gpu': True, 'gpus': 'all', 'tracking': 'None', 'tracking_uri': '', 'tracking_experiment_name': '', 'model_filename': 'model.pt', 'client_id': 'user-xyz', 'bundle_path': '/home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation'}
[2023-04-26 18:49:56,906] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:49) - Before:: /home/ubuntu/qiushenjie/monai:
[2023-04-26 18:49:56,906] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:53) - After:: /home/ubuntu/qiushenjie/monai::/home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation
[2023-04-26 18:49:56,907] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:65) - COMMAND:: /home/ubuntu/qiushenjie/monai/venv/bin/python -m monailabel.interfaces.utils.app -m train -r {"model":"swin_unetr_btcv_segmentation","device":"cuda","pretrained":true,"max_epochs":50,"val_split":0.2,"multi_gpu":true,"gpus":"all","tracking":"None","tracking_uri":"","tracking_experiment_name":"","model_filename":"model.pt","client_id":"user-xyz","bundle_path":"/home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation"}
[2023-04-26 18:49:57,161] [21773] [MainThread] [INFO] (main:37) - Initializing App from: /home/ubuntu/qiushenjie/monai/monaibundle; studies: /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr; conf: {'models': 'swin_unetr_btcv_segmentation'}
[2023-04-26 18:49:59,904] [21773] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.utils.others.generic:303) - +++ Adding Bundle from Zoo: swin_unetr_btcv_segmentation => /home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.utils.others.generic:313) - +++ Using Bundle Models: ['swin_unetr_btcv_segmentation']
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.interfaces.app:134) - Init Datastore for: /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: False; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '*.xml']
[2023-04-26 18:50:08,925] [21773] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-04-26 18:50:09,267] [21773] [MainThread] [INFO] (main:67) - +++ Adding Inferer:: swin_unetr_btcv_segmentation => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x7f3b323a3490>
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (main:81) - +++ Adding Trainer:: swin_unetr_btcv_segmentation => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x7f3b323a34d0>
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (main:91) - Active Learning Strategies:: ['random', 'first']
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/ubuntu/.cache/monailabel/sessions
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
usage: app.py [-h] [-a APP] [-s STUDIES] -m
{infer,train,info,batch_infer,scoring} [-r REQUEST] [-o OUTPUT]
[-d] [--multi_gpu] [--local_rank LOCAL_RANK]
optional arguments:
-h, --help show this help message and exit
-a APP, --app APP
-s STUDIES, --studies STUDIES
-m {infer,train,info,batch_infer,scoring}, --method {infer,train,info,batch_infer,scoring}
-r REQUEST, --request REQUEST
-o OUTPUT, --output OUTPUT
-d, --debug
--multi_gpu
--local_rank LOCAL_RANK
[2023-04-26 18:50:09,696] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 255

Thanks for your quick reply, please see attached as the whole log from the service start.

diazandr3s · 2023-04-28T11:45:27Z

diazandr3s
Apr 28, 2023
Maintainer

Hi @spencer03172023,

Thanks for the logs.
I want to check the process you're following. You are using the bundle swin_unetr_btcv_segmentation on the dataset /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr, right?
Which MONAI Label version are you using? Can you share the full command you used to start the server? Are you trying to use the dropout technique as active learning?

2 replies

spencer03172023 Apr 28, 2023
Author

Thanks for your reply.

correct, I use bundle Swin_unetr_btcv_sgmentation on the dataset as you mentioned.

my steps as below:

Start app service on terminal with command line
almost duplicate the procedure as
https://github.com/Project-MONAI/tutorials/blob/main/monailabel/monailabel_monaibundle_3dslicer_multiorgan_seg.ipynb
then I start 3d slice, and load the service.
use automation segmentation to get one sample's segmentation result
modified the segmentation and save the label
run "training"
got the log as I posted

two things I want to mentioned here, since in the first time I tried this, I got cuda out of memory error, then I set 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512' to fix this.

another error I met is that "id" = "training" error, then I modified the code change train to training(I may post the detail later)
then I did fix this error.

after that, I can do active learning for training process, but seems no updated weights output and all I saw was the log I posted before

diazandr3s May 5, 2023
Maintainer

Thanks for the detailed post, @spencer03172023.

This is strange.

Can you please try the Segmentation model (https://github.com/Project-MONAI/MONAILabel/tree/main/sample-apps/radiology#segmentation), which also segments abdominal organs, and let us know if you experience the same issue?

I'd like to check whether this is a bundle model issue.

Please let us know.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

about the active learning #1419

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

about the active learning #1419

Uh oh!

spencer03172023 Apr 26, 2023

Replies: 2 comments · 3 replies

Uh oh!

tangy5 Apr 26, 2023 Maintainer

Uh oh!

Uh oh!

spencer03172023 Apr 26, 2023 Author

Uh oh!

diazandr3s Apr 28, 2023 Maintainer

Uh oh!

Uh oh!

spencer03172023 Apr 28, 2023 Author

Uh oh!

diazandr3s May 5, 2023 Maintainer

spencer03172023
Apr 26, 2023

Replies: 2 comments 3 replies

tangy5
Apr 26, 2023
Maintainer

spencer03172023 Apr 26, 2023
Author

diazandr3s
Apr 28, 2023
Maintainer

spencer03172023 Apr 28, 2023
Author

diazandr3s May 5, 2023
Maintainer