Successful import label at MONAI label plugin in 3D slicer but failed to model training #491
Unanswered
ragwingtmu921
asked this question in
Q&A
Replies: 2 comments
-
|
@diazandr3s can you please help |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hi @ragwingtmu921, thanks for your message and sorry we just reply. Let me ask you some questions to first understand the data's nature. Is this a single-label task? How many segments are you trying to segment? Do all the images have segmentation? BTW, MONAI Label has changed a lot since Nov 2021, could you please make sure you used the latest version? Thanks again and I hope this helps, |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Operating system: Ubuntu 18.04
Slicer version: Slicer-4.13.0-2021-09-29-linux-amd64
The version of Python: Python 3.7.6
Expected behavior: Import label at MONAI label at 3D slicer & update model
Actual behavior: show RuntimeError: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7f9809fa36a0> during model training
Dear all, I want build a new model from cold start, like the scenario 1 at the MONAI label teaching video at Bootcamp 2021
I prepared 200 up case of CT images with whole CT slices & lesion segmetation, both file formats are nifti (CT image: nii.gz & label: nii). Then setup & launch the MONAI label sever successfully in bare matal, also fetched the MONAI label sever successfully in 3D slicer
I loaded the each case with click the Next sample button and import the label of each case in 3D slicer, then submit label to MONAI label sever successfully( The label file appeared in the datasets folder). Then the MONAI label sever start model training, the cases was splited as training & vaildation and start training as 50 ecophes with showing the training loss
But the model training always interrupted with the ending of “ RuntimeError: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7f9809fa36a0>”.
I tried to click the update model to training again but also show same error.
I tried to set MONAI server & MONAI label in docker images & mount my study folder, then launch the MONAI label sever, fetched the MONAI label sever(docker) successfully in 3D slicer(local) and restart the whole course again but received same error; it ended with “ RuntimeError: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7f9809fa36a0>” again
How can I fixed this error and complete the training course? Thank you for everything.
The following log is the failure of training
[2021-11-01 08:47:30,525] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Train request: {'deepedit_train': {'name': 'model_01', 'pretrained': 1, 'device': 'cuda', 'max_epochs': 50, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1}}
[2021-11-01 08:47:30,528] [Thread-138] [INFO] (monailabel.utils.async_tasks.utils:49) - COMMAND:: /home/usm500/anaconda3/lib/python3.7/site-packages/monailabel/scripts/run_monailabel_app.sh /mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon/apps/deepedit /mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon_03 train {"deepedit_train": {"name": "model_01", "pretrained": 1, "device": "cuda", "max_epochs": 50, "val_split": 0.2, "train_batch_size": 1, "val_batch_size": 1}}
Virtual Env:
USING PYTHON: /home/usm500/anaconda3/bin/python
Using PYTHONPATH:: /home/usm500:/home/usm500/.local/bin
[2021-11-01 08:47:30,681] [MainThread] [INFO] (main:38) - Initializing App from: /mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon/apps/deepedit; studies: /mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon_03; conf: {}
[2021-11-01 08:47:32,496] [MainThread] [INFO] (monailabel.utils.others.class_utils:34) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2021-11-01 08:47:33,133] [MainThread] [INFO] (main:95) - EPISTEMIC Enabled: 0; Samples: 5
[2021-11-01 08:47:33,133] [MainThread] [INFO] (main:99) - TTA Enabled: 0; Samples: 5
[2021-11-01 08:47:33,133] [MainThread] [INFO] (monailabel.interfaces.app:112) - Init Datastore for: /mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon_03
[2021-11-01 08:47:33,133] [MainThread] [INFO] (monailabel.datastore.local:125) - Extensions: ['.nii.gz', '.nii']
[2021-11-01 08:47:33,133] [MainThread] [INFO] (monailabel.datastore.local:126) - Auto Reload: True
[2021-11-01 08:47:33,141] [MainThread] [INFO] (monailabel.datastore.local:519) - reconcile datastore...
[2021-11-01 08:47:33,162] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2021-11-01 08:47:33,163] [MainThread] [INFO] (monailabel.datastore.local:146) - Start observing external modifications on datastore (AUTO RELOAD)
[2021-11-01 08:47:33,164] [MainThread] [INFO] (monailabel.interfaces.app:331) - Running training: deepedit_train: {'description': 'Train DeepEdit model for 3D Images', 'config': {'name': 'model_01', 'pretrained': 1, 'device': 'cuda', 'max_epochs': 50, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1}} => {'name': 'model_01', 'pretrained': 1, 'device': 'cuda', 'max_epochs': 50, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1}
[2021-11-01 08:47:33,164] [MainThread] [INFO] (monailabel.tasks.train.basic_train:251) - Train Request (input): {'name': 'model_01', 'pretrained': 1, 'device': 'cuda', 'max_epochs': 50, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1}
[2021-11-01 08:47:33,164] [MainThread] [INFO] (monailabel.tasks.train.basic_train:252) - Train Request (final): {'name': 'model_01', 'pretrained': 1, 'device': 'cuda', 'max_epochs': 50, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1}
[2021-11-01 08:47:33,174] [MainThread] [INFO] (monailabel.tasks.train.basic_train:269) - Total Records for Training: 79
[2021-11-01 08:47:33,174] [MainThread] [INFO] (monailabel.tasks.train.basic_train:270) - Total Records for Validation: 20
[2021-11-01 08:47:35,559] [MainThread] [INFO] (monailabel.tasks.train.basic_train:161) - Adding Validation Handler to run every '1' interval
[2021-11-01 08:47:35,559] [MainThread] [INFO] (monailabel.tasks.train.basic_train:299) - Load Path /mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon/apps/deepedit/model/model_01/model.pt
[2021-11-01 08:47:35,563] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:696) - Engine run resuming from iteration 0, epoch 0 until 50 epochs
[2021-11-01 08:47:35,614] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:138) - Restored all variables from /mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon/apps/deepedit/model/model_01/model.pt
[2021-11-01 08:47:49,745] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 1/79 -- train_loss: 0.6727
[2021-11-01 08:48:01,730] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 2/79 -- train_loss: 0.9453
[2021-11-01 08:48:14,027] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 3/79 -- train_loss: 0.9789
[2021-11-01 08:48:25,996] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 4/79 -- train_loss: 0.5917
[2021-11-01 08:48:46,620] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 5/79 -- train_loss: 0.9977
[2021-11-01 08:48:56,440] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 6/79 -- train_loss: 0.6041
[2021-11-01 08:49:08,031] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 7/79 -- train_loss: 0.4042
[2021-11-01 08:49:28,486] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 8/79 -- train_loss: 0.6898
[2021-11-01 08:49:41,842] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 9/79 -- train_loss: 0.9957
[2021-11-01 08:49:55,193] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:229) - Epoch: 1/50, Iter: 10/79 -- train_loss: 0.6804
USING:: app = /mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon/apps/deepedit
USING:: studies = /mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon_03
USING:: method = train
USING:: request = {"deepedit_train": {"name": "model_01", "pretrained": 1, "device": "cuda", "max_epochs": 50, "val_split": 0.2, "train_batch_size": 1, "val_batch_size": 1}}
USING:: output = None
USING:: debug = False
SETTINGS
{
"MONAI_LABEL_API_STR": "",
"MONAI_LABEL_PROJECT_NAME": "MONAILabel",
"MONAI_LABEL_APP_DIR": "/mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon/apps/deepedit",
"MONAI_LABEL_STUDIES": "/mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon_03",
"MONAI_LABEL_APP_CONF": {},
"MONAI_LABEL_DICOMWEB_USERNAME": "None",
"MONAI_LABEL_DICOMWEB_PASSWORD": "None",
"MONAI_LABEL_QIDO_PREFIX": "",
"MONAI_LABEL_WADO_PREFIX": "",
"MONAI_LABEL_STOW_PREFIX": "",
"MONAI_LABEL_DATASTORE_AUTO_RELOAD": true,
"MONAI_LABEL_DATASTORE_FILE_EXT": [
".nii.gz",
".nii"
],
"MONAI_LABEL_SERVER_PORT": 8000,
"MONAI_LABEL_CORS_ORIGINS": []
}
=== Transform input info -- AddInitialSeedPointd ===
[2021-11-01 08:50:07,465] [MainThread] [INFO] (DataStats:99) -
=== Transform input info -- AddInitialSeedPointd ===
image statistics:
Type: <class 'numpy.ndarray'>
Shape: (1, 128, 128, 64)
Value range: (-0.6703714728355408, 5.180522918701172)
[2021-11-01 08:50:07,466] [MainThread] [INFO] (DataStats:597) - image statistics:
Type: <class 'numpy.ndarray'>
Shape: (1, 128, 128, 64)
Value range: (-0.6703714728355408, 5.180522918701172)
label statistics:
Type: <class 'numpy.ndarray'>
Shape: (1, 128, 128, 64)
Value range: (0.0, 0.0)
[2021-11-01 08:50:07,467] [MainThread] [INFO] (DataStats:597) - label statistics:
Type: <class 'numpy.ndarray'>
Shape: (1, 128, 128, 64)
Value range: (0.0, 0.0)
image_meta_dict statistics:
Type: <class 'dict'>
Value: {'sizeof_hdr': array(348, dtype=int32), 'extents': array(0, dtype=int32), 'session_error': array(0, dtype=int16), 'dim_info': array(0, dtype=uint8), 'dim': array([ 3, 512, 512, 298, 1, 1, 1, 1], dtype=int16), 'intent_p1': array(0., dtype=float32), 'intent_p2': array(0., dtype=float32), 'intent_p3': array(0., dtype=float32), 'intent_code': array(0, dtype=int16), 'datatype': array(4, dtype=int16), 'bitpix': array(16, dtype=int16), 'slice_start': array(0, dtype=int16), 'pixdim': array([1. , 0.84765625, 0.84765625, 2. , 0. ,
0. , 0. , 0. ], dtype=float32), 'vox_offset': array(0., dtype=float32), 'scl_slope': array(nan, dtype=float32), 'scl_inter': array(nan, dtype=float32), 'slice_end': array(0, dtype=int16), 'slice_code': array(0, dtype=uint8), 'xyzt_units': array(2, dtype=uint8), 'cal_max': array(0., dtype=float32), 'cal_min': array(0., dtype=float32), 'slice_duration': array(0., dtype=float32), 'toffset': array(0., dtype=float32), 'glmax': array(0, dtype=int32), 'glmin': array(0, dtype=int32), 'qform_code': array(1, dtype=int16), 'sform_code': array(0, dtype=int16), 'quatern_b': array(0., dtype=float32), 'quatern_c': array(0., dtype=float32), 'quatern_d': array(1., dtype=float32), 'qoffset_x': array(214.57617, dtype=float32), 'qoffset_y': array(334.07617, dtype=float32), 'qoffset_z': array(-568.7, dtype=float32), 'srow_x': array([0., 0., 0., 0.], dtype=float32), 'srow_y': array([0., 0., 0., 0.], dtype=float32), 'srow_z': array([0., 0., 0., 0.], dtype=float32), 'affine': array([[ 1. , 0. , 0. , -218.42382812],
[ 0. , 1. , 0. , -98.92382812],
[ 0. , 0. , 1. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'original_affine': array([[ -0.84765625, 0. , 0. , 214.57617188],
[ 0. , -0.84765625, 0. , 334.07617188],
[ 0. , 0. , 2. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'as_closest_canonical': False, 'spatial_shape': array([512, 512, 298], dtype=int16), 'original_channel_dim': 'no_channel', 'filename_or_obj': '/mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon_03/602339.nii.gz'}
[2021-11-01 08:50:07,469] [MainThread] [INFO] (DataStats:597) - image_meta_dict statistics:
Type: <class 'dict'>
Value: {'sizeof_hdr': array(348, dtype=int32), 'extents': array(0, dtype=int32), 'session_error': array(0, dtype=int16), 'dim_info': array(0, dtype=uint8), 'dim': array([ 3, 512, 512, 298, 1, 1, 1, 1], dtype=int16), 'intent_p1': array(0., dtype=float32), 'intent_p2': array(0., dtype=float32), 'intent_p3': array(0., dtype=float32), 'intent_code': array(0, dtype=int16), 'datatype': array(4, dtype=int16), 'bitpix': array(16, dtype=int16), 'slice_start': array(0, dtype=int16), 'pixdim': array([1. , 0.84765625, 0.84765625, 2. , 0. ,
0. , 0. , 0. ], dtype=float32), 'vox_offset': array(0., dtype=float32), 'scl_slope': array(nan, dtype=float32), 'scl_inter': array(nan, dtype=float32), 'slice_end': array(0, dtype=int16), 'slice_code': array(0, dtype=uint8), 'xyzt_units': array(2, dtype=uint8), 'cal_max': array(0., dtype=float32), 'cal_min': array(0., dtype=float32), 'slice_duration': array(0., dtype=float32), 'toffset': array(0., dtype=float32), 'glmax': array(0, dtype=int32), 'glmin': array(0, dtype=int32), 'qform_code': array(1, dtype=int16), 'sform_code': array(0, dtype=int16), 'quatern_b': array(0., dtype=float32), 'quatern_c': array(0., dtype=float32), 'quatern_d': array(1., dtype=float32), 'qoffset_x': array(214.57617, dtype=float32), 'qoffset_y': array(334.07617, dtype=float32), 'qoffset_z': array(-568.7, dtype=float32), 'srow_x': array([0., 0., 0., 0.], dtype=float32), 'srow_y': array([0., 0., 0., 0.], dtype=float32), 'srow_z': array([0., 0., 0., 0.], dtype=float32), 'affine': array([[ 1. , 0. , 0. , -218.42382812],
[ 0. , 1. , 0. , -98.92382812],
[ 0. , 0. , 1. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'original_affine': array([[ -0.84765625, 0. , 0. , 214.57617188],
[ 0. , -0.84765625, 0. , 334.07617188],
[ 0. , 0. , 2. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'as_closest_canonical': False, 'spatial_shape': array([512, 512, 298], dtype=int16), 'original_channel_dim': 'no_channel', 'filename_or_obj': '/mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon_03/602339.nii.gz'}
label_meta_dict statistics:
Type: <class 'dict'>
Value: {'sizeof_hdr': array(348, dtype=int32), 'extents': array(0, dtype=int32), 'session_error': array(0, dtype=int16), 'dim_info': array(0, dtype=uint8), 'dim': array([ 3, 512, 512, 298, 1, 1, 1, 1], dtype=int16), 'intent_p1': array(0., dtype=float32), 'intent_p2': array(0., dtype=float32), 'intent_p3': array(0., dtype=float32), 'intent_code': array(0, dtype=int16), 'datatype': array(4, dtype=int16), 'bitpix': array(16, dtype=int16), 'slice_start': array(0, dtype=int16), 'pixdim': array([1. , 0.84765625, 0.84765625, 2. , 0. ,
0. , 0. , 0. ], dtype=float32), 'vox_offset': array(0., dtype=float32), 'scl_slope': array(nan, dtype=float32), 'scl_inter': array(nan, dtype=float32), 'slice_end': array(0, dtype=int16), 'slice_code': array(0, dtype=uint8), 'xyzt_units': array(2, dtype=uint8), 'cal_max': array(0., dtype=float32), 'cal_min': array(0., dtype=float32), 'slice_duration': array(0., dtype=float32), 'toffset': array(0., dtype=float32), 'glmax': array(0, dtype=int32), 'glmin': array(0, dtype=int32), 'qform_code': array(1, dtype=int16), 'sform_code': array(1, dtype=int16), 'quatern_b': array(0., dtype=float32), 'quatern_c': array(0., dtype=float32), 'quatern_d': array(1., dtype=float32), 'qoffset_x': array(214.57617, dtype=float32), 'qoffset_y': array(334.07617, dtype=float32), 'qoffset_z': array(-568.7, dtype=float32), 'srow_x': array([ -0.84765625, 0. , 0. , 214.57617 ],
dtype=float32), 'srow_y': array([ 0. , -0.84765625, 0. , 334.07617 ],
dtype=float32), 'srow_z': array([ 0. , 0. , 2. , -568.7], dtype=float32), 'affine': array([[ 1. , 0. , 0. , -218.42382812],
[ 0. , 1. , 0. , -98.92382812],
[ 0. , 0. , 1. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'original_affine': array([[ -0.84765625, 0. , 0. , 214.57617188],
[ 0. , -0.84765625, 0. , 334.07617188],
[ 0. , 0. , 2. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'as_closest_canonical': False, 'spatial_shape': array([512, 512, 298], dtype=int16), 'original_channel_dim': 'no_channel', 'filename_or_obj': '/mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon_03/labels/final/602339.nii.gz'}
[2021-11-01 08:50:07,471] [MainThread] [INFO] (DataStats:597) - label_meta_dict statistics:
Type: <class 'dict'>
Value: {'sizeof_hdr': array(348, dtype=int32), 'extents': array(0, dtype=int32), 'session_error': array(0, dtype=int16), 'dim_info': array(0, dtype=uint8), 'dim': array([ 3, 512, 512, 298, 1, 1, 1, 1], dtype=int16), 'intent_p1': array(0., dtype=float32), 'intent_p2': array(0., dtype=float32), 'intent_p3': array(0., dtype=float32), 'intent_code': array(0, dtype=int16), 'datatype': array(4, dtype=int16), 'bitpix': array(16, dtype=int16), 'slice_start': array(0, dtype=int16), 'pixdim': array([1. , 0.84765625, 0.84765625, 2. , 0. ,
0. , 0. , 0. ], dtype=float32), 'vox_offset': array(0., dtype=float32), 'scl_slope': array(nan, dtype=float32), 'scl_inter': array(nan, dtype=float32), 'slice_end': array(0, dtype=int16), 'slice_code': array(0, dtype=uint8), 'xyzt_units': array(2, dtype=uint8), 'cal_max': array(0., dtype=float32), 'cal_min': array(0., dtype=float32), 'slice_duration': array(0., dtype=float32), 'toffset': array(0., dtype=float32), 'glmax': array(0, dtype=int32), 'glmin': array(0, dtype=int32), 'qform_code': array(1, dtype=int16), 'sform_code': array(1, dtype=int16), 'quatern_b': array(0., dtype=float32), 'quatern_c': array(0., dtype=float32), 'quatern_d': array(1., dtype=float32), 'qoffset_x': array(214.57617, dtype=float32), 'qoffset_y': array(334.07617, dtype=float32), 'qoffset_z': array(-568.7, dtype=float32), 'srow_x': array([ -0.84765625, 0. , 0. , 214.57617 ],
dtype=float32), 'srow_y': array([ 0. , -0.84765625, 0. , 334.07617 ],
dtype=float32), 'srow_z': array([ 0. , 0. , 2. , -568.7], dtype=float32), 'affine': array([[ 1. , 0. , 0. , -218.42382812],
[ 0. , 1. , 0. , -98.92382812],
[ 0. , 0. , 1. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'original_affine': array([[ -0.84765625, 0. , 0. , 214.57617188],
[ 0. , -0.84765625, 0. , 334.07617188],
[ 0. , 0. , 2. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'as_closest_canonical': False, 'spatial_shape': array([512, 512, 298], dtype=int16), 'original_channel_dim': 'no_channel', 'filename_or_obj': '/mnt/7901e5c1-d8b9-4209-b4e2-9e4953dfe4d6/Colon_03/labels/final/602339.nii.gz'}
image_transforms statistics:
Type: <class 'list'>
Value: [{'class': 'Spacingd', 'id': 140503054217744, 'orig_size': (512, 512, 298), 'extra_info': {'meta_key': 'image_meta_dict', 'old_affine': array([[ -0.84765625, 0. , 0. , 214.57617188],
[ 0. , -0.84765625, 0. , 334.07617188],
[ 0. , 0. , 2. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'mode': 'bilinear', 'padding_mode': 'border', 'align_corners': False}}, {'class': 'Orientationd', 'id': 140503054217872, 'orig_size': (434, 434, 595), 'extra_info': {'meta_key': 'image_meta_dict', 'old_affine': array([[ -1. , 0. , 0. , 214.57617188],
[ 0. , -1. , 0. , 334.07617188],
[ 0. , 0. , 1. , -568.70001221],
[ 0. , 0. , 0. , 1. ]])}}, {'class': 'RandRotated', 'id': 140503054218256, 'orig_size': (434, 434, 595), 'extra_info': {'rot_mat': array([[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.]]), 'mode': 'bilinear', 'padding_mode': 'border', 'align_corners': False}, 'do_transforms': False}, {'class': 'Resized', 'id': 140503054218320, 'orig_size': (434, 434, 595), 'extra_info': {'mode': 'area', 'align_corners': 'none'}}]
[2021-11-01 08:50:07,472] [MainThread] [INFO] (DataStats:597) - image_transforms statistics:
Type: <class 'list'>
Value: [{'class': 'Spacingd', 'id': 140503054217744, 'orig_size': (512, 512, 298), 'extra_info': {'meta_key': 'image_meta_dict', 'old_affine': array([[ -0.84765625, 0. , 0. , 214.57617188],
[ 0. , -0.84765625, 0. , 334.07617188],
[ 0. , 0. , 2. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'mode': 'bilinear', 'padding_mode': 'border', 'align_corners': False}}, {'class': 'Orientationd', 'id': 140503054217872, 'orig_size': (434, 434, 595), 'extra_info': {'meta_key': 'image_meta_dict', 'old_affine': array([[ -1. , 0. , 0. , 214.57617188],
[ 0. , -1. , 0. , 334.07617188],
[ 0. , 0. , 1. , -568.70001221],
[ 0. , 0. , 0. , 1. ]])}}, {'class': 'RandRotated', 'id': 140503054218256, 'orig_size': (434, 434, 595), 'extra_info': {'rot_mat': array([[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.]]), 'mode': 'bilinear', 'padding_mode': 'border', 'align_corners': False}, 'do_transforms': False}, {'class': 'Resized', 'id': 140503054218320, 'orig_size': (434, 434, 595), 'extra_info': {'mode': 'area', 'align_corners': 'none'}}]
label_transforms statistics:
Type: <class 'list'>
Value: [{'class': 'Spacingd', 'id': 140503054217744, 'orig_size': (512, 512, 298), 'extra_info': {'meta_key': 'label_meta_dict', 'old_affine': array([[ -0.84765625, 0. , 0. , 214.57617188],
[ 0. , -0.84765625, 0. , 334.07617188],
[ 0. , 0. , 2. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'mode': 'nearest', 'padding_mode': 'border', 'align_corners': False}}, {'class': 'Orientationd', 'id': 140503054217872, 'orig_size': (434, 434, 595), 'extra_info': {'meta_key': 'label_meta_dict', 'old_affine': array([[ -1. , 0. , 0. , 214.57617188],
[ 0. , -1. , 0. , 334.07617188],
[ 0. , 0. , 1. , -568.70001221],
[ 0. , 0. , 0. , 1. ]])}}, {'class': 'RandRotated', 'id': 140503054218256, 'orig_size': (434, 434, 595), 'extra_info': {'rot_mat': array([[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.]]), 'mode': 'nearest', 'padding_mode': 'border', 'align_corners': False}, 'do_transforms': False}, {'class': 'Resized', 'id': 140503054218320, 'orig_size': (434, 434, 595), 'extra_info': {'mode': 'nearest', 'align_corners': 'none'}}]
[2021-11-01 08:50:07,472] [MainThread] [INFO] (DataStats:597) - label_transforms statistics:
Type: <class 'list'>
Value: [{'class': 'Spacingd', 'id': 140503054217744, 'orig_size': (512, 512, 298), 'extra_info': {'meta_key': 'label_meta_dict', 'old_affine': array([[ -0.84765625, 0. , 0. , 214.57617188],
[ 0. , -0.84765625, 0. , 334.07617188],
[ 0. , 0. , 2. , -568.70001221],
[ 0. , 0. , 0. , 1. ]]), 'mode': 'nearest', 'padding_mode': 'border', 'align_corners': False}}, {'class': 'Orientationd', 'id': 140503054217872, 'orig_size': (434, 434, 595), 'extra_info': {'meta_key': 'label_meta_dict', 'old_affine': array([[ -1. , 0. , 0. , 214.57617188],
[ 0. , -1. , 0. , 334.07617188],
[ 0. , 0. , 1. , -568.70001221],
[ 0. , 0. , 0. , 1. ]])}}, {'class': 'RandRotated', 'id': 140503054218256, 'orig_size': (434, 434, 595), 'extra_info': {'rot_mat': array([[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.]]), 'mode': 'nearest', 'padding_mode': 'border', 'align_corners': False}, 'do_transforms': False}, {'class': 'Resized', 'id': 140503054218320, 'orig_size': (434, 434, 595), 'extra_info': {'mode': 'nearest', 'align_corners': 'none'}}]
[2021-11-01 08:50:07,472] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:847) - Current run is terminating due to exception: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7fc96aa2f510>
[2021-11-01 08:50:07,472] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:147) - Exception: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7fc96aa2f510>
Traceback (most recent call last):
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/transforms/transform.py", line 92, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/transforms/transform.py", line 60, in _apply_transform
return transform(parameters)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/apps/deepgrow/transforms.py", line 148, in call
d[self.guidance] = json.dumps(self._apply(d[self.label], self.sid).astype(int).tolist())
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/apps/deepgrow/transforms.py", line 119, in _apply
raise AssertionError("Not a valid Label")
AssertionError: Not a valid Label
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 801, in _run_once_on_dataset
self.state.batch = next(self._dataloader_iter)
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/data/dataset.py", line 95, in getitem
return self._transform(index)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/data/dataset.py", line 288, in _transform
return self._post_transform(pre_random_item)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/data/dataset.py", line 234, in _post_transform
item_transformed = apply_transform(_transform, item_transformed)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/transforms/transform.py", line 116, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7fc96aa2f510>
[2021-11-01 08:50:07,474] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:773) - Engine run is terminating due to exception: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7fc96aa2f510>
[2021-11-01 08:50:07,474] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:147) - Exception: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7fc96aa2f510>
Traceback (most recent call last):
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/transforms/transform.py", line 92, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/transforms/transform.py", line 60, in _apply_transform
return transform(parameters)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/apps/deepgrow/transforms.py", line 148, in call
d[self.guidance] = json.dumps(self._apply(d[self.label], self.sid).astype(int).tolist())
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/apps/deepgrow/transforms.py", line 119, in _apply
raise AssertionError("Not a valid Label")
AssertionError: Not a valid Label
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 744, in _internal_run
time_taken = self._run_once_on_dataset()
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 848, in _run_once_on_dataset
self._handle_exception(e)
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 424, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/handlers/stats_handler.py", line 148, in exception_raised
raise e
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 801, in _run_once_on_dataset
self.state.batch = next(self._dataloader_iter)
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/data/dataset.py", line 95, in getitem
return self._transform(index)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/data/dataset.py", line 288, in _transform
return self._post_transform(pre_random_item)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/data/dataset.py", line 234, in _post_transform
item_transformed = apply_transform(_transform, item_transformed)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/transforms/transform.py", line 116, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7fc96aa2f510>
Traceback (most recent call last):
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/transforms/transform.py", line 92, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/transforms/transform.py", line 60, in _apply_transform
return transform(parameters)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/apps/deepgrow/transforms.py", line 148, in call
d[self.guidance] = json.dumps(self._apply(d[self.label], self.sid).astype(int).tolist())
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/apps/deepgrow/transforms.py", line 119, in _apply
raise AssertionError("Not a valid Label")
AssertionError: Not a valid Label
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/usm500/anaconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/usm500/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monailabel/interfaces/utils/app.py", line 114, in
run_main()
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monailabel/interfaces/utils/app.py", line 99, in run_main
result = a.train(request)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monailabel/interfaces/app.py", line 333, in train
result = task(req, self.datastore())
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monailabel/tasks/train/basic_train.py", line 327, in call
trainer.run()
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/engines/trainer.py", line 56, in run
super().run()
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/engines/workflow.py", line 250, in run
super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 701, in run
return self._internal_run()
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 774, in _internal_run
self._handle_exception(e)
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 424, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/handlers/stats_handler.py", line 148, in exception_raised
raise e
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 744, in _internal_run
time_taken = self._run_once_on_dataset()
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 848, in _run_once_on_dataset
self._handle_exception(e)
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 467, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 424, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/handlers/stats_handler.py", line 148, in exception_raised
raise e
File "/home/usm500/.local/lib/python3.7/site-packages/ignite/engine/engine.py", line 801, in _run_once_on_dataset
self.state.batch = next(self._dataloader_iter)
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/usm500/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/data/dataset.py", line 95, in getitem
return self._transform(index)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/data/dataset.py", line 288, in _transform
return self._post_transform(pre_random_item)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/data/dataset.py", line 234, in _post_transform
item_transformed = apply_transform(_transform, item_transformed)
File "/home/usm500/anaconda3/lib/python3.7/site-packages/monai/transforms/transform.py", line 116, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.apps.deepgrow.transforms.AddInitialSeedPointd object at 0x7fc96aa2f510>
[2021-11-01 08:50:07,936] [Thread-138] [INFO] (monailabel.utils.async_tasks.utils:71) - Return code: 1
Beta Was this translation helpful? Give feedback.
All reactions