Tensorflow crashes

[ronias]

Thanks for the previous reply about where to locate the T1w files.

I managed to get the files and run the code, but when the python code (in the file orig_mod_NNEval.py) is executed, it crashes at line 130:
b = sess.run(pred,feed_dict={X:mri}

When this line is exectued it just give the following warnings:
2023-09-18 01:44:09.620351: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 805306368 exceeds 10% of free system memory.
2023-09-18 01:44:16.856842: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 805306368 exceeds 10% of free system memory.

The 805306368 bytes in the warning equals to just 0.8GB of memory and I have 8GB of memory in my machine.

After one minute the program process ends with no error, I even wrapped it in a try-except (like try-catch) and added a print, but it doesnt get into the execpt part.

I'm using tensor flow cpu mode.
Dou you have any idea what can cause it?

Thanks.

[jocwilliams]

Hi @ronias, I think that warning is acceptable. I have a few questions to start getting to the bottom of this issue.

1. Can you post your full call to main_CNNStriatalSegmentation?
2. Can you post a full list of intermediates and final outputs that exist after you run main_CNNStriatalSegmentation?
3. How do you know it is crashing in orig_mod_NNEval.py at line 130? If it is just that warning, I don't think that means a crash on its own.

Thanks.

Best,
John

[ronias]

1. This is the line in the code with the varaible names:
   main_CNNStriatalSegmentation('T1_acpc_template_brain',T1_acpc_template_brain, ...
   'template_acpc_brainmask',template_acpc_brainmask, ...
   'segmentation_outputs_directory',segmentation_outputs_directory, ...
   'BOLD_template_image',BOLD_template_image);

When I debug it and see the variables values then this is what sent:
main_CNNStriatalSegmentation ('T1_acpc_template_brain','D:/Workspace/striatal_segmentation/MNINonLinear/T1w_restore.nii', ...
'template_acpc_brainmask', 'D:/Workspace/striatal_segmentation/MNINonLinear/brainmask_fs.nii', ...
'segmentation_outputs_directory','D:/Workspace/striatal_segmentation/testdir', ...
'BOLD_template_image',"");

2. As I mentioned the python code is crashing thus the main_CNNStriatalSegmentation prints errors after the call to the python code, so I am not sure which outputs do you want?
   But here is a list of all intermediates and their values up to the point of the call to python:

checkpointFileMatch = 0
checkpointFilePath = 'D:\Workspace\striatal_segmentation\StriatalSegmentation\checkpoint'
checkpointFileString = 'model_checkpoint_path: "D:'
checkpointFileText = 'odel_checkpoint_path: "D:/Workspace/striatal_segmentation/StriatalSegmentation/model"all_model_checkpoint_paths: "D:/Workspace/striatal_segmentation/StriatalSegmentation/model"'
segmentation_model_file = 'D:\Workspace\striatal_segmentation\StriatalSegmentation\model'
segmentation_network_weights_directory = 'D:\Workspace\striatal_segmentation\StriatalSegmentation'
segmentation_outputs_directory = 'D:/Workspace/striatal_segmentation/testdir'
segmentation_python_code = 'D:\Workspace\striatal_segmentation\orig_mod_NNEval.py'
segmentation_python_output_intermediate_fullpath = 'D:\Workspace\striatal_segmentation\testdir\CNN_striatal_python_output_intermediate.mat'
T1_acpc_restore_brain = 'D:\Workspace\striatal_segmentation\testdir\striatalCNNres_striatalCNNrotated_T1w_restore.nii'

3. I added prints in the python code before and after line 130 (the line: b = sess.run(pred,feed_dict={X:mri}).
   I got the print on the screen of before it, but not the print after it, I didnt even got an error or exception.
   I also tried to run the python code seperatly in Pycharm and just added the neccecry input agruments, I debuged it and saw the crash at this line.

4. The input files I'm using are from the Human Connectome Project - Young Adult, package: WU-Minn HCP Data - 1200 Subjects, single subject, Structural Preprocessed.
   The file is in T1w_restore.nii.gz and the mask is in brainmask_fs.nii.gz

Thank you.