Error initialising source docker for GPUs in the TRE

[mxochicale]

🐛Bug

TRE
https://tre.arc.ucl.ac.uk/

• CHME003924252a    Automatic Medical Image Reporting (AMIR)
• CHME003924252b    Automatic Medical Image Reporting (AMIR)

To reproduce

Running Open HPC Desktop https://tre.arc.ucl.ac.uk/ > following https://docs.tre.arc.ucl.ac.uk/guides/gpu/

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

run the pytorch container

[sungshic]

try using "--gpus all" instead of the --device option?

[mxochicale]

Just tried:

docker run \
  -it \
  --rm \
  --gpus all \
  --security-opt=label=disable \
  pytorch/pytorch \
  python

but there is still errors trying to pull registry

[sungshic]

It seems you are running the docker command inside TRE? That being the case, using the pytorch/pytorch image would not work as there is no Internet connection from within TRE. Instead of using the pytorch/pytorch image, trying running it with the image that you built based on pytorch/pytorch, saved as a tar file, uploaded to TRE, and loaded to TRE's docker environment.

[mxochicale]

Q1. When dockerimage testing data paths locally I see this error when trying to find a file "No such file or directory: '/root/datasets/chest-xrays-indiana-university/chest-xrays-indiana-university/indiana_projections.csv'", any suggestions how to point to the right paht? We only need "datasets/chest-xrays-indiana-university/chest-xrays-indiana-university/indiana_projections.csv'"?

                )
E               FileNotFoundError: [Errno 2] No such file or directory: '/root/datasets/chest-xrays-indiana-university/chest-xrays-indiana-university/indiana_projections.csv'

/usr/local/lib/python3.12/site-packages/pandas/io/common.py:873: FileNotFoundError
============================================================================================= short test summary info =============================================================================================
FAILED tests/test_ml_pipeline.py::test_CheXNet_CNN_Dataset - FileNotFoundError: [Errno 2] No such file or directory: '/root/datasets/chest-xrays-indiana-university/chest-xrays-indiana-university/indiana_projections.csv'
================================================================================================ 1 failed in 2.99s ================================================================================================
root@4cfdcd069916:/ammir# 

[mxochicale]

Q2. Any idea how to mitigate space left in TREs desk? Apparently the ammir-image.tar [17.3G] is having some issues?

• upload image via airlock then load tre deskop and look for image:
  
• load image docker load -i ~/ammir-image.tar but it seems there is no space left?

[mxochicale]

Re Q2. Suggestions by Sunny:

sudo mkdir /shared/inbound/<ucl-id>/tmp
sudo rm -rf /var/tmp
sudo ln -s /shared/inbound/<ucl-id>/tmp /var/tmp
sudo docker load -i /shared/allreadwrite/basic-tre-docker-image1.tar.gz
sudo docker images

Re: Q1. Sunny suggested setting up Docker Environment Variables
Some references:

• https://phoenixnap.com/kb/docker-environment-variables

[mxochicale]

Thanks @sungshic

docker load seems working in CHME003924254a

but it shows some failures when running it:

which image to choose?