[Feature] Support for arm64 for CUDA (immich-machine-learning)

[gowthamravichandran]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Docker Image for immich-machine-learning-cuda does not support arm64 processors at the moment.

Get the following message when trying to pull ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda

no matching manifest for linux/arm64/v8 in the manifest list entries

Running immich in arm64 processor with cuda enabled gpus like Nvidia Jetson devices isnt possible and is currently forced to use the cpu instead of gpu.

Can we support a docker image for cuda enabled arm64 processors?

Platform

• Server
• Web
• Mobile

[mertalev]

Good point! We can add arm64 as a target for the CUDA image.

[gowthamravichandran]

I use Jetson Orin

CPU(s): 12 (aarch64)
RAM: 64GB
Kernel: 5.10.104-tegra
Jetpack: 5.1

[mattias-pp]

Hello @mertalev , wondering if you made any progress on this? Thanks!

[mertalev]

Sorry, I haven't had the time. The PR is #12456 if anyone wants to rebase and test it.

[gowthamravichandran]

Thanks @mertalev for the effort, I got some time to test it out, the image builds successfully and the container runs. but dont see GPU usage at all. Attached a screenshot of CPU usage being high while processing uploaded images, but GPU being idle.

Not sure if I am missing something.

[ghost]

@mertalev you are the hero we need!!!! we are counting on you!!!

[bogdanr]

I would also like to use this on Jetson. I have the newer one (Jetson Orin Nano). 5.15.148-tegra kernel with Jetpack 6.1.

[mpalitto]

I have too the old (but cheapest) Jetson Nano 4Gb.
Distro: Ubuntu 18.04 bionic
CPUs: 4 (aarch64)
RAM: 4.0GB
Kernel: 4.9.337-tegra
Jetpack: 4.6.5 (L4T 32.7.5)

[afsy-d]

Also looking forward to this being an option!

Jetson Orin NX
Distro Ubuntu 20.04
CPUs: 8
RAM: 16GB
Kernel: 5.10.192-tegra
Jetpack: Jetpack 5.1.3

[mattias-pp]

Any update on this?

Jetson Xavier AGX
Linux version 5.10.216-tegra

[pwentrys]

Would also love this option.

Jetson Orin Nano 8GB here.

[loaonline]

Support for this is highly desired.

Jetson Orin Nano 8GB

[phamchin]

I also looking forward to this being an option.
Jetson Nano B01 4GB

[torbenAndersen]

I have Immich running on a Jetson nano. It works very well, even without CUDA support, and responds quickly. I had expected the micro SD card to slow it down but it does not appear to be a problem.

[ghost]

Also waiting for the ghcr.io/immich-app/immich-machine-learning:release-cuda image build for ARM64. i am running the AI features on the jetson nano 8gb, and the rest on the pi5

[minamion]

For NVIDIA Jetson series devices, the standard nvidia/cuda images cannot be used - they require the nvidia/l4t-base as the base image, with versions may tied to specific JetPack SDK releases. I've locally modified the Dockerfile (using l4t-base:r34.1) and successfully tested it on Jetson Xavier NX. However, compatibility with other Jetson products remains to be verified. Importantly, Jetson devices shouldn't simply use the arm64 tag as they have a unique architecture and lack PCIe GPUs, and nvidia-smi is unavailable .

but looks like Jetson is the only one arm+cuda devices

[minamion]

https://elinux.org/Jetson_Zoo#ONNX_Runtime Looks like there is a useful document here

[Vascolas007]

During ML tasks, I can see some spikes with a short duration, in a jtop GPU window. Also the GPU RAM usage is increased when running the container. Is that an indication that it is working?

[minamion]

During ML tasks, I can see some spikes with a short duration, in a jtop GPU window. Also the GPU RAM usage is increased when running the container. Is that an indication that it is working?

If you see "Setting execution providers to ['CUDAExecutionProvider', 'CPUExecutionProvider']" in the logs, it looks like it ran successfully.

[Vascolas007]

From Jetson zoo, I see that for jetpack 4 the most recent python version is 3.9.
When building from an image from a Dockerfile:

Using Python 3.8.0 environment at: /usr × No solution found when resolving dependencies: ╰─▶ Because the current Python version (3.8) does not satisfy Python>=3.10,<4.0 and immich-ml==1.129.0 depends on Python>=3.10,<4.0, we can conclude that immich-ml==1.129.0 cannot be used. And because only immich-ml==1.129.0 is available and you require immich-ml, we can conclude that your requirements are unsatisfiable.

[minamion]

From Jetson zoo, I see that for jetpack 4 the most recent python version is 3.9. When building from an image from a Dockerfile:

Using Python 3.8.0 environment at: /usr × No solution found when resolving dependencies: ╰─▶ Because the current Python version (3.8) does not satisfy Python>=3.10,<4.0 and immich-ml==1.129.0 depends on Python>=3.10,<4.0, we can conclude that immich-ml==1.129.0 cannot be used. And because only immich-ml==1.129.0 is available and you require immich-ml, we can conclude that your requirements are unsatisfiable.

Yes, I also found this problem. I noticed that there is no Ubuntu 22.04 and python 3.1x until jetpack 6. However, I found that https://github.com/dusty-nv/jetson-containers seems to be able to build an image that meets the requirements and is supported by a lower version of jetpack. I am trying

[minamion]

Yes i made it work!

What I did:

• Rewrote the dockerfile
• Removed -cuda = [ { name = "onnxruntime-gpu" }, ] and
  { name = "onnxruntime-gpu", marker = "extra == 'cuda'", specifier = ">=1.17.0,<2", index = "https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/" },
  from uv.lock, because here we need to manually download from nvidia

Dockerfile:

ARG DEVICE=cuda

# For JetPack 5.2.x
FROM nvcr.io/nvidia/l4t-base:35.4.1 AS builder

ARG DEVICE
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    VIRTUAL_ENV=/opt/venv \
    PATH="/opt/venv/bin:$PATH"

# Install Python 3.11 & g++ from unofficial ppa
RUN apt-get update && \
    apt install software-properties-common -y && \
    add-apt-repository ppa:deadsnakes/ppa -y && \
    add-apt-repository -y ppa:ubuntu-toolchain-r/test -y && \
    apt-get install -y --no-install-recommends \
    build-essential \
    wget \
    g++-11 \
    python3.11 \
    python3.11-dev \
    python3.11-venv \
    libcudnn8-dev && \
    rm -rf /var/lib/apt/lists/*

# Set Python 3.11 as default
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1

# create vdev and install pip
RUN python3.11 -m venv /opt/venv && \
    /opt/venv/bin/pip install --upgrade pip 

# install ONNX Runtime GPU for Jetson
RUN wget -q -O onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl \
    https://nvidia.box.com/shared/static/n6wf6n6vwydgts0ivf7h2479vhghxt02.whl && \
    /opt/venv/bin/pip install onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl 
    #&& \
    #rm onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl

# uv
COPY --from=ghcr.io/astral-sh/uv:latest@sha256:4faec156e35a5f345d57804d8858c6ba1cf6352ce5f4bffc11b7fdebdef46a38 /uv /uvx /bin/
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --extra ${DEVICE} --no-dev --no-editable --no-install-project --compile-bytecode --no-progress --active --link-mode copy

# runtime
FROM nvcr.io/nvidia/l4t-cuda:11.4.19-runtime

# install again ()
RUN apt-get update && \
    apt install software-properties-common -y && \
    add-apt-repository ppa:deadsnakes/ppa -y && \
    add-apt-repository -y ppa:ubuntu-toolchain-r/test -y && \
    apt-get install -y --no-install-recommends \
    python3.11 \
    g++-11 \
    libcudnn8-dev \
    tini && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# set cuda 11
ENV LD_LIBRARY_PATH=/usr/local/cuda-11/compat:$LD_LIBRARY_PATH \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PATH="/opt/venv/bin:$PATH" \
    PYTHONPATH=/usr/src \
    DEVICE=${DEVICE} \
    VIRTUAL_ENV=/opt/venv \
    MACHINE_LEARNING_CACHE_FOLDER=/cache


COPY --from=builder /opt/venv /opt/venv
# There may be redundancy but the build is not passed in
RUN wget -q -O onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl \
    https://nvidia.box.com/shared/static/n6wf6n6vwydgts0ivf7h2479vhghxt02.whl && \
    /opt/venv/bin/pip install onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl 
 
RUN echo "hard core 0" >> /etc/security/limits.conf && \
    echo "fs.suid_dumpable 0" >> /etc/sysctl.conf && \
    echo 'ulimit -S -c 0 > /dev/null 2>&1' >> /etc/profile

WORKDIR /usr/src
COPY scripts/healthcheck.py .
COPY immich_ml immich_ml


ARG BUILD_ID
ARG BUILD_IMAGE
ARG BUILD_SOURCE_REF
ARG BUILD_SOURCE_COMMIT

ENV IMMICH_BUILD=${BUILD_ID}
ENV IMMICH_BUILD_URL=https://github.com/immich-app/immich/actions/runs/${BUILD_ID}
ENV IMMICH_BUILD_IMAGE=${BUILD_IMAGE}
ENV IMMICH_BUILD_IMAGE_URL=https://github.com/immich-app/immich/pkgs/container/immich-machine-learning
ENV IMMICH_REPOSITORY=immich-app/immich
ENV IMMICH_REPOSITORY_URL=https://github.com/immich-app/immich
ENV IMMICH_SOURCE_REF=${BUILD_SOURCE_REF}
ENV IMMICH_SOURCE_COMMIT=${BUILD_SOURCE_COMMIT}
ENV IMMICH_SOURCE_URL=https://github.com/immich-app/immich/commit/${BUILD_SOURCE_COMMIT}

ENTRYPOINT ["tini", "--"]
CMD ["python", "-m", "immich_ml"]

HEALTHCHECK CMD python3 healthcheck.py

Some notes:

• According to the nvidia manual, l4t-cuda does not specify whether it can be higher than the host version, because it does not mark the jetpack version at all. I did not try this because it worked and I did not want to mess with it.
• onnxruntime-gpu comes from https://elinux.org/Jetson_Zoo#ONNX_Runtime, which seems to be strictly related to the jetpack version, but considering that Nano/TX1/TX2 can only be upgraded to jetpack4.6.1, and the highest version that supports 4.6.1 is onnxruntime 1.11.0, and the minimum project requirement is 1.17, I think you may need to do something yourself
  If you have Xavier or even Orin like me, maybe it can run according to my operation

It works, but I don’t think my changes are enough to submit a PR. It would be great if someone can improve it and merge it into the mainline

[torwag]

Could you tell us which jetpack version you used?
Also some benchmarking would be interesting. I will give it a try on an Orin 16GB.

[minamion]

Could you tell us which jetpack version you used? Also some benchmarking would be interesting. I will give it a try on an Orin 16GB.

hi,it is in the jtop interface in the first picture:)
5.1.4 (the last version on Xavier NX)

If you want to use it on Orin, I guess its version is jetpack 6, which may require a lot of modifications. I'm not sure if it is backward compatible. l4t-base:36.x should have built-in a lot of the components I added manually.

[minamion]

Could you tell us which jetpack version you used? Also some benchmarking would be interesting. I will give it a try on an Orin 16GB.

If you know Dockerfile, you should only need to remove the part of adding python and setting cuda11 (I heard that L4T 36.x comes with python3.11 and cuda12, otherwise you still need to add them manually), and modify the download address and file name of onnxruntime_gpu (which can be found at https://elinux.org/Jetson_Zoo#ONNX_Runtime),

[afsy-d]

EDIT! Please ignore what I wrote below, this project was confused with frigate, apologies.

Hey All, just want to point out that in the 1.6 beta versions, JP6 is available. Note that JP5 and JP4 will no longer be supported as they will not run on the latest version of Debian on which the docker image is based.

https://github.com/blakeblackshear/frigate/releases

[minamion]

The frigate container depends on ffmpeg and onnx, immich-machine-learning only needs onnx,ffmpeg is in immich-server, so this problem may not be related to this issue,
and I don't think JP4 and JP5 are not supported by frigate because of the Debian version, it may be because l4t-ffmpeg and other things are not updated by nvidia

[afsy-d]

You are correct; thanks for pointing this out.

[jamesagarside]

I've added the suggested changes above by @minamion to a fork incase anyone else wants to test this.
Im testing on Jetson Nano

https://github.com/jamesagarside/immich/tree/jetson-ml-image

[CappyT]

I've tested both @jamesagarside and @minamion solutions on my AGX Orin (64GB Version) and sadly, it doesn't work.

I get the following log:

immich_machine_learning  | [08/12/25 09:33:41] INFO     Starting gunicorn 23.0.0                           
immich_machine_learning  | [08/12/25 09:33:41] INFO     Listening at: http://[::]:3003 (21)                
immich_machine_learning  | [08/12/25 09:33:41] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
immich_machine_learning  | [08/12/25 09:33:41] INFO     Booting worker with pid: 22                        
immich_machine_learning  | [08/12/25 09:33:43] INFO     Started server process [22]                        
immich_machine_learning  | [08/12/25 09:33:43] INFO     Waiting for application startup.                   
immich_machine_learning  | [08/12/25 09:33:44] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [08/12/25 09:33:44] INFO     Initialized request thread pool with 12 threads.   
immich_machine_learning  | [08/12/25 09:33:44] INFO     Application startup complete.                      
immich_machine_learning  | [08/12/25 09:34:05] INFO     Loading visual model 'ViT-B-32__openai' to memory  
immich_machine_learning  | [08/12/25 09:34:05] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
immich_machine_learning  |                              in descending order of preference                  
immich_machine_learning  | terminate called after throwing an instance of 'onnxruntime::OnnxRuntimeException'
immich_machine_learning  |   what():  /home/yifanl/onnxruntime/onnxruntime/contrib_ops/cuda/bert/tensorrt_fused_multihead_attention/cudaDriverWrapper.cc:42 onnxruntime::contrib::cuda::CUDADriverWrapper::CUDADriverWrapper() handle != nullptr was false. 
immich_machine_learning  | 
immich_machine_learning  | [08/12/25 09:34:06] ERROR    Worker (pid:22) was sent code 134!                 
immich_machine_learning  | [08/12/25 09:34:06] INFO     Booting worker with pid: 64                        
immich_machine_learning  | [08/12/25 09:34:08] INFO     Started server process [64]                        
immich_machine_learning  | [08/12/25 09:34:08] INFO     Waiting for application startup.                   
immich_machine_learning  | [08/12/25 09:34:08] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [08/12/25 09:34:08] INFO     Initialized request thread pool with 12 threads.   
immich_machine_learning  | [08/12/25 09:34:08] INFO     Application startup complete. 

If anyone manages to push a working image, i would gladly try that and help debugging.

[minamion]

Sorry I don't have any Orin series device to test, could you upload your Dockerfile, maybe I can help (it seems that our Dockerfile is for earlier nano and nx devices)

[caldsoft]

I was seeing the same on the Jetson Orin Nano Super, updating the Dockerfile above from @minamion to use nvcr.io/nvidia/l4t-base:r36.2.0 with onnx runtime https://nvidia.box.com/shared/static/fy55jvniujjbigr4gwkv8z1ma6ipgspg.whl gets a working container with Jetpack 6.2, CUDA 12.2, cuDNN 8.9.4, onnxruntime 1.18.0, python 3.11

ARG DEVICE=cuda

# For JetPack 6.2.x
FROM nvcr.io/nvidia/l4t-base:r36.2.0 AS builder

ARG DEVICE
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    VIRTUAL_ENV=/opt/venv \
    PATH="/opt/venv/bin:$PATH" \
    DEBIAN_FRONTEND=noninteractive

# Install Python 3.11
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    build-essential \
    wget \
    g++-11 \
    python3.11 \
    python3.11-dev \
    python3.11-venv \
    libcudnn8-dev && \
    rm -rf /var/lib/apt/lists/*

# Set Python 3.11 as default
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1

# create venv and install pip
RUN python3.11 -m venv /opt/venv && \
    /opt/venv/bin/pip install --upgrade pip

# install ONNX Runtime GPU for Jetson
RUN wget -q -O onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl \
    https://nvidia.box.com/shared/static/fy55jvniujjbigr4gwkv8z1ma6ipgspg.whl && \
    /opt/venv/bin/pip install onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl && \
    rm onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl

# uv
COPY --from=ghcr.io/astral-sh/uv:latest@sha256:4faec156e35a5f345d57804d8858c6ba1cf6352ce5f4bffc11b7fdebdef46a38 /uv /uvx /bin/
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --extra ${DEVICE} --no-dev --no-editable --no-install-project --compile-bytecode --no-progress --active --link-mode copy

# runtime
FROM nvcr.io/nvidia/l4t-base:r36.2.0

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PATH="/opt/venv/bin:$PATH" \
    PYTHONPATH=/usr/src \
    DEVICE=${DEVICE} \
    VIRTUAL_ENV=/opt/venv \
    MACHINE_LEARNING_CACHE_FOLDER=/cache \
    DEBIAN_FRONTEND=noninteractive

# install again
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    python3.11 \
    g++-11 \
    libcudnn8-dev \
    tini

RUN apt-get install -y \
    cuda-toolkit-12-2 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# set cuda 12
ENV LD_LIBRARY_PATH=/usr/local/cuda-12/compat:$LD_LIBRARY_PATH

COPY --from=builder /opt/venv /opt/venv

# There may be redundancy but the build is not passed in
RUN wget -q -O onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl \
    https://nvidia.box.com/shared/static/fy55jvniujjbigr4gwkv8z1ma6ipgspg.whl && \
    /opt/venv/bin/pip install onnxruntime_gpu-1.18.0-cp311-cp311-linux_aarch64.whl

RUN echo "hard core 0" >> /etc/security/limits.conf && \
    echo "fs.suid_dumpable 0" >> /etc/sysctl.conf && \
    echo 'ulimit -S -c 0 > /dev/null 2>&1' >> /etc/profile

WORKDIR /usr/src
COPY scripts/healthcheck.py .
COPY immich_ml immich_ml

ARG BUILD_ID
ARG BUILD_IMAGE
ARG BUILD_SOURCE_REF
ARG BUILD_SOURCE_COMMIT

ENV IMMICH_BUILD=${BUILD_ID}
ENV IMMICH_BUILD_URL=https://github.com/immich-app/immich/actions/runs/${BUILD_ID}
ENV IMMICH_BUILD_IMAGE=${BUILD_IMAGE}
ENV IMMICH_BUILD_IMAGE_URL=https://github.com/immich-app/immich/pkgs/container/immich-machine-learning
ENV IMMICH_REPOSITORY=immich-app/immich
ENV IMMICH_REPOSITORY_URL=https://github.com/immich-app/immich
ENV IMMICH_SOURCE_REF=${BUILD_SOURCE_REF}
ENV IMMICH_SOURCE_COMMIT=${BUILD_SOURCE_COMMIT}
ENV IMMICH_SOURCE_URL=https://github.com/immich-app/immich/commit/${BUILD_SOURCE_COMMIT}

ENTRYPOINT ["tini", "--"]
CMD ["python", "-m", "immich_ml"]

HEALTHCHECK CMD python3 healthcheck.py