Tensorflow 2.14 on NVIDIA DGX Spark GB10 on CUDA 13 - ARM64

[obriensystems]

michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ vi environments/windows/Dockerfile 
michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ sudo ./build.sh 
[+] Building 0.3s (8/8) FINISHED                                                                                                                                                   docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                         0.0s
 => => transferring dockerfile: 545B                                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/linaro/tensorflow-arm64-build:2.14-multipython                                                                                                    0.1s
 => [internal] load .dockerignore                                                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                                                              0.0s
 => [1/3] FROM docker.io/linaro/tensorflow-arm64-build:2.14-multipython@sha256:6924bf224dec0acf3f8ca29650b8eae45a2cf5fb8fc011880fdfc5b42d371352                                              0.0s
 => => resolve docker.io/linaro/tensorflow-arm64-build:2.14-multipython@sha256:6924bf224dec0acf3f8ca29650b8eae45a2cf5fb8fc011880fdfc5b42d371352                                              0.0s
 => [internal] load build context                                                                                                                                                            0.0s
 => => transferring context: 57B                                                                                                                                                             0.0s
 => CACHED [2/3] WORKDIR /src                                                                                                                                                                0.0s
 => CACHED [3/3] COPY /src/tflow.py .                                                                                                                                                        0.0s
 => exporting to image                                                                                                                                                                       0.0s
 => => exporting layers                                                                                                                                                                      0.0s
 => => writing image sha256:0cc082cb7126f885bdc2c605c287c1a1ac3962b546b3ac38a220ee1f834dfa06                                                                                                 0.0s
 => => naming to docker.io/library/ml-tensorflow-arm                                                                                                                                         0.0s
Traceback (most recent call last):
  File "tflow.py", line 1, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ vi environments/windows/Dockerfile 
michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ python3 -m venv venv-t214
michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ source venv-t214/bin/activate
(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ python --version
Python 3.12.3
(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ python -m pip install --upgrade pip
Requirement already satisfied: pip in ./venv-t214/lib/python3.12/site-packages (24.0)
Collecting pip
  Downloading pip-25.3-py3-none-any.whl.metadata (4.7 kB)
Downloading pip-25.3-py3-none-any.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 12.5 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
Successfully installed pip-25.3
(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ python -m pip install tensorflow==2.14
ERROR: Could not find a version that satisfies the requirement tensorflow==2.14 (from versions: 2.16.0rc0, 2.16.1, 2.16.2, 2.17.0rc0, 2.17.0rc1, 2.17.0, 2.17.1, 2.18.0rc0, 2.18.0rc1, 2.18.0rc2, 2.18.0, 2.18.1, 2.19.0rc0, 2.19.0, 2.19.1, 2.20.0rc0, 2.20.0)
ERROR: No matching distribution found for tensorflow==2.14


check 64 bit
(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ python
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import struct;print(struct.calcsize("P") * 8)
64

same as
https://stackoverflow.com/questions/77297953/tensorflow-2-14-gpu-not-getting-recognized

running cuda 13
(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Aug_20_01:57:39_PM_PDT_2025
Cuda compilation tools, release 13.0, V13.0.88
Build cuda_13.0.r13.0/compiler.36424714_0

nope

pip install tensorflow[and-cuda]==2.14.0

[obriensystems]

Installing latest tensorflow 2.20 instead (and redo tf 2.12 metrics)

(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ pip install tensorflow[and-cuda]==2.14.0
ERROR: Could not find a version that satisfies the requirement tensorflow==2.14.0 (from versions: 2.16.0rc0, 2.16.1, 2.16.2, 2.17.0rc0, 2.17.0rc1, 2.17.0, 2.17.1, 2.18.0rc0, 2.18.0rc1, 2.18.0rc2, 2.18.0, 2.18.1, 2.19.0rc0, 2.19.0, 2.19.1, 2.20.0rc0, 2.20.0)
ERROR: No matching distribution found for tensorflow==2.14.0
(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ python --version
Python 3.12.3
(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ pip install tensorflow[and-cuda]
Collecting tensorflow[and-cuda]
  Downloading tensorflow-2.20.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.5 kB)
Collecting absl-py>=1.0.0 (from tensorflow[and-cuda])
  Downloading absl_py-2.3.1-py3-none-any.whl.metadata (3.3 kB)
Collecting astunparse>=1.6.0 (from tensorflow[and-cuda])
  Downloading astunparse-1.6.3-py2.py3-none-any.whl.metadata (4.4 kB)
Collecting flatbuffers>=24.3.25 (from tensorflow[and-cuda])
  Downloading flatbuffers-25.9.23-py2.py3-none-any.whl.metadata (875 bytes)
Collecting gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 (from tensorflow[and-cuda])
  Downloading gast-0.6.0-py3-none-any.whl.metadata (1.3 kB)
Collecting google_pasta>=0.1.1 (from tensorflow[and-cuda])
  Downloading google_pasta-0.2.0-py3-none-any.whl.metadata (814 bytes)
Collecting libclang>=13.0.0 (from tensorflow[and-cuda])
  Downloading libclang-18.1.1-py2.py3-none-manylinux2014_aarch64.whl.metadata (5.2 kB)
Collecting opt_einsum>=2.3.2 (from tensorflow[and-cuda])
  Downloading opt_einsum-3.4.0-py3-none-any.whl.metadata (6.3 kB)
Collecting packaging (from tensorflow[and-cuda])
  Downloading packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting protobuf>=5.28.0 (from tensorflow[and-cuda])
  Downloading protobuf-6.33.0-cp39-abi3-manylinux2014_aarch64.whl.metadata (593 bytes)
Collecting requests<3,>=2.21.0 (from tensorflow[and-cuda])
  Downloading requests-2.32.5-py3-none-any.whl.metadata (4.9 kB)
Collecting setuptools (from tensorflow[and-cuda])
  Downloading setuptools-80.9.0-py3-none-any.whl.metadata (6.6 kB)
Collecting six>=1.12.0 (from tensorflow[and-cuda])
  Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting termcolor>=1.1.0 (from tensorflow[and-cuda])
  Downloading termcolor-3.2.0-py3-none-any.whl.metadata (6.4 kB)
Collecting typing_extensions>=3.6.6 (from tensorflow[and-cuda])
  Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Collecting wrapt>=1.11.0 (from tensorflow[and-cuda])
  Downloading wrapt-2.0.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (9.0 kB)
Collecting grpcio<2.0,>=1.24.3 (from tensorflow[and-cuda])
  Downloading grpcio-1.76.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (3.7 kB)
Collecting tensorboard~=2.20.0 (from tensorflow[and-cuda])
  Downloading tensorboard-2.20.0-py3-none-any.whl.metadata (1.8 kB)
Collecting keras>=3.10.0 (from tensorflow[and-cuda])
  Downloading keras-3.12.0-py3-none-any.whl.metadata (5.9 kB)
Collecting numpy>=1.26.0 (from tensorflow[and-cuda])
  Downloading numpy-2.3.4-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (62 kB)
Collecting h5py>=3.11.0 (from tensorflow[and-cuda])
  Downloading h5py-3.15.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (3.0 kB)
Collecting ml_dtypes<1.0.0,>=0.5.1 (from tensorflow[and-cuda])
  Downloading ml_dtypes-0.5.3-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (8.9 kB)
Collecting nvidia-cublas-cu12<13.0,>=12.5.3.2 (from tensorflow[and-cuda])
  Downloading nvidia_cublas_cu12-12.9.1.4-py3-none-manylinux_2_27_aarch64.whl.metadata (1.7 kB)
Collecting nvidia-cuda-cupti-cu12<13.0,>=12.5.82 (from tensorflow[and-cuda])
  Downloading nvidia_cuda_cupti_cu12-12.9.79-py3-none-manylinux_2_25_aarch64.whl.metadata (1.8 kB)
Collecting nvidia-cuda-nvcc-cu12<13.0,>=12.5.82 (from tensorflow[and-cuda])
  Downloading nvidia_cuda_nvcc_cu12-12.9.86-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (1.7 kB)
Collecting nvidia-cuda-nvrtc-cu12<13.0,>=12.5.82 (from tensorflow[and-cuda])
  Downloading nvidia_cuda_nvrtc_cu12-12.9.86-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (1.7 kB)
Collecting nvidia-cuda-runtime-cu12<13.0,>=12.5.82 (from tensorflow[and-cuda])
  Downloading nvidia_cuda_runtime_cu12-12.9.79-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (1.7 kB)
Collecting nvidia-cudnn-cu12<10.0,>=9.3.0.75 (from tensorflow[and-cuda])
  Downloading nvidia_cudnn_cu12-9.15.0.57-py3-none-manylinux_2_27_aarch64.whl.metadata (1.8 kB)
Collecting nvidia-cufft-cu12<12.0,>=11.2.3.61 (from tensorflow[and-cuda])
  Downloading nvidia_cufft_cu12-11.4.1.4-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (1.8 kB)
Collecting nvidia-curand-cu12<11.0,>=10.3.6.82 (from tensorflow[and-cuda])
  Downloading nvidia_curand_cu12-10.3.10.19-py3-none-manylinux_2_27_aarch64.whl.metadata (1.7 kB)
Collecting nvidia-cusolver-cu12<12.0,>=11.6.3.83 (from tensorflow[and-cuda])
  Downloading nvidia_cusolver_cu12-11.7.5.82-py3-none-manylinux_2_27_aarch64.whl.metadata (1.9 kB)
Collecting nvidia-cusparse-cu12<13.0,>=12.5.1.3 (from tensorflow[and-cuda])
  Downloading nvidia_cusparse_cu12-12.5.10.65-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (1.8 kB)
Collecting nvidia-nccl-cu12<3.0,>=2.25.1 (from tensorflow[and-cuda])
  Downloading nvidia_nccl_cu12-2.28.7-py3-none-manylinux_2_18_aarch64.whl.metadata (2.0 kB)
Collecting nvidia-nvjitlink-cu12<13.0,>=12.5.82 (from tensorflow[and-cuda])
  Downloading nvidia_nvjitlink_cu12-12.9.86-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (1.7 kB)
Collecting charset_normalizer<4,>=2 (from requests<3,>=2.21.0->tensorflow[and-cuda])
  Downloading charset_normalizer-3.4.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (37 kB)
Collecting idna<4,>=2.5 (from requests<3,>=2.21.0->tensorflow[and-cuda])
  Downloading idna-3.11-py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3<3,>=1.21.1 (from requests<3,>=2.21.0->tensorflow[and-cuda])
  Downloading urllib3-2.5.0-py3-none-any.whl.metadata (6.5 kB)
Collecting certifi>=2017.4.17 (from requests<3,>=2.21.0->tensorflow[and-cuda])
  Downloading certifi-2025.10.5-py3-none-any.whl.metadata (2.5 kB)
Collecting markdown>=2.6.8 (from tensorboard~=2.20.0->tensorflow[and-cuda])
  Downloading markdown-3.10-py3-none-any.whl.metadata (5.1 kB)
Collecting pillow (from tensorboard~=2.20.0->tensorflow[and-cuda])
  Downloading pillow-12.0.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (8.8 kB)
Collecting tensorboard-data-server<0.8.0,>=0.7.0 (from tensorboard~=2.20.0->tensorflow[and-cuda])
  Downloading tensorboard_data_server-0.7.2-py3-none-any.whl.metadata (1.1 kB)
Collecting werkzeug>=1.0.1 (from tensorboard~=2.20.0->tensorflow[and-cuda])
  Downloading werkzeug-3.1.3-py3-none-any.whl.metadata (3.7 kB)
Collecting wheel<1.0,>=0.23.0 (from astunparse>=1.6.0->tensorflow[and-cuda])
  Downloading wheel-0.45.1-py3-none-any.whl.metadata (2.3 kB)
Collecting rich (from keras>=3.10.0->tensorflow[and-cuda])
  Downloading rich-14.2.0-py3-none-any.whl.metadata (18 kB)
Collecting namex (from keras>=3.10.0->tensorflow[and-cuda])
  Downloading namex-0.1.0-py3-none-any.whl.metadata (322 bytes)
Collecting optree (from keras>=3.10.0->tensorflow[and-cuda])
  Downloading optree-0.17.0-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl.metadata (33 kB)
Collecting MarkupSafe>=2.1.1 (from werkzeug>=1.0.1->tensorboard~=2.20.0->tensorflow[and-cuda])
  Downloading markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (2.7 kB)
Collecting markdown-it-py>=2.2.0 (from rich->keras>=3.10.0->tensorflow[and-cuda])
  Downloading markdown_it_py-4.0.0-py3-none-any.whl.metadata (7.3 kB)
Collecting pygments<3.0.0,>=2.13.0 (from rich->keras>=3.10.0->tensorflow[and-cuda])
  Downloading pygments-2.19.2-py3-none-any.whl.metadata (2.5 kB)
Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich->keras>=3.10.0->tensorflow[and-cuda])
  Downloading mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
Downloading tensorflow-2.20.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (259.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 259.5/259.5 MB 32.9 MB/s  0:00:07
Downloading grpcio-1.76.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (6.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.4/6.4 MB 75.5 MB/s  0:00:00
Downloading ml_dtypes-0.5.3-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (5.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.0/5.0 MB 58.7 MB/s  0:00:00
Downloading nvidia_cublas_cu12-12.9.1.4-py3-none-manylinux_2_27_aarch64.whl (575.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 575.0/575.0 MB 30.1 MB/s  0:00:18
Downloading nvidia_cuda_cupti_cu12-12.9.79-py3-none-manylinux_2_25_aarch64.whl (10.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.1/10.1 MB 32.2 MB/s  0:00:00
Downloading nvidia_cuda_nvcc_cu12-12.9.86-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (39.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.4/39.4 MB 23.8 MB/s  0:00:01
Downloading nvidia_cuda_nvrtc_cu12-12.9.86-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (44.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.5/44.5 MB 29.3 MB/s  0:00:01
Downloading nvidia_cuda_runtime_cu12-12.9.79-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 30.4 MB/s  0:00:00
Downloading nvidia_cudnn_cu12-9.15.0.57-py3-none-manylinux_2_27_aarch64.whl (646.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 646.6/646.6 MB 29.9 MB/s  0:00:21
Downloading nvidia_cufft_cu12-11.4.1.4-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (200.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.8/200.8 MB 30.2 MB/s  0:00:06
Downloading nvidia_curand_cu12-10.3.10.19-py3-none-manylinux_2_27_aarch64.whl (68.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.3/68.3 MB 30.5 MB/s  0:00:02
Downloading nvidia_cusolver_cu12-11.7.5.82-py3-none-manylinux_2_27_aarch64.whl (337.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 337.9/337.9 MB 29.0 MB/s  0:00:11
Downloading nvidia_cusparse_cu12-12.5.10.65-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (366.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 366.4/366.4 MB 29.8 MB/s  0:00:12
Downloading nvidia_nccl_cu12-2.28.7-py3-none-manylinux_2_18_aarch64.whl (296.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 296.7/296.7 MB 29.5 MB/s  0:00:10
Downloading nvidia_nvjitlink_cu12-12.9.86-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (39.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.5/39.5 MB 30.4 MB/s  0:00:01
Downloading requests-2.32.5-py3-none-any.whl (64 kB)
Downloading charset_normalizer-3.4.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (148 kB)
Downloading idna-3.11-py3-none-any.whl (71 kB)
Downloading tensorboard-2.20.0-py3-none-any.whl (5.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 101.0 MB/s  0:00:00
Downloading tensorboard_data_server-0.7.2-py3-none-any.whl (2.4 kB)
Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB)
Downloading urllib3-2.5.0-py3-none-any.whl (129 kB)
Downloading absl_py-2.3.1-py3-none-any.whl (135 kB)
Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Downloading wheel-0.45.1-py3-none-any.whl (72 kB)
Downloading certifi-2025.10.5-py3-none-any.whl (163 kB)
Downloading flatbuffers-25.9.23-py2.py3-none-any.whl (30 kB)
Downloading gast-0.6.0-py3-none-any.whl (21 kB)
Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
Downloading h5py-3.15.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (4.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 41.7 MB/s  0:00:00
Downloading keras-3.12.0-py3-none-any.whl (1.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 64.2 MB/s  0:00:00
Downloading libclang-18.1.1-py2.py3-none-manylinux2014_aarch64.whl (23.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.8/23.8 MB 36.1 MB/s  0:00:00
Downloading markdown-3.10-py3-none-any.whl (107 kB)
Downloading numpy-2.3.4-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (14.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.3/14.3 MB 90.6 MB/s  0:00:00
Downloading opt_einsum-3.4.0-py3-none-any.whl (71 kB)
Downloading protobuf-6.33.0-cp39-abi3-manylinux2014_aarch64.whl (324 kB)
Downloading setuptools-80.9.0-py3-none-any.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 60.3 MB/s  0:00:00
Downloading termcolor-3.2.0-py3-none-any.whl (7.7 kB)
Downloading werkzeug-3.1.3-py3-none-any.whl (224 kB)
Downloading markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (24 kB)
Downloading wrapt-2.0.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (123 kB)
Downloading namex-0.1.0-py3-none-any.whl (5.9 kB)
Downloading optree-0.17.0-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (365 kB)
Downloading packaging-25.0-py3-none-any.whl (66 kB)
Downloading pillow-12.0.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (6.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 109.5 MB/s  0:00:00
Downloading rich-14.2.0-py3-none-any.whl (243 kB)
Downloading pygments-2.19.2-py3-none-any.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 61.9 MB/s  0:00:00
Downloading markdown_it_py-4.0.0-py3-none-any.whl (87 kB)
Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Installing collected packages: namex, libclang, flatbuffers, wrapt, wheel, urllib3, typing_extensions, termcolor, tensorboard-data-server, six, setuptools, pygments, protobuf, pillow, packaging, opt_einsum, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-nvcc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, mdurl, MarkupSafe, markdown, idna, gast, charset_normalizer, certifi, absl-py, werkzeug, requests, optree, nvidia-cusparse-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, ml_dtypes, markdown-it-py, h5py, grpcio, google_pasta, astunparse, tensorboard, rich, nvidia-cusolver-cu12, keras, tensorflow
Successfully installed MarkupSafe-3.0.3 absl-py-2.3.1 astunparse-1.6.3 certifi-2025.10.5 charset_normalizer-3.4.4 flatbuffers-25.9.23 gast-0.6.0 google_pasta-0.2.0 grpcio-1.76.0 h5py-3.15.1 idna-3.11 keras-3.12.0 libclang-18.1.1 markdown-3.10 markdown-it-py-4.0.0 mdurl-0.1.2 ml_dtypes-0.5.3 namex-0.1.0 numpy-2.3.4 nvidia-cublas-cu12-12.9.1.4 nvidia-cuda-cupti-cu12-12.9.79 nvidia-cuda-nvcc-cu12-12.9.86 nvidia-cuda-nvrtc-cu12-12.9.86 nvidia-cuda-runtime-cu12-12.9.79 nvidia-cudnn-cu12-9.15.0.57 nvidia-cufft-cu12-11.4.1.4 nvidia-curand-cu12-10.3.10.19 nvidia-cusolver-cu12-11.7.5.82 nvidia-cusparse-cu12-12.5.10.65 nvidia-nccl-cu12-2.28.7 nvidia-nvjitlink-cu12-12.9.86 opt_einsum-3.4.0 optree-0.17.0 packaging-25.0 pillow-12.0.0 protobuf-6.33.0 pygments-2.19.2 requests-2.32.5 rich-14.2.0 setuptools-80.9.0 six-1.17.0 tensorboard-2.20.0 tensorboard-data-server-0.7.2 tensorflow-2.20.0 termcolor-3.2.0 typing_extensions-4.15.0 urllib3-2.5.0 werkzeug-3.1.3 wheel-0.45.1 wrapt-2.0.1

(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ python -c 'import tensorflow as tf; print(tf.__version__)'
2.20.0

[obriensystems]

TF is not picking up the GPU
check
https://forums.developer.nvidia.com/t/unable-to-work-with-tensorflow-in-docker-on-dgx-spark/350419

(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ cat tensorflow-version.py 
import tensorflow as tf
print(tf.__version__)
print("GPUs available:", tf.config.list_physical_devices('GPU'))

(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning$ python tensorflow-version.py 
2.20.0
GPUs available: []


(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning/environments/windows/src$ python tflow.py 
Downloading data from https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz
169001437/169001437 ━━━━━━━━━━━━━━━━━━━━ 4s 0us/step  
Batch: 2048 Epochs: 25
2025-11-09 18:03:21.971966: W tensorflow/core/framework/dataset.cc:993] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
Epoch 1/25
14/25 ━━━━━━━━━━━━━━━━━━━━ 1:45 10s/step - accuracy: 0.0174 - loss: 5.9732

install transformers, torch and accelerate

(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning/environments/windows/src$ pip3 install -U transformers
Collecting transformers
  Downloading transformers-4.57.1-py3-none-any.whl.metadata (43 kB)
Collecting filelock (from transformers)
  Downloading filelock-3.20.0-py3-none-any.whl.metadata (2.1 kB)
Collecting huggingface-hub<1.0,>=0.34.0 (from transformers)
  Downloading huggingface_hub-0.36.0-py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: numpy>=1.17 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from transformers) (2.3.4)
Requirement already satisfied: packaging>=20.0 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from transformers) (25.0)
Collecting pyyaml>=5.1 (from transformers)
  Downloading pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (2.4 kB)
Collecting regex!=2019.12.17 (from transformers)
  Downloading regex-2025.11.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (40 kB)
Requirement already satisfied: requests in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from transformers) (2.32.5)
Collecting tokenizers<=0.23.0,>=0.22.0 (from transformers)
  Downloading tokenizers-0.22.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (6.8 kB)
Collecting safetensors>=0.4.3 (from transformers)
  Downloading safetensors-0.6.2-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.1 kB)
Collecting tqdm>=4.27 (from transformers)
  Downloading tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
Collecting fsspec>=2023.5.0 (from huggingface-hub<1.0,>=0.34.0->transformers)
  Downloading fsspec-2025.10.0-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (4.15.0)
Collecting hf-xet<2.0.0,>=1.1.3 (from huggingface-hub<1.0,>=0.34.0->transformers)
  Downloading hf_xet-1.2.0-cp37-abi3-manylinux_2_28_aarch64.whl.metadata (4.9 kB)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from requests->transformers) (3.4.4)
Requirement already satisfied: idna<4,>=2.5 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from requests->transformers) (3.11)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from requests->transformers) (2.5.0)
Requirement already satisfied: certifi>=2017.4.17 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from requests->transformers) (2025.10.5)
Downloading transformers-4.57.1-py3-none-any.whl (12.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 75.8 MB/s  0:00:00
Downloading huggingface_hub-0.36.0-py3-none-any.whl (566 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 566.1/566.1 kB 21.7 MB/s  0:00:00
Downloading hf_xet-1.2.0-cp37-abi3-manylinux_2_28_aarch64.whl (3.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 85.4 MB/s  0:00:00
Downloading tokenizers-0.22.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 19.1 MB/s  0:00:00
Downloading fsspec-2025.10.0-py3-none-any.whl (200 kB)
Downloading pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (775 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 775.1/775.1 kB 34.8 MB/s  0:00:00
Downloading regex-2025.11.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (798 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 798.6/798.6 kB 35.9 MB/s  0:00:00
Downloading safetensors-0.6.2-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (473 kB)
Downloading tqdm-4.67.1-py3-none-any.whl (78 kB)
Downloading filelock-3.20.0-py3-none-any.whl (16 kB)
Installing collected packages: tqdm, safetensors, regex, pyyaml, hf-xet, fsspec, filelock, huggingface-hub, tokenizers, transformers
Successfully installed filelock-3.20.0 fsspec-2025.10.0 hf-xet-1.2.0 huggingface-hub-0.36.0 pyyaml-6.0.3 regex-2025.11.3 safetensors-0.6.2 tokenizers-0.22.1 tqdm-4.67.1 transformers-4.57.1
(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning/environments/windows/src$ pip3 install -U torch
Collecting torch
  Downloading torch-2.9.0-cp312-cp312-manylinux_2_28_aarch64.whl.metadata (30 kB)
Requirement already satisfied: filelock in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from torch) (3.20.0)
Requirement already satisfied: typing-extensions>=4.10.0 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from torch) (4.15.0)
Requirement already satisfied: setuptools in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from torch) (80.9.0)
Collecting sympy>=1.13.3 (from torch)
  Downloading sympy-1.14.0-py3-none-any.whl.metadata (12 kB)
Collecting networkx>=2.5.1 (from torch)
  Downloading networkx-3.5-py3-none-any.whl.metadata (6.3 kB)
Collecting jinja2 (from torch)
  Downloading jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Requirement already satisfied: fsspec>=0.8.5 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from torch) (2025.10.0)
Collecting mpmath<1.4,>=1.1.0 (from sympy>=1.13.3->torch)
  Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from jinja2->torch) (3.0.3)
Downloading torch-2.9.0-cp312-cp312-manylinux_2_28_aarch64.whl (104.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.1/104.1 MB 89.4 MB/s  0:00:01
Downloading networkx-3.5-py3-none-any.whl (2.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 67.1 MB/s  0:00:00
Downloading sympy-1.14.0-py3-none-any.whl (6.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 100.3 MB/s  0:00:00
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 19.0 MB/s  0:00:00
Downloading jinja2-3.1.6-py3-none-any.whl (134 kB)
Installing collected packages: mpmath, sympy, networkx, jinja2, torch
Successfully installed jinja2-3.1.6 mpmath-1.3.0 networkx-3.5 sympy-1.14.0 torch-2.9.0
(venv-t214) michael@spark-7d19:~/wse_github/ObrienlabsDev/machine-learning/environments/windows/src$ pip3 install -U accelerate
Collecting accelerate
  Downloading accelerate-1.11.0-py3-none-any.whl.metadata (19 kB)
Requirement already satisfied: numpy>=1.17 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from accelerate) (2.3.4)
Requirement already satisfied: packaging>=20.0 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from accelerate) (25.0)
Collecting psutil (from accelerate)
  Downloading psutil-7.1.3-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (23 kB)
Requirement already satisfied: pyyaml in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from accelerate) (6.0.3)
Requirement already satisfied: torch>=2.0.0 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from accelerate) (2.9.0)
Requirement already satisfied: huggingface_hub>=0.21.0 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from accelerate) (0.36.0)
Requirement already satisfied: safetensors>=0.4.3 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from accelerate) (0.6.2)
Requirement already satisfied: filelock in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (3.20.0)
Requirement already satisfied: fsspec>=2023.5.0 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (2025.10.0)
Requirement already satisfied: requests in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (2.32.5)
Requirement already satisfied: tqdm>=4.42.1 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (4.67.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (4.15.0)
Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (1.2.0)
Requirement already satisfied: setuptools in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (80.9.0)
Requirement already satisfied: sympy>=1.13.3 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (1.14.0)
Requirement already satisfied: networkx>=2.5.1 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (3.5)
Requirement already satisfied: jinja2 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (3.1.6)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=2.0.0->accelerate) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from jinja2->torch>=2.0.0->accelerate) (3.0.3)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from requests->huggingface_hub>=0.21.0->accelerate) (3.4.4)
Requirement already satisfied: idna<4,>=2.5 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from requests->huggingface_hub>=0.21.0->accelerate) (3.11)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from requests->huggingface_hub>=0.21.0->accelerate) (2.5.0)
Requirement already satisfied: certifi>=2017.4.17 in /home/michael/wse_github/ObrienlabsDev/machine-learning/venv-t214/lib/python3.12/site-packages (from requests->huggingface_hub>=0.21.0->accelerate) (2025.10.5)
Downloading accelerate-1.11.0-py3-none-any.whl (375 kB)
Downloading psutil-7.1.3-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (264 kB)
Installing collected packages: psutil, accelerate
Successfully installed accelerate-1.11.0 psutil-7.1.3

[obriensystems]

pytorch via docker is ok
#49