Error due to apt repository release info changes

[leonce-m]

Steps to reproduce the bug

1. Check out 2338f7e (latest update of 3.2)
2. Execute run_dev.sh with a custom image layer as described in the docs for 3.2 that contains apt-get update.
3. The script automatically finds the pre-built base images nvcr.io/nvidia/isaac/ros:x86_64-ros2_humble_f8edad8d48e9aa1b5b240c2374c55d60 and nvcr.io/nvidia/isaac/ros:aarch64-ros2_humble_adc428c7077de4984a00b63c55903b0a.

Expected Behavior

The custom image layer builds as expected.

Actual Behavior

The image fails to build with the error message:

[+] Building 11.5s (8/21)                                                                                                                                                                                                                      docker:default
 => [internal] load build definition from Dockerfile.srd_acu                                                                                                                                                                                             0.0s
 => => transferring dockerfile: 4.01kB                                                                                                                                                                                                                   0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)                                                                                                                              0.0s
 => [internal] load metadata for nvcr.io/nvidia/isaac/ros:x86_64-ros2_humble_f8edad8d48e9aa1b5b240c2374c55d60                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                                          0.0s
 => [stage-0  1/17] FROM nvcr.io/nvidia/isaac/ros:x86_64-ros2_humble_f8edad8d48e9aa1b5b240c2374c55d60                                                                                                                                                    0.0s
 => [internal] load build context                                                                                                                                                                                                                        0.1s
 => => transferring context: 2.23kB                                                                                                                                                                                                                      0.0s
 => CACHED [stage-0  2/17] RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null |     gpg --dearmor - | sudo tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null                                                 0.0s
 => [stage-0  3/17] RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked     --mount=target=/var/cache/apt,type=cache,sharing=locked     rm -f /etc/apt/apt.conf.d/docker-clean &&     apt-get update                                         6.6s
 => ERROR [stage-0  4/17] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get install -y     ros-humble-can-msgs     ros-humble-image-pipeline     ros-humble-joint-state-publisher     ros-humble-joint-state-publisher-gui      4.8s 
------                                                                                                                                                                                                                                                        
 > [stage-0  4/17] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get install -y     ros-humble-can-msgs     ros-humble-image-pipeline     ros-humble-joint-state-publisher     ros-humble-joint-state-publisher-gui     ros-humble-magic-enum     ros-humble-rmw-cyclonedds-cpp     ros-humble-robot-localization     ros-humble-ros2-control     ros-humble-ros2-controllers     ros-humble-ros2-socketcan     ros-humble-rosbag2-storage-mcap     ros-humble-rqt-common-plugins     ros-humble-rqt-image-overlay     ros-humble-teleop-twist-joy     ros-humble-teleop-twist-keyboard     ros-humble-twist-stamper     ros-humble-xacro:                                                                                                                       
0.274 Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease [1,581 B]                                                                                                                                                   
0.281 Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]                                                                                                                                                                               
0.281 Get:3 https://isaac.download.nvidia.com/isaac-ros/release-3 jammy InRelease [7,900 B]
0.307 Hit:5 https://dl.yarnpkg.com/debian stable InRelease
0.323 Hit:6 https://deb.nodesource.com/node_18.x nodistro InRelease
0.390 Hit:7 https://ppa.launchpadcontent.net/mosquitto-dev/mosquitto-ppa/ubuntu jammy InRelease
0.396 Hit:8 http://archive.ubuntu.com/ubuntu jammy InRelease
0.404 Get:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages [2,143 kB]
0.453 Get:10 http://packages.ros.org/ros2/ubuntu jammy InRelease [4,682 B]
0.476 Get:11 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
0.676 Hit:4 https://repo.download.nvidia.com/jetson/x86_64/jammy r36.4 InRelease
0.842 Get:12 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
0.961 Get:13 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [60.9 kB]
0.985 Get:14 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [3,530 kB]
1.086 Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1,289 kB]
1.125 Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5,982 kB]
1.379 Get:17 http://packages.ros.org/ros2/ubuntu jammy/main amd64 Packages [1,729 kB]
1.650 Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,595 kB]
2.140 Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3,861 kB]
2.581 Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [69.2 kB]
2.586 Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [6,181 kB]
3.334 Reading package lists...
4.551 W: https://isaac.download.nvidia.com/isaac-ros/release-3/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
4.551 E: Repository 'https://isaac.download.nvidia.com/isaac-ros/release-3 jammy InRelease' changed its 'Origin' value from 'Artifactory' to ''
4.551 E: Repository 'https://isaac.download.nvidia.com/isaac-ros/release-3 jammy InRelease' changed its 'Label' value from 'Artifactory' to ''
4.551 W: https://dl.yarnpkg.com/debian/dists/stable/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
4.551 W: http://repo.download.nvidia.com/jetson/x86_64/jammy/dists/r36.4/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
------

System info

x86 Host PC

Operating System: Ubuntu 22.04.5 LTS
Kernel: Linux 6.8.0-86-generic
CPU: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
GPU: GP102 [GeForce GTX 1080 Ti]

NVIDIA Jetson AGX Orin Developer Kit

Operating System: Ubuntu 22.04.5 LTS
Kernel: Linux jetson 5.15.148-tegra

[leonce-m]

A temporary fix was to put --allow-releaseinfo-change behind every apt-get update in the custom image layer, but this is not good practice as it can hide bugs related to bad apt metadata or corrupted lists.

[leonce-m]

I came up with an alternative fix by adding

# clean stale apt metadata and remove old repo entries
RUN rm -rf /var/lib/apt/lists/* && \
    sed -i '/isaac-ros\/release-3/d' /etc/apt/sources.list

# install isaac ros repo key using modern method
RUN --mount=type=cache,target=/var/cache/apt \
    apt-get update && apt-get install -y wget gnupg && \
    wget -qO - https://isaac.download.nvidia.com/isaac-ros/repos.key \
        | gpg --dearmor -o /usr/share/keyrings/isaac-ros.gpg && \
    echo "deb [signed-by=/usr/share/keyrings/isaac-ros.gpg] \
        https://isaac.download.nvidia.com/isaac-ros/release-3 \
        $(. /etc/os-release && echo $VERSION_CODENAME) release-3.0" \
        > /etc/apt/sources.list.d/isaac-ros.list && \
    apt-get update

to my custom image layer.