rosdep update is failing due to raw.github unauthenticated limit

[mllofriu]

I've seen rosdep update calls failing almost consistently from different IPs with error 429s.

rosdep update --rosdistro jazzy
/usr/bin/rosdep:6: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import load_entry_point
reading in sources list data from /etc/ros/rosdep/sources.list.d
ERROR: unable to process source [https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/osx-homebrew.yaml]:
        HTTP Error 429: Too Many Requests (https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/osx-homebrew.yaml)
ERROR: unable to process source [https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/base.yaml]:
        HTTP Error 429: Too Many Requests (https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/base.yaml)
ERROR: unable to process source [https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/python.yaml]:
        HTTP Error 429: Too Many Requests (https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/python.yaml)
ERROR: unable to process source [https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/ruby.yaml]:
        HTTP Error 429: Too Many Requests (https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/ruby.yaml)
ERROR: unable to process source [https://raw.githubusercontent.com/ros/rosdistro/master/releases/fuerte.yaml]:
        Failed to download target platform data for gbpdistro:
        HTTP Error 429: Too Many Requests
Query rosdistro index https://raw.githubusercontent.com/ros/rosdistro/master/index-v4.yaml
ERROR: error loading sources list
        HTTP Error 429: Too Many Requests (https://raw.githubusercontent.com/ros/rosdistro/master/index-v4.yaml)

Seems related to this discussion - https://github.com/orgs/community/discussions/159123.

Has anyone found a workaround? Using the mirrors seems to still end up hitting raw.github

[Rezenders]

Has anyone found a workaround? Using the mirrors seems to still end up hitting raw.github

I managed to get it working by creating a local mirror and using a github personal access token

I'm using a multi-staged docker image for this:

ARG ROS_DISTRO=jazzy
### Stage 1: create local mirror
FROM ubuntu:24.04 AS rosdistro-mirror

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y --no-install-recommends git ca-certificates && \
    rm -rf /var/lib/apt/lists/*

# We use BuildKit secret "gh_token" if provided. If not provided, we fall back to anonymous clone.
# The secret is available only during this RUN and is not persisted in the layer.
RUN --mount=type=secret,id=gh_token \
    set -eux; \
    TOKEN_FILE="/run/secrets/gh_token"; \
    if [ -f "$TOKEN_FILE" ] && [ -s "$TOKEN_FILE" ]; then \
        GITHUB_TOKEN="$(cat "$TOKEN_FILE")"; \
        git clone "https://x-access-token:${GITHUB_TOKEN}@github.com/ros/rosdistro.git" /tmp/rosdistro; \
    else \
        git clone https://github.com/ros/rosdistro.git /tmp/rosdistro; \
    fi; \
    rm -rf /tmp/rosdistro/.git

### Stage 2: Actually builds my image
FROM ros:${ROS_DISTRO}-ros-core

### bunch of stuff specific to my build (changed to another user)

# bring in the mirror prepared in Stage 1
COPY --from=rosdistro-mirror /tmp/rosdistro /home/ubuntu/rosdistro-mirror

# make rosdep use ONLY the local mirror
# Replace rosdep source list with local file:// URIs only
RUN set -eux; \
    sudo mkdir -p /etc/ros/rosdep/sources.list.d; \
    sudo rm -f /etc/ros/rosdep/sources.list.d/20-default.list; \
    sudo printf '%s\n%s\n%s\n%s\n%s\n' \
      'yaml file:///home/ubuntu/rosdistro-mirror/rosdep/base.yaml' \
      'yaml file:///home/ubuntu/rosdistro-mirror/rosdep/python.yaml' \
      'yaml file:///home/ubuntu/rosdistro-mirror/rosdep/ruby.yaml' \
      | sudo tee /etc/ros/rosdep/sources.list.d/01-local-rosdistro.list > /dev/null

# Force rosdep to use the local rosdistro index instead of GitHub
ENV ROSDISTRO_INDEX_URL=file:///home/ubuntu/rosdistro-mirror/index-v4.yaml

WORKDIR $HOME/my_workspace/
RUN ["/bin/bash", "-c", "source /opt/ros/${ROS_DISTRO}/setup.bash \
    && sudo apt update \
    && rosdep update \
    && rosdep install --from-paths src --ignore-src -r -y \
    && sudo rm -rf /var/lib/apt/lists/"]

Then I pass my gh token as a secret when building the image:

docker build --secret id=gh_token,src=<(gh auth token) --build-arg ROS_DISTRO=jazzy -t my_image -f Dockerfile .

[cottsay]

Cross-link to colcon/colcon-mixin#50, if we can verify that the approach taken there is effective at mitigating the 429s from GitHub, we can port the changes to rosdep and rosdistro as well.