Flaky CI

[droctothorpe]

Let's use this discussion to post flaky CI runs. Hopefully aggregating these results across multiple PRs / branches will help identify patterns and root cause.

[zazulam]

local execution test with docker for SDK Tests - https://github.com/kubeflow/pipelines/actions/runs/18879131433/job/53876987256?pr=12248#step:8:2095

[droctothorpe]

Apparently, this has been a recurring source of flakiness:

https://github.com/kubeflow/pipelines/actions/runs/18879131433/job/53876987256?pr=12248

=================================== FAILURES ===================================
____________ TestDockerRunner.test_execution[Container Hello World] ____________

self = <docker.api.client.APIClient object at 0x7f07ae0ed6d0>
response = <Response [404]>

    def _raise_for_status(self, response):
        """Raises stored :class:`APIError`, if one occurred."""
        try:
>           response.raise_for_status()

/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/docker/api/client.py:268: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Response [404]>

    def raise_for_status(self):
        """Raises :class:`HTTPError`, if one occurred."""
    
        http_error_msg = ""
        if isinstance(self.reason, bytes):
            # We attempt to decode utf-8 first because some servers
            # choose to localize their reason strings. If the string
            # isn't utf-8, we fall back to iso-8859-1 for all other
            # encodings. (See PR #3538)
            try:
                reason = self.reason.decode("utf-8")
            except UnicodeDecodeError:
                reason = self.reason.decode("iso-8859-1")
        else:
            reason = self.reason
    
        if 400 <= self.status_code < 500:
            http_error_msg = (
                f"{self.status_code} Client Error: {reason} for url: {self.url}"
            )
    
        elif 500 <= self.status_code < 600:
            http_error_msg = (
                f"{self.status_code} Server Error: {reason} for url: {self.url}"
            )
    
        if http_error_msg:
>           raise HTTPError(http_error_msg, response=self)
E           requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.48/images/public.ecr.aws/docker/library/python:3.12/json

[zazulam]

Happened again on master after a successful build
https://github.com/kubeflow/pipelines/actions/runs/18880973690/job/53883667520#step:8:3462