AWS registry is not working properly with 21.4.26 #343
Description
We cannot build the images using Kaniko with the nevest version of platformregistery on AWS clusters.
The problem appears when the Kaniko pulls cached layer from our registry to speedup the image build.
At this step, our registry fails with error:
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
2021-06-29 07:55:44,393 - aiohttp.server - ERROR - Error handling request
Traceback (most recent call last):
File "/root/.local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
resp = await self._request_handler(request)
File "/root/.local/lib/python3.7/site-packages/sentry_sdk/integrations/aiohttp.py", line 123, in sentry_app_handle
reraise(*_capture_exception(hub))
File "/root/.local/lib/python3.7/site-packages/sentry_sdk/_compat.py", line 54, in reraise
raise value
File "/root/.local/lib/python3.7/site-packages/sentry_sdk/integrations/aiohttp.py", line 113, in sentry_app_handle
response = await old_handle(self, request)
File "/root/.local/lib/python3.7/site-packages/aiohttp/web_app.py", line 499, in _handle
resp = await handler(request)
File "/root/.local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
return await handler(request)
File "/root/.local/lib/python3.7/site-packages/aiohttp_remotes/x_forwarded.py", line 94, in middleware
return await handler(request)
File "/root/.local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
return await handler(request)
File "/root/.local/lib/python3.7/site-packages/platform_registry_api/api.py", line 599, in handle
auth_headers=auth_headers,
File "/root/.local/lib/python3.7/site-packages/platform_registry_api/api.py", line 711, in _proxy_request
async for chunk in client_response.content.iter_any():
File "/root/.local/lib/python3.7/site-packages/aiohttp/streams.py", line 39, in __anext__
rv = await self.read_func()
File "/root/.local/lib/python3.7/site-packages/aiohttp/streams.py", line 386, in readany
raise self._exception
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Which means that the image layer was not download compeletely and, as a result, Kaniko fails on the layer checksum verification.
Example of Kaniko error message:
INFO[0153] Found cached layer, extracting to filesystem
error building image: error building stage: failed to execute command: extracting fs from image: error verifying sha256 checksum; got "9a593afe07a07cc862136409824a9b391a352945a4dcbbd7de8084ab8b13d572", want "92c6c2f9925e03fe28f4e343e6aa128aeb78eccc59c50589cefa15db665857c8"
This problem is not reproducible with platform registry api v21.2.11.
As a result, we downgraded our platformregistryapi on AWS clusters from v21.4.26 to v21.2.11.
IDK yet, what might be the reason for this.