codecov worker crashing

[isatfg]

Hi

Firstly thanks for creating this chart. Im trying to deploy the chart and all workloads are running except for the worker. It's giving the error below even thought I have setup the s3 configuration.

codecov-worker-7557d59c7d-t67qd codecov Traceback (most recent call last):
codecov-worker-7557d59c7d-t67qd codecov   File "/worker/main.py", line 154, in <module>
codecov-worker-7557d59c7d-t67qd codecov     main()
codecov-worker-7557d59c7d-t67qd codecov     ~~~~^^
codecov-worker-7557d59c7d-t67qd codecov   File "/worker/main.py", line 150, in main
codecov-worker-7557d59c7d-t67qd codecov     cli(obj={})
codecov-worker-7557d59c7d-t67qd codecov     ~~~^^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/click/core.py", line 1157, in __call__
codecov-worker-7557d59c7d-t67qd codecov     return self.main(*args, **kwargs)
codecov-worker-7557d59c7d-t67qd codecov            ~~~~~~~~~^^^^^^^^^^^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/click/core.py", line 1078, in main
codecov-worker-7557d59c7d-t67qd codecov     rv = self.invoke(ctx)
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/click/core.py", line 1688, in invoke
codecov-worker-7557d59c7d-t67qd codecov     return _process_result(sub_ctx.command.invoke(sub_ctx))
codecov-worker-7557d59c7d-t67qd codecov                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/click/core.py", line 1434, in invoke
codecov-worker-7557d59c7d-t67qd codecov     return ctx.invoke(self.callback, **ctx.params)
codecov-worker-7557d59c7d-t67qd codecov            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/click/core.py", line 783, in invoke
codecov-worker-7557d59c7d-t67qd codecov     return __callback(*args, **kwargs)
codecov-worker-7557d59c7d-t67qd codecov   File "/worker/main.py", line 116, in worker
codecov-worker-7557d59c7d-t67qd codecov     setup_worker()
codecov-worker-7557d59c7d-t67qd codecov     ~~~~~~~~~~~~^^
codecov-worker-7557d59c7d-t67qd codecov   File "/worker/main.py", line 84, in setup_worker
codecov-worker-7557d59c7d-t67qd codecov     storage_client = shared.storage.get_appropriate_storage_service()
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/shared/storage/__init__.py", line 27, in get_appropriate_storage_service
codecov-worker-7557d59c7d-t67qd codecov     _get_appropriate_storage_service_given_storage(chosen_storage)
codecov-worker-7557d59c7d-t67qd codecov     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/shared/storage/__init__.py", line 53, in _get_appropriate_storage_service_given_storage
codecov-worker-7557d59c7d-t67qd codecov     return MinioStorageService(minio_config)
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/shared/storage/minio.py", line 31, in __init__
codecov-worker-7557d59c7d-t67qd codecov     self.minio_client = self.init_minio_client(
codecov-worker-7557d59c7d-t67qd codecov                         ~~~~~~~~~~~~~~~~~~~~~~^
codecov-worker-7557d59c7d-t67qd codecov         self.minio_config["host"],
codecov-worker-7557d59c7d-t67qd codecov         ^^^^^^^^^^^^^^^^^^^^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov     ...<6 lines>...
codecov-worker-7557d59c7d-t67qd codecov         self.minio_config.get("region"),
codecov-worker-7557d59c7d-t67qd codecov         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov     )
codecov-worker-7557d59c7d-t67qd codecov     ^
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/shared/storage/minio.py", line 96, in init_minio_client
codecov-worker-7557d59c7d-t67qd codecov     return Minio(
codecov-worker-7557d59c7d-t67qd codecov         host,
codecov-worker-7557d59c7d-t67qd codecov     ...<3 lines>...
codecov-worker-7557d59c7d-t67qd codecov         region=region,
codecov-worker-7557d59c7d-t67qd codecov     )
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/minio/api.py", line 130, in __init__
codecov-worker-7557d59c7d-t67qd codecov     self._base_url = BaseURL(
codecov-worker-7557d59c7d-t67qd codecov                      ~~~~~~~^
codecov-worker-7557d59c7d-t67qd codecov         ("https://" if secure else "http://") + endpoint,
codecov-worker-7557d59c7d-t67qd codecov         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov         region,
codecov-worker-7557d59c7d-t67qd codecov         ^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov     )
codecov-worker-7557d59c7d-t67qd codecov     ^
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/minio/helpers.py", line 426, in __init__
codecov-worker-7557d59c7d-t67qd codecov     self._accelerate_host_flag = host.startswith("s3-accelerate.")
codecov-worker-7557d59c7d-t67qd codecov                                  ^^^^^^^^^^^^^^^
codecov-worker-7557d59c7d-t67qd codecov AttributeError: 'NoneType' object has no attribute 'startswith'
codecov-worker-7557d59c7d-t67qd codecov Exception ignored in: <function Minio.__del__ at 0x7fc1f9d48540>
codecov-worker-7557d59c7d-t67qd codecov Traceback (most recent call last):
codecov-worker-7557d59c7d-t67qd codecov   File "/usr/local/lib/python3.13/site-packages/minio/api.py", line 155, in __del__
codecov-worker-7557d59c7d-t67qd codecov     self._http.clear()
codecov-worker-7557d59c7d-t67qd codecov AttributeError: 'Minio' object has no attribute '_http'
- codecov-worker-7557d59c7d-t67qd › codecov

My values file

codecov_config: |
  s3:
    host: s3.amazonaws.com
....

and

minio:
#  embedded: false
#  # externalHost: "minio.example.com"
#  # externalPort: "443"
#  # embedded minio is not supported
#  # codecov needs a external available minio
#  # please use external minio
#  # pr are welcomed
#  # embedded: false
#  # externalMinio:
# If using external storage. Comment above and uncomment below
    host: s3.amazonaws.com
    bucket: xxx
    region: eu-west-1
    verify_ssl: true
    port: 443
    access_key_id: 'xxx'
    secret_access_key: 'xxx+'

Any idea what I'm doing wrong?

[LeoQuote]

Sorry for the confusion, the values file is not well documented.

You need to set s3 config inside codecov_config instead of minio.host

Example:

codecov_config: |
  services:
    minio:
      bucket: codecov
      region: us-east-1
      verify_ssl: true

Refer to the official example for more hints.

Any kind of pull requests are welcomed.

https://github.com/codecov/self-hosted/blob/main/config/codecov.yml

[isatfg]

I have resolved the above mention woker issue by adding the s3 config as below.

codecov_config: |
  setup:
...
  minio:
     externalHost: "s3.amazonaws.com"
     bucket: "xx"
     region: "eu-west-1"
     verify_ssl: true
     externalPort: 443
     access_key_id: 'XX'
     secret_access_key: 'XX'

....
minio:
# If using external storage. Comment above and uncomment below
   externalHost: "s3.amazonaws.com"
   bucket: "xx"
   region: "eu-west-1"
   verify_ssl: true
   externalPort: 443
   access_key_id: 'XX'
   secret_access_key: 'XX'
   iam_auth: false #<boolean, default false> # set to true in AWS to attempt to authenticate via Instance role

However now when I try to upload the report either using the cli or via github actions I get the following error in the api logs saying InvalidAccessKeyId but the access id is valid and can access the bucket.

Any idea what could be the problem?

codecov-api-686fb6d9d-m6qzj codecov-api {"message": "Internal Server Error: /upload/github/xxxxxxxx::::xxxxxxxx/commits/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/reports/default/uploads", "asctime": "2025-04-08 14:33:48,597", "name": "django.request", "levelname": "ERROR", "lineno": 241, "pathname": "/usr/local/lib/python3.12/site-packages/django/utils/log.py", "funcName": "log_response", "threadName": "MainThread", "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py\", line 55, in inner\n    response = get_response(request)\n               ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/django/core/handlers/base.py\", line 197, in _get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/django/views/decorators/csrf.py\", line 56, in wrapper_view\n    return view_func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/django/views/generic/base.py\", line 104, in view\n    return self.dispatch(request, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/views.py\", line 509, in dispatch\n    response = self.handle_exception(exc)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/views.py\", line 469, in handle_exception\n    self.raise_uncaught_exception(exc)\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/views.py\", line 480, in raise_uncaught_exception\n    raise exc\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/views.py\", line 506, in dispatch\n    response = handler(request, *args, **kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/generics.py\", line 246, in post\n    return self.create(request, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/mixins.py\", line 20, in create\n    headers = self.get_success_headers(serializer.data)\n                                       ^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/serializers.py\", line 571, in data\n    ret = super().data\n          ^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/serializers.py\", line 249, in data\n    self._data = self.to_representation(self.instance)\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/serializers.py\", line 538, in to_representation\n    ret[field.field_name] = field.to_representation(attribute)\n                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/rest_framework/fields.py\", line 1870, in to_representation\n    return method(value)\n           ^^^^^^^^^^^^^\n  File \"/app/upload/serializers.py\", line 56, in get_raw_upload_location\n    return archive_service.create_presigned_put(obj.storage_path)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/shared/api_archive/archive.py\", line 173, in create_presigned_put\n    return self.storage.create_presigned_put(self.root, path, self.ttl)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/shared/api_archive/storage.py\", line 48, in create_presigned_put\n    return self.minio_client.presigned_put_object(bucket, path, expires)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/minio/api.py\", line 2150, in presigned_put_object\n    return self.get_presigned_url(\n           ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/minio/api.py\", line 2053, in get_presigned_url\n    region = self._get_region(bucket_name, None)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/minio/api.py\", line 467, in _get_region\n    response = self._url_open(\n               ^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/minio/api.py\", line 386, in _url_open\n    raise response_error\nminio.error.S3Error: S3 operation failed; code: InvalidAccessKeyId, message: The AWS Access Key Id you provided does not exist in our records., resource: None, request_id: xxx, host_id: xxx=", "taskName": null, "status_code": 500, "request": "<WSGIRequest: POST '/upload/github/xx-xx::::xx/commits/xx/reports/default/uploads'>", "utctime": "2025-04-08T14:33:48.597000", "logger.name": "django.request", "logger.thread_name": "MainThread", "level": "ERROR"}

[LeoQuote]

For security reasons, I did not place the access key to the config file, instead I set the value by environment variable, I think this could be a fix but a more reliable solution is to check the request with the request id, and check the source code you are using to see if there’s anything wrong.

extraEnvs:
- name: services__minio__access_key_id
  valueFrom:
    secretKeyRef:
      name: codecov
      key: minio__access_key_id
- name: services__minio__secret_access_key
  valueFrom:
    secretKeyRef:
      name: codecov
      key: minio__secret_access_key

There’s one more thing you should be aware, you need to set minio.externalHost and minio.externalPort as the chart renders these two value to environment variables as well, this could override the config in your codecov.yaml

I haven’t touched this code for over two years 😅 I can’t remember why it’s designed this way, any kind of refinement is welcomed 🙏

[isatfg]

Thanks for the help thus far and sorry for all the messages.

I update the chart as you recommended now I get this error. I'm 100% sure the access key is fine, when I test using docker-compose it uploads just fine with the same access key.

codecov-api-84f8bd9ff-t27d5 codecov-api {"message": "Internal Server Error: /upload/github/xxxxxxxxxxxxxxxthelm/commits/67e7879a0a834386f21ac1ef407ca90184de91fc/reports/default/uploads", "asctime": "2025-04-09 12:38:27,385", "name": "django.request", "levelname": "ERROR", "lineno": 241, "pathname": "/usr/local/lib/python3.12/site-packages/django/utils/log.py", "funcName": "log_response", "threadName": "MainThread", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py\", line 55, in inner\n response = get_response(request)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/django/core/handlers/base.py\", line 197, in _get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/django/views/decorators/csrf.py\", line 56, in wrapper_view\n return view_func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/django/views/generic/base.py\", line 104, in view\n return self.dispatch(request, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/views.py\", line 509, in dispatch\n response = self.handle_exception(exc)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/views.py\", line 469, in handle_exception\n self.raise_uncaught_exception(exc)\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/views.py\", line 480, in raise_uncaught_exception\n raise exc\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/views.py\", line 506, in dispatch\n response = handler(request, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/generics.py\", line 246, in post\n return self.create(request, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/mixins.py\", line 20, in create\n headers = self.get_success_headers(serializer.data)\n ^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/serializers.py\", line 571, in data\n ret = super().data\n ^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/serializers.py\", line 249, in data\n self._data = self.to_representation(self.instance)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/serializers.py\", line 538, in to_representation\n ret[field.field_name] = field.to_representation(attribute)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/rest_framework/fields.py\", line 1870, in to_representation\n return method(value)\n ^^^^^^^^^^^^^\n File \"/app/upload/serializers.py\", line 56, in get_raw_upload_location\n return archive_service.create_presigned_put(obj.storage_path)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/shared/api_archive/archive.py\", line 173, in create_presigned_put\n return self.storage.create_presigned_put(self.root, path, self.ttl)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/shared/api_archive/storage.py\", line 48, in create_presigned_put\n return self.minio_client.presigned_put_object(bucket, path, expires)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/minio/api.py\", line 2150, in presigned_put_object\n return self.get_presigned_url(\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/minio/api.py\", line 2053, in get_presigned_url\n region = self._get_region(bucket_name, None)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/minio/api.py\", line 467, in _get_region\n response = self._url_open(\n ^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/site-packages/minio/api.py\", line 386, in _url_open\n raise response_error\nminio.error.S3Error: S3 operation failed; code: AccessDenied, message: Access Denied, resource: None, request_id: RCENM2RFXD85XW2K, host_id: wtFnMXk8uMijNj/w3mADpCBTWlEZ5Od9Ut+q0YkNom0DEELrPdhGKvAFGmMDud1bxTWKpI6nVMI=", "taskName": null, "status_code": 500, "request": "<WSGIRequest: POST '/upload/github/xxx::::demo/commits/67e7879a0a834386f21ac1ef407ca90184de91fc/reports/default/uploads'>", "utctime": "2025-04-09T12:38:27.385000", "logger.name": "django.request", "logger.thread_name": "MainThread", "level": "ERROR"}

[LeoQuote]

If docker-compose works for you, I’d suggest comparing the configs and environment variables in both k8s and docker compose, make their configs identical, including image tag, etc..basically k8s is just a heavier docker compose.

You can cat the config file, printenv to show envs, anyway i think it’s a good news that docker compose works for you, you can at least have one right configuration.

[isatfg]

Thanks again. I decided to use my own chart and took some configs from yours such as setting env variable services__* my chart seems to be working will keep testing over the next few days. I was unable to trace the root cause of the issue I was getting using your chart.