[CLI/API] When the CLI command `empty-upload --force` is used on a repo with carry forward flags, the notifications fail

[drazisil-codecov]

When the CLI command empty-upload --force is used on a repo with carry forward flags, Notifications fail due to the notifications task being called by API before the report has a chance to be populated with the report totals

[eliatcodecov]

Note that --force isn't documented and is only discoverable in code. At a minimum we should document this and state that it doesn't work with carryforward flags.

More generally, we should align on why this feature exists, if it should exist, and if so how to document it. It appears this was just implemented as a convenience method to send notifications without necessarily having to call send notification manually. Do we even to have this flag?

[erickpeirson]

I believe that we are impacted by this issue. We use empty-upload to ensure that we trigger a CodeCov report regardless of whether CI has generated coverage data, so that we can globally require the codecov/patch status to merge pull requests.

We use carry-forward flags extensively.

Note that without the --force flag, calls to empty-upload fail with:

info - 2024-09-10 18:24:39,260 -- Process Empty Upload complete
debug - 2024-09-10 18:24:39,260 -- Empty Upload result --- {"result": "RequestResult(error=RequestError(code='HTTP Error 404', params={}, description='{\"detail\":\"Unable to get pull request for commit: 19cc963c9bda826fe1a8e4d2e00bf8b05cede023\"}'), warnings=[], status_code=404, text='{\"detail\":\"Unable to get pull request for commit: 19cc963c9bda826fe1a8e4d2e00bf8b05cede023\"}')"}
error - 2024-09-10 18:24:39,260 -- Empty Upload failed: {"detail":"Unable to get pull request for commit: 19cc963c9bda826fe1a8e4d2e00bf8b05cede023"}

What is the correct way to ensure that CodeCov always generates a report on PRs in a project that relies on carry-forward flags?

[drazisil-codecov]

Hi @erickpeirson,

Can you please try calling create-commit before empty-upload?

You might need to call create-report in between those two, but I think empty-upload handles the same thing.

You should then get your notifications after we generate the "empty" report with all the carried forward coverage data.

[erickpeirson]

Can you please try calling create-commit before empty-upload?

Yes, we are already doing this today

You might need to call create-report in between those two, but I think empty-upload handles the same thing.

This appears to have done the trick!

[drazisil-codecov]

(Forgot this was was a bug ticket and not a customer one 🤦‍♀

[erickpeirson]

@drazisil-codecov bad news.... this is now more broken than before. This is having a pretty big impact on CI for us.

debug - 2024-09-12 20:15:37,714 -- Using ci service from provider name: BuildKite
--
  | debug - 2024-09-12 20:15:37,715 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
  | debug - 2024-09-12 20:15:37,717 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
  | debug - 2024-09-12 20:15:37,718 -- Loading config from /workdir/codecov.yml
  | debug - 2024-09-12 20:15:37,735 -- Starting create commit process --- {"commit_sha": "aa202051e59e09d04ca3f2e8b08069f6ba9a6148", "parent_sha": null, "pr": null, "branch": "main", "slug": "datagravity-ai/dquality", "token": "6******************", "service": "github", "enterprise_url": null}
  | info - 2024-09-12 20:15:37,900 -- Process Commit creating complete
  | debug - 2024-09-12 20:15:37,900 -- Commit creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=201, text='{\"message\":\"...\",\"timestamp\":\"2024-09-12T15:53:32Z\",\"ci_passed\":false,\"state\":\"complete\",\"repository\":{\"name\":\"dquality\",\"is_private\":true,\"active\":true,\"language\":\"python\",\"yaml\":{\"ignore\":[\"^ai_lib/sql_fixer.*\",\"^ai_lib/sql_generator.*\",\"^www/manage.py.*\",\"(?s:www/site_apps/main/migrations/[^\\\\\\\\/]*_auto_[^\\\\\\\\/]*\\\\\\\\.py)\\\\\\\\Z\",\"^www/tests.*\"],\"comment\":{\"layout\":\"reach, diff, components, files\",\"show_carryforward_flags\":false},\"flag_management\":{\"default_rules\":{\"statuses\":[{\"base\":\"auto\",\"type\":\"project\",\"target\":\"auto\",\"threshold\":1.0,\"if_ci_failed\":\"error\"},{\"type\":\"patch\",\"target\":100.0,\"if_ci_failed\":\"error\"}],\"carryforward\":true},\"individual_flags\":[{\"name\":\"dquality\",\"paths\":[\".*\"],\"carryforward\":true},{\"name\":\"backtests\",\"paths\":[\".*\"],\"carryforward\":true},{\"name\":\"shared\",\"paths\":[\"shared/.*\"],\"carryforward\":true},{\"name\":\"web-backend\",\"paths\":[\"www/.*\"],\"carryforward\":true}]},\"component_management\":{\"individual_components\":[{\"name\":\"Data warehouse connectors\",\"paths\":[\"^dquality/db.*\"],\"component_id\":\"dbconnectors\"},{\"name\":\"Data quality core\",\"paths\":[\"^dquality.*\"],\"component_id\":\"dquality\"},{\"name\":\"Backend web app\",\"paths\":[\"^www.*\"],\"component_id\":\"backend\"},{\"name\":\"Shared library\",\"paths\":[\"^shared.*\"],\"component_id\":\"shared\"},{\"name\":\"CLI apps\",\"paths\":[\"^apps.*\"],\"component_id\":\"apps\"}]}}},\"author\":{\"avatar_url\":\"https://avatars0.githubusercontent.com/u/6236330?v=3&s=55\",\"service\":\"github\",\"username\":\"kfoxb\",\"name\":\"Kyle Bradford\",\"ownerid\":3670035},\"commitid\":\"aa202051e59e09d04ca3f2e8b08069f6ba9a6148\",\"parent_commit_id\":\"309fd0073938a8a90743e84fb53e14b1161013c6\",\"pullid\":null,\"branch\":\"main\"}')"}
  | debug - 2024-09-12 20:15:38,179 -- Using ci service from provider name: BuildKite
  | debug - 2024-09-12 20:15:38,181 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
  | debug - 2024-09-12 20:15:38,182 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
  | debug - 2024-09-12 20:15:38,183 -- Loading config from /workdir/codecov.yml
  | debug - 2024-09-12 20:15:38,193 -- Starting create report process --- {"commit_sha": "aa202051e59e09d04ca3f2e8b08069f6ba9a6148", "code": "default", "slug": "datagravity-ai/dquality", "service": "github", "enterprise_url": null, "token": "6******************"}
  | info - 2024-09-12 20:15:38,320 -- Process Report creating complete
  | debug - 2024-09-12 20:15:38,320 -- Report creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=201, text='{\"code\":null}')"}
  | info - 2024-09-12 20:15:38,320 -- Finished creating report successfully --- {"response": "{\"code\":null}"}
  | debug - 2024-09-12 20:15:38,598 -- Using ci service from provider name: BuildKite
  | debug - 2024-09-12 20:15:38,600 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
  | debug - 2024-09-12 20:15:38,601 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
  | debug - 2024-09-12 20:15:38,602 -- Loading config from /workdir/codecov.yml
  | debug - 2024-09-12 20:15:38,612 -- Starting empty upload process --- {"commit_sha": "aa202051e59e09d04ca3f2e8b08069f6ba9a6148", "slug": "datagravity-ai/dquality", "token": "6******************", "service": "github", "enterprise_url": null, "fail_on_error": false}
  | warning - 2024-09-12 20:15:38,765 -- Response status code was 500. --- {"retry": 0}
  | warning - 2024-09-12 20:15:38,765 -- Request failed. Retrying --- {"retry": 0}
  | warning - 2024-09-12 20:15:39,414 -- Response status code was 500. --- {"retry": 1}
  | warning - 2024-09-12 20:15:39,414 -- Request failed. Retrying --- {"retry": 1}
  | warning - 2024-09-12 20:15:40,588 -- Response status code was 500. --- {"retry": 2}
  | warning - 2024-09-12 20:15:40,588 -- Request failed. Retrying --- {"retry": 2}
  | Traceback (most recent call last):
  | File "/usr/local/bin/codecovcli", line 8, in <module>
  | sys.exit(run())
  | File "/usr/local/lib/python3.10/site-packages/codecov_cli/main.py", line 81, in run
  | cli(obj={})
  | File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
  | return self.main(*args, **kwargs)
  | File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
  | rv = self.invoke(ctx)
  | File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
  | return _process_result(sub_ctx.command.invoke(sub_ctx))
  | File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
  | return ctx.invoke(self.callback, **ctx.params)
  | File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
  | return __callback(*args, **kwargs)
  | File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
  | return f(get_current_context(), *args, **kwargs)
  | File "/usr/local/lib/python3.10/site-packages/codecov_cli/commands/empty_upload.py", line 41, in empty_upload
  | return empty_upload_logic(
  | File "/usr/local/lib/python3.10/site-packages/codecov_cli/services/empty_upload/__init__.py", line 22, in empty_upload_logic
  | sending_result = send_post_request(
  | File "/usr/local/lib/python3.10/site-packages/codecov_cli/helpers/request.py", line 82, in wrapper
  | raise Exception("Request failed after too many retries")
  | Exception: Request failed after too many retries