Skip to content

GH bot doesn't edit 'tests failed' comment to report coverage once tests are passing #3595

@arcward

Description

@arcward

Describe the bug

The codecov github bot will leave its usual coverage report comment when tests pass, and edit that comment to report test failures, but will not edit that comment again once tests pass (to include the coverage report) - it gets 'stuck.'

To Reproduce

I've experienced this in a github monorepo (python/typescript) which uploads multiple coverage reports using codecov/[email protected], and pytest reports using codecov/test-results-action@v1.

Steps to reproduce the behavior:

  1. Open a PR with passing tests
  2. See the codecov bot leave a PR comment/review with coverage results
  3. Push a change to the PR that results in broken test(s)
  4. See the bot edit the PR comment reporting the failing tests
  5. Push a change so all tests pass
  6. Check the comment again - it'll be 'stale' (not edited) and still reflect the broken test(s)

Expected behavior

If the bot edits its own comment from its usual coverage report, to instead reflect broken tests, I expect it to edit that comment again once tests are passing, to reflect the change in coverage.

Screenshots

Example part of the initial 'normal' coverage report comment:

Image

Which then gets edited in-place like this:

Image

But does not get re-edited once all tests are passing.

Additional context

This is our current config:

comment:
  layout: "header, diff, flags, files, feedback, components, footer"
  behavior: default
  require_changes: false
  require_base: false
  require_head: false
  hide_project_coverage: false
codecov:
  branch: develop
  require_ci_to_pass: false
  notify:
    wait_for_ci: true
    notify_error: true
coverage:
  status:
    project: true

There is one scenario I've found where it actually will edit the comment again: If codecov.notify.wait_for_ci is set to false. But, since we have multiple jobs uploading coverage reports, that results in the bot editing the comment multiple times, each time only reflecting one of the coverage reports (which is not surprising, but only creates a separate problem for our case)

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions