Upload multiple reports with different flags using single config

[Ekleog-NEAR]

Hello!

I’ll start with, this is a duplicate of closed-by-author codecov/codecov-action#243, and of the discussion that happened after codecov/codecov-action#50 was closed.

I’d like to upload multiple reports with different flags. It currently seems to basically require calling the codecov actions multiple times. However, it results in a lot of copy-paste, and workarounds like using steps: fromJson() with a custom generator script is even more painful.

Would it be possible to upload multiple reports with different flags? I’m thinking of a syntax like:

- uses: actions/codecov-action@v??
  with:
    token: ${{ secrets.CODECOV_TOKEN }}
    fail_ci_if_error: true
    files:
      - { path: "unit-tests.json", flags: "unit-tests" }
      - { path: "integration-tests.json", flags: "integration-tests" }

What do you think?

[Ekleog-NEAR]

Also, for some context: we currently have 11 reports, because of the test matrix and splitting unit and integration tests, and I think it’s quite likely to grow.

Having the ability to take all in the files argument would hopefully also allow using files: fromJson() and auto-computing the list of files with flags, for the bigger workloads :)

[thomasrockhu-codecov]

@Ekleog-NEAR, no this is not possible as specified. You can instead run the action twice, each specifying a file and a flag

e.g.

- uses: actions/codecov-action@??
  with:
    token: ${{ secrets.CODECOV_TOKEN }}
    fail_ci_if_error: true
    files:
      - unit-tests.json
    flags:
      - unit-tests

- uses: actions/codecov-action@??
  with:
    token: ${{ secrets.CODECOV_TOKEN }}
    fail_ci_if_error: true
    files:
      - integration-tests.json
    flags:
      - integration-tests

I would recommend trying this with v4-beta, but it should work on v3 too

[Ekleog-NEAR]

@thomasrockhu-codecov Sorry for being unclear in my message!

I totally understand that it is currently not possible. What I’m wondering is, would you be interested in changing codecov-action, so that it becomes possible? It feels like a change that would be welcome in v4, and probably not be too hard to implement.

Right now our solution is indeed to have multiple calls to the action, but it’s quite a lot of copy-paste, which is never good :/

[thomasrockhu-codecov]

@Ekleog-NEAR oh I see. I'm going to move this our feedback repo and get some input from our PMs.

[drazisil-codecov]

Hi @Ekleog-NEAR ,

This is a feature request, as Tom mentioned, so we'll need for the product team to take a look.

Meanwhile, have you check https://docs.codecov.com/docs/components ? It was created to partly address the ask of wanting to cover multiple coverage sections with one upload.

[Ekleog-NEAR]

Great, thank you for submitting the idea to product team!

As for components, it’s the next thing I’m planning to look into, as soon as I have some time to dig into our code coverage setup again. But AFAICT it’s about splitting reports by source file path basically, is it not?

Here I have so many reports not because I split them by file path, but because we run just as many broad types of tests (unit tests on 3 OSes/configurations, integration tests on 2 OSes/configs, and a few end-to-end tests with different code configurations) and want to know for each coverage line from which source they came from, in order to set different code coverage targets.

AFAIU components would thus not be able to address this need, because here all the reports each affects the whole source tree. Am I understanding correctly?

[drazisil-codecov]

Hi @Ekleog-NEAR , You are indeed. Sounds like components would not address your use case.

(cc: @rohan-at-sentry :)

[Ekleog-NEAR]

Got it, thank you! :)

[btrautmann]

Just joining the convo here (seems it has gone stale though).. We are exploring a monorepo setup for our OSS projects and would like to upload/maintain reports for each of the discrete packages, but seems we will need to run this action 9 times to have the desired effect. Would love for this to be implemented!