Prevent auto-merge when UT fail

[whitneywhtsang]

@pbchekin #4423 got automerged when there are unit test failures, we may need to fix the merge requirements.

Originally posted by @whitneywhtsang in #4424 (comment)

[kwasd]

Looks like the branch protection rules are correct, but we encountered this: https://emmer.dev/blog/skippable-github-status-checks-aren-t-really-required/

If you have a GitHub branch protection rule , and that rule requires "status checks to pass before merging", GitHub will treat skipped GitHub Actions jobs as "passing".

Read that again. The actual verbiage in the branch protection settings page, as of writing, is:

☑ Require status checks to pass before merging
Choose which status checks must pass before branches can be merged into a branch that matches this rule.

I'm fairly confident stating most people would not expect "skipped" to equate to "passed". If my GitHub Actions job (e.g. running npm test) never ran, then it cannot pass.

[whitneywhtsang]

What can be done in our case, can we make all the subtasks as required too?

[kwasd]

Yes, it is possible by modifying the workflow, but needs a bit of designing. I think I will do it tomorrow.

As a quick workaround, we can set rules to require also the build job to be completed. This will cover the case when build was not ran at all.

Applied this now.

before	after

[anmyachev]

We could run Reports job always and check results from all needed previous jobs using https://github.com/marketplace/actions/alls-green like it done for example in pytest: https://github.com/pytest-dev/pytest/blame/e1af2c9609c8391394b8dc4f31ba6225c899b02f/.github/workflows/test.yml#L308. @kwasd WDYT?

[kwasd]

Yes, this is exactly what I'm thinking about :)