[PROPOSAL] Process for Merging PRs with Incomplete Checks

[peterzhuamazon]

Process for Merging PRs with Incomplete Checks

Background

• As of now, only admins can force-merge PRs with failed/incomplete checks
• During release phases, slow/flaky checks block progress
• No clear process exists - teams randomly ping admins for force-merges
• No criteria for which PRs qualify for bypass

Prior issues

In opensearch-project/.github#356, it was suggested to give maintainer certain admin access to force merge PR, which need further discussion. Therefore, I want to expand that into actually having a proper review process.

Similar project

https://lf-onap.atlassian.net/wiki/spaces/DW/pages/16376959/Super+Committers

Proposed Process

1. Requirements before requesting force-merge:

   • PR must have 2+ maintainer approvals
   • Must show that failures are due to flaky/slow checks, not code issues
   • If a test is blocked because of workflow rules, fix the rules first
   • Critical fixes only during release phases (2week periods from RC1 generation to release)
   • Failed checks must be documented

2. Process:

   • PR owner opens ticket through existing github request: https://github.com/opensearch-project/.github/issues/new?template=GITHUB_REQUEST_TEMPLATE.yaml
     • PR link
     • Which checks failed and why
     • Release impact
   • 2+ Maintainer reviews and approves bypass request
   • 2+ Admin reviews and approves bypass request

3. Tracking:

   • Include issue to fix flaky test

Conclusion

We would like this to be a official documented process so it is easier for both the contributors and admins to reference and practice. It will also encourage more fixes to existing flaky test to further lower the percentage of these flakiness going on.

Thanks.

[peterzhuamazon]

Adding the BIG folks into the discussion on this.
@msfroh @getsaurabh02 @bugmakerrrrrr @virajsanghvi

Thanks.

[cwperks]

@peterzhuamazon I'm not inherently opposed to this (as a member of the admin group too), but there needs to be some backpressure mechanism to get repos to prioritize fixing the checks so that we can rely on them to catch issues as early as possible and before they get into released artifacts.

Can we use any metrics dashboard to catch the number of times that PRs are merging without (required) checks passing?

I know we've had an issue on the security repo lately where plugin install is very flaky on windows bc startup time for the OpenSearch process has degraded.

[peterzhuamazon]

Thanks @cwperks , let me add @prudhvigodithi into the discussion on metrics.