Webhook (or other alerts) when a build has been clogging up the queue

[charliegroll]

Problem

Some of our build checks will be pending until someone takes action (approving visual regression snapshot changes, for example). With kodiak, most people tend to "set it to merge and forget it."

Proposed solution

Set a configurable threshold time that publishes to a webhook after that time has passed to notify that the queue is blocked (and maybe repeat the check X times as a reminder?). My team could subscribe to this in Slack (or PagerDuty, or email, etc) so we can unblock our queue before it backs up too much.

[chdsbd]

I think implementation wise this is definitely doable

one workaround that might work in the meantime would be to try this: https://kodiakhq.com/docs/config-reference#mergedont_wait_on_status_checks

Kodiak won’t wait for those PRs that have that status check in a missing state

[chdsbd]

@charliegroll Do you use AWS? I was thinking we could allow users to configure an SNS topic in their .kodiak.toml that Kodiak would publish to. I think that's a little easier than trying to build a web hook system

[chdsbd]

Another follow up question. When a developer approves a snapshot change, will that GitHub Status Check ever be in the pending state again?

For example, if a developer approves the snapshot changes, and then Kodiak updates the PR branch before merging, would that snapshot GitHub Status Check rerun and temporarily be in the pending state before returning to "success"?

If the status check remains either in a "pending" state or "success" state until there's a developers approval, I think merge.dont_wait_on_status_checks should work

[charliegroll]

Another follow up question. When a developer approves a snapshot change, will that GitHub Status Check ever be in the pending state again?

For example, if a developer approves the snapshot changes, and then Kodiak updates the PR branch before merging, would that snapshot GitHub Status Check rerun and temporarily be in the pending state before returning to "success"?

I believe you're asking if approval -> update from main -> pending again, in which case I believe that's a possibility, say, if there's a flaky snapshot.

Do you use AWS? I was thinking we could allow users to configure an SNS topic in their .kodiak.toml that Kodiak would publish to. I think that's a little easier than trying to build a web hook system

We theoretically have access to AWS, but I was hoping to set something up directly with slack without an intermediary (plus, we don't generally interact with AWS on my team and would need another team's assistance)

[charliegroll]

I'll check out https://kodiakhq.com/docs/config-reference#mergedont_wait_on_status_checks @chdsbd , thanks!

[charliegroll]

I haven't noticed our queue getting blocked so I think this is working ❤️

[chdsbd]

That's great to hear!