scm-workflow integrations get disabled even when there is no error

[dirkmueller]

Issue Description

For the last couple of months, we are facing continuously the automatic disablement of obs scm-workflow integrations with GitHub, even when there is no error in delivering the response. for example:

https://build.opensuse.org/my/tokens/9212/workflow_runs/717523#tab-pane-scm-reports-tab-content717523

Expected Result

Workflows do not get auto disabled when there is no reason for them to be disabled.

How to Reproduce

we don't know exactly what circumstance triggers it, but it is happening multiple times per month and therefore is highly unreliable. without any notification, we're back to monitoring the situation and reenabling the service whenever it gets disabled. This is starting to cause real productivity loss.

[Arbaaz123676]

@dirkmueller

• Do we log the exact HTTP status codes returned by GitHub before the integration gets disabled?
• Is there a failure counter / threshold that could be triggered even when later deliveries succeed?
• Are 202 / 204 responses treated as non-success in the SCM workflow logic?
  If there’s a specific component or job responsible for disabling integrations, I can try tracing it.

[hennevogel]

If there’s a specific component or job responsible for disabling integrations, I can try tracing it.

open-build-service/src/api/app/models/workflow_run.rb

Lines 71 to 85 in 4dc5753

	def update_as_failed(message)
	update(response_body: message, status: 'fail')
	#
	# Circuit breaker for authorization problems
	#
	# If message is one of these strings, we disable the token:
	#
	# "Failed to report back to GitLab: Unauthorized request. Please check your credentials again."
	# "Failed to report back to GitLab: Request forbidden."
	# "Failed to report back to GitHub: Unauthorized request. Please check your credentials again."
	# "Failed to report back to GitHub: Request is forbidden."
	token.update(enabled: false) if message.include?('Unauthorized request') || /Request (is )?forbidden/.match?(message)
	end

[dirkmueller]

so it happened again. all the deliveries were successful on both sides. build service says this:

https://build.opensuse.org/my/tokens/9212/workflow_runs/743095

and github says success:

yet the token was disabled right afterwards.

[perlpunk]

Yeah, it happened for us again as well (user os-autoinst-obs-workflow). A token was disabled.