fix(code-review): Consolidate code review checks #105561

suejung-sentry · 2025-12-31T05:34:43Z

Consolidate all the different checks for code review to one spot so we can use it in our new webhook flow.

Note that I left the billing check off for now and will check with the team on how best to transition it on for GA

Closes CW-88

src/sentry/seer/code_review/preflight.py

linear · 2025-12-31T05:38:33Z

CW-88 List features and related features for code review

armenzg

Well done! 🎉

Besides tests, could you write a ticket to help other parts of Sentry use your logic? (or see how easy it is to switch over) That way we have a centralized place and we prevent unintentional regressions.

src/sentry/seer/code_review/webhooks/check_run.py

src/sentry/seer/code_review/webhooks/handlers.py

src/sentry/seer/code_review/settings.py

src/sentry/seer/code_review/preflight.py

Until we officially freeze the code review beta list, consider anyone with the feature toggle ON as part of the beta too. Note that we'll replace this with the logic [here](#105561) shortly, but wanted to unblock this.

…checks

suejung-sentry · 2026-01-05T05:55:27Z

Besides tests, could you write a ticket to help other parts of Sentry use your logic? (or see how easy it is to switch over) That way we have a centralized place and we prevent unintentional regressions.

Here's a ticket CW-111. I think easiest is to do a hard cutover to this new flow path and the old one within overwatch can just die on the vine, but we can see how the timing works out.

fixtures/github.py

suejung-sentry · 2026-01-05T05:31:00Z

src/sentry/overwatch/endpoints/overwatch_rpc.py

-    except OrganizationContributors.DoesNotExist:
-        metrics.incr(
-            "overwatch.code_review.contributor_not_found",
-            tags={"organization_id": organization.id, "repository_id": repository_id},


Note that when I moved this over I removed the tag for repository id as I thought it would add too much cardinality on the metric. Will connect with @ajay-sentry if that is going to be a problem..

suejung-sentry · 2026-01-05T05:33:17Z

src/sentry/seer/code_review/webhooks/handlers.py

        organization: The Sentry organization that the webhook event belongs to
        repo: The repository that the webhook event is for
-        **kwargs: Additional keyword arguments including integration
+        integration: The GitHub integration


Looks like we need integration for the preflight billing check

suejung-sentry · 2026-01-05T05:34:01Z

src/sentry/seer/code_review/webhooks/handlers.py

+        pr_author_external_id=get_pr_author_id(event),
+    ).check()
+    if not preflight.allowed:
+        # TODO: add metric


Will add a metric consistent with any new standards in this PR - https://github.com/getsentry/sentry/pull/105587/files

suejung-sentry · 2026-01-05T05:36:44Z

tests/sentry/seer/code_review/test_webhooks.py


-    @patch("sentry.seer.code_review.webhooks.task.process_github_webhook_event")
-    @with_feature({"organizations:gen-ai-features"})
-    def test_check_run_runs_when_code_review_beta_flag_disabled_but_pr_review_test_generation_enabled(


this testcase is covered in the preflight class tests

suejung-sentry · 2026-01-05T06:13:12Z

src/sentry/seer/code_review/utils.py

    }
+
+
+def get_pr_author_id(event: Mapping[str, Any]) -> str | None:


moved from https://github.com/codecov/overwatch/blob/11ff522b04eda10cd7fc2d8a66b9b38e3cd241c2/app/app/Services/SeerService.php#L609

armenzg

Good from my POV. Feel free to merge.
You may want input from someone familiar with billing for the tests you added.

armenzg · 2026-01-05T20:29:06Z

src/sentry/seer/code_review/webhooks/handlers.py

+        integration_id=integration.id if integration else None,
+        pr_author_external_id=get_pr_author_id(event),
+    ).check()
+    if not preflight.allowed:


Great to see it being handled in the main handler function 🦾 .

…checks

suejung-sentry · 2026-01-06T00:42:14Z

Good from my POV. Feel free to merge. You may want input from someone familiar with billing for the tests you added.

Thanks, I'll connect with @ajay-sentry when he's back on Wednesday. The billing checks aren't live until we GA so should be OK to merge in the meantime

src/sentry/seer/code_review/webhooks/handlers.py

src/sentry/seer/code_review/preflight.py

…checks

suejung-sentry · 2026-01-06T01:46:56Z

src/sentry/seer/code_review/preflight.py

+    def _check_billing(self) -> DenialReason:
+        return None
+
+        # TODO: Once we're ready to actually gate billing (when it's time for GA), uncomment this


Note that I updated this to always let the billing check go through for now since it's not turned on for the overwatch flow yet either (just logged here). This will be turned on with GA.

src/sentry/seer/code_review/utils.py

suejung-sentry · 2026-01-06T01:52:24Z

src/sentry/seer/code_review/utils.py

+    if (user_id := event.get("user", {}).get("id")) is not None:
+        return str(user_id)
+
+    # Check sender.id (for check_run events). Sender is the user who triggered the event


I'll confirm with @ajay-sentry / Product who/what we want to increment for this webhook event. It's possible they would want this one to be "always free"

src/sentry/seer/code_review/webhooks/handlers.py

armenzg

Some of this code I did not merge into my PR since it did not seem necessary. Please let me know if I got it wrong!

armenzg · 2026-01-06T14:20:18Z

tests/sentry/seer/code_review/test_webhooks.py

+    github_integration: Integration | None = None
+
+    @pytest.fixture(autouse=True)
+    def mock_billing_quota(self) -> Generator[None]:


When I merged this code into #105673 I did not port this fixture.

armenzg · 2026-01-06T14:20:35Z

tests/sentry/seer/code_review/test_webhooks.py

        self.organization.update_option("sentry:enable_pr_review_test_generation", True)

+        # Setup billing data
+        self.github_integration = self.create_github_integration()


I did not merge this code.

armenzg · 2026-01-06T14:21:01Z

tests/sentry/seer/code_review/test_webhooks.py

        self.organization.update_option("sentry:enable_pr_review_test_generation", True)
+
+        # Setup billing data
+        self.github_integration = self.create_github_integration()


I did not merge this.

armenzg · 2026-01-06T14:21:07Z

tests/sentry/seer/code_review/test_webhooks.py

        repo_id = int(self.event_dict["repository"]["id"])

-        integration = self.create_github_integration()
+        integration = self.github_integration or self.create_github_integration()


armenzg · 2026-01-06T14:22:05Z

tests/sentry/seer/code_review/test_webhooks.py

            self.mock_seer.assert_not_called()

-    @with_feature({"organizations:gen-ai-features"})
    def test_runs_when_code_review_beta_flag_disabled_but_pr_review_test_generation_enabled(


Merged as this:

sentry/tests/sentry/seer/code_review/webhooks/test_issue_comment.py

Lines 100 to 110 in 463c021

def test_runs_when_code_review_beta_flag_disabled_but_pr_review_test_generation_enabled(

self,

) -> None:

"""Test that processing runs with gen-ai-features flag alone when org option is enabled."""

with self.code_review_setup(features={"organizations:gen-ai-features"}), self.tasks():

event = self._build_issue_comment_event(f"Please {SENTRY_REVIEW_COMMAND} this PR")

response = self._send_issue_comment_event(event)

assert response.status_code == 204

self.mock_seer.assert_called_once()

fix(code-review): Consolidate code review checks

05b44fd

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Dec 31, 2025

suejung-sentry commented Dec 31, 2025

View reviewed changes

src/sentry/seer/code_review/preflight.py Show resolved Hide resolved

vercel bot deployed to Preview December 31, 2025 05:36 View deployment

armenzg reviewed Dec 31, 2025

View reviewed changes

suejung-sentry mentioned this pull request Jan 2, 2026

fix(code-review): Look up code review beta orgs dynamically #105603

Merged

suejung-sentry added 2 commits January 2, 2026 16:04

Merge remote-tracking branch 'origin/master' into sshin/consolidated-…

17b7e2a

…checks

incorporate pr comments

66e863e

vercel bot deployed to Preview January 5, 2026 04:46 View deployment

vercel bot deployed to Preview January 5, 2026 05:27 View deployment

fix tests

9a8da6d

suejung-sentry force-pushed the sshin/consolidated-checks branch from d9e3314 to 9a8da6d Compare January 5, 2026 05:39

vercel bot deployed to Preview January 5, 2026 05:42 View deployment

suejung-sentry commented Jan 5, 2026

View reviewed changes

suejung-sentry marked this pull request as ready for review January 5, 2026 06:02

suejung-sentry requested a review from a team as a code owner January 5, 2026 06:02

suejung-sentry requested review from armenzg and removed request for a team January 5, 2026 06:02

suejung-sentry commented Jan 5, 2026

View reviewed changes

armenzg approved these changes Jan 5, 2026

View reviewed changes

suejung-sentry marked this pull request as draft January 5, 2026 22:59

Merge remote-tracking branch 'origin/master' into sshin/consolidated-…

6fd4aff

…checks

vercel bot deployed to Preview January 5, 2026 23:43 View deployment

vercel bot deployed to Preview January 5, 2026 23:54 View deployment

fix merge conflicts

d46d0e1

suejung-sentry force-pushed the sshin/consolidated-checks branch from 44295f1 to d46d0e1 Compare January 6, 2026 00:37

vercel bot deployed to Preview January 6, 2026 00:40 View deployment

suejung-sentry marked this pull request as ready for review January 6, 2026 01:17

sentry bot reviewed Jan 6, 2026

View reviewed changes

src/sentry/seer/code_review/webhooks/handlers.py Show resolved Hide resolved

src/sentry/seer/code_review/preflight.py Show resolved Hide resolved

Merge remote-tracking branch 'origin/master' into sshin/consolidated-…

1a69e22

…checks

vercel bot deployed to Preview January 6, 2026 01:30 View deployment

suejung-sentry commented Jan 6, 2026

View reviewed changes

vercel bot deployed to Preview January 6, 2026 01:48 View deployment

suejung-sentry force-pushed the sshin/consolidated-checks branch from de5360b to 67c1c83 Compare January 6, 2026 01:50

sentry bot reviewed Jan 6, 2026

View reviewed changes

src/sentry/seer/code_review/utils.py Show resolved Hide resolved

suejung-sentry commented Jan 6, 2026

View reviewed changes

vercel bot deployed to Preview January 6, 2026 01:53 View deployment

suejung-sentry force-pushed the sshin/consolidated-checks branch from 67c1c83 to ddea878 Compare January 6, 2026 02:01

sentry bot reviewed Jan 6, 2026

View reviewed changes

src/sentry/seer/code_review/webhooks/handlers.py Show resolved Hide resolved

vercel bot deployed to Preview January 6, 2026 02:04 View deployment

fix check run billing gate

3de8ad8

suejung-sentry force-pushed the sshin/consolidated-checks branch from ddea878 to 3de8ad8 Compare January 6, 2026 02:12

vercel bot deployed to Preview January 6, 2026 02:15 View deployment

suejung-sentry enabled auto-merge (squash) January 6, 2026 02:17

suejung-sentry merged commit d8d46a2 into master Jan 6, 2026
65 checks passed

suejung-sentry deleted the sshin/consolidated-checks branch January 6, 2026 02:33

armenzg reviewed Jan 6, 2026

View reviewed changes

github-actions bot mentioned this pull request Jan 10, 2026

ci(release): Switch from action-prepare-release to Craft #106005

Open

		}


		def get_pr_author_id(event: Mapping[str, Any]) -> str \| None:

	def test_runs_when_code_review_beta_flag_disabled_but_pr_review_test_generation_enabled(
	self,
	) -> None:
	"""Test that processing runs with gen-ai-features flag alone when org option is enabled."""
	with self.code_review_setup(features={"organizations:gen-ai-features"}), self.tasks():
	event = self._build_issue_comment_event(f"Please {SENTRY_REVIEW_COMMAND} this PR")

	response = self._send_issue_comment_event(event)
	assert response.status_code == 204

	self.mock_seer.assert_called_once()

Uh oh!

fix(code-review): Consolidate code review checks #105561

fix(code-review): Consolidate code review checks #105561

Uh oh!

Conversation

suejung-sentry commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

linear bot commented Dec 31, 2025

Uh oh!

armenzg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

suejung-sentry commented Jan 5, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

armenzg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suejung-sentry commented Jan 6, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

armenzg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

suejung-sentry commented Dec 31, 2025 •

edited

Loading