fix(tokens): Add async flush outboxes #105264

Christinarlong · 2025-12-18T19:08:52Z

oh man this one's a doozy

Christinarlong · 2025-12-18T19:23:05Z

src/sentry/models/apitoken.py

+
+        # Schedule async replication if using async mode
+        if not self._should_flush_outbox():
+            transaction.on_commit(


ok I'm not actually sure if this does anything since we've already completed all the prev save steps? But I saw this being used for drain_shard and we'd want to enqueue the replication task after the token has committed 🤷‍♀️

Because you're scheduling a task, it is best to do that after the transaction has commit as you can ensure that all the records are saved. Without this it is possible for the task to be processed while the transaction has not complete if postgres is slow and kafka is fast.

src/sentry/models/apitoken.py

codecov · 2025-12-18T20:10:56Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
30761	1	30760	246

View the top 1 failed test(s) by shortest run time

tests.sentry.models.test_apitoken.ApiTokenTest::test_organization_id_for_non_internal

Stack Traces | 4.04s run time

#x1B[1m#x1B[.../sentry/models/test_apitoken.py#x1B[0m:69: in test_organization_id_for_non_internal
    assert ApiTokenReplica.objects.get(apitoken_id=token.id).organization_id is None
#x1B[1m#x1B[31mE   AssertionError: assert 4557306177650688 is None#x1B[0m
#x1B[1m#x1B[31mE    +  where 4557306177650688 = <ApiTokenReplica at 0x7f9ff8cfa0d0: id=28, user_id=535, token='4ea336dad930b4dcd424978cd06a3ae33c96e186039d73ab844623da67c11f9b', application_id=20>.organization_id#x1B[0m
#x1B[1m#x1B[31mE    +    where <ApiTokenReplica at 0x7f9ff8cfa0d0: id=28, user_id=535, token='4ea336dad930b4dcd424978cd06a3ae33c96e186039d73ab844623da67c11f9b', application_id=20> = <bound method QuerySet.get of <sentry.db.models.manager.base.BaseManager object at 0x7fa09dc6cc20>>(apitoken_id=28)#x1B[0m
#x1B[1m#x1B[31mE    +      where <bound method QuerySet.get of <sentry.db.models.manager.base.BaseManager object at 0x7fa09dc6cc20>> = <sentry.db.models.manager.base.BaseManager object at 0x7fa09dc6cc20>.get#x1B[0m
#x1B[1m#x1B[31mE    +        where <sentry.db.models.manager.base.BaseManager object at 0x7fa09dc6cc20> = ApiTokenReplica.objects#x1B[0m
#x1B[1m#x1B[31mE    +      and   28 = <ApiToken at 0x7f9ff8cef020: id=28, user_id=535, token='4ea336dad930b4dcd424978cd06a3ae33c96e186039d73ab844623da67c11f9b', application_id=20>.id#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

src/sentry/models/apitoken.py

GabeVillalobos · 2025-12-18T23:24:41Z

src/sentry/models/apitoken.py

+        has_async_flush = self.user_id in options.get("users:api-token-async-flush")
+        logger.info(
+            "async_flush_check",
+            extra={
+                "has_async_flush": has_async_flush,
+                "user_id": self.user_id,
+                "token_id": self.id,
+            },
+        )
+        if has_async_flush:
+            return False
+
+        return True


Discussed in person, but I don't think we need a user-level flag for this. I think a global option that sets the value of the class's default_flush property should be sufficient for us. Having an all-or-nothing toggle is probably okay for this change, and I don't expect the majority of users to even notice the impact of this change.

Setting class properties based on options will be tricky as classes are imported during django's startup before the ORM is ready. The current approach of overriding _maybe_prepare_outboxes will work though.

ended up doing some property decorator shenanigans since that should be loaded after the startup 🤔

GabeVillalobos · 2025-12-18T23:26:27Z

src/sentry/models/apitoken.py

+        drain_outbox_shards_control.delay(
+            outbox_identifier_low=first_row.id,
+            outbox_identifier_hi=last_row.id + 1,
+            outbox_name="sentry.ControlOutbox",
+        )


I wonder if we should move this entire function to the parent ControlOutbox or even Outbox class to trigger instead. Being able to enqueue a task to more quickly drain a shard seems like a useful feature in other contexts, beyond just this one.

We could add a new task that accepts the shard_scope, shard_identifier, category, and object_identifier as parameters, and that task could do the range query + call process_outbox_batch() 🤷

slapped it into the parent ControlOutboxProducingModel

markstory · 2025-12-30T21:32:10Z

src/sentry/models/apitoken.py

+
+        # Schedule async replication if using async mode
+        if not self._should_flush_outbox():
+            transaction.on_commit(


Because you're scheduling a task, it is best to do that after the transaction has commit as you can ensure that all the records are saved. Without this it is possible for the task to be processed while the transaction has not complete if postgres is slow and kafka is fast.

src/sentry/models/apitoken.py

markstory · 2025-12-30T21:41:11Z

src/sentry/models/apitoken.py

+        has_async_flush = self.user_id in options.get("users:api-token-async-flush")
+        logger.info(
+            "async_flush_check",
+            extra={
+                "has_async_flush": has_async_flush,
+                "user_id": self.user_id,
+                "token_id": self.id,
+            },
+        )
+        if has_async_flush:
+            return False
+
+        return True


Setting class properties based on options will be tricky as classes are imported during django's startup before the ORM is ready. The current approach of overriding _maybe_prepare_outboxes will work though.

src/sentry/models/apitoken.py

markstory · 2025-12-30T21:51:55Z

src/sentry/models/apitoken.py

+        drain_outbox_shards_control.delay(
+            outbox_identifier_low=first_row.id,
+            outbox_identifier_hi=last_row.id + 1,
+            outbox_name="sentry.ControlOutbox",
+        )


We could add a new task that accepts the shard_scope, shard_identifier, category, and object_identifier as parameters, and that task could do the range query + call process_outbox_batch() 🤷

tests/sentry/models/test_apitoken.py

src/sentry/hybridcloud/outbox/base.py

src/sentry/models/apitoken.py

markstory

Looks correct to me.

add async flush outboxes 💀

318b823

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Dec 18, 2025

Merge branch 'master' into crl/async-outbox

65d33af

vercel bot deployed to Preview December 18, 2025 19:12 View deployment

typo

8ce458c

Christinarlong commented Dec 18, 2025

View reviewed changes

vercel bot deployed to Preview December 18, 2025 19:36 View deployment

Christinarlong marked this pull request as ready for review December 18, 2025 19:38

Christinarlong requested review from a team, GabeVillalobos, leeandher and markstory December 18, 2025 19:38

cursor bot reviewed Dec 18, 2025

View reviewed changes

src/sentry/models/apitoken.py Outdated Show resolved Hide resolved

This comment was marked as outdated.

Sign in to view

This comment was marked as duplicate.

Sign in to view

Christinarlong added 2 commits December 18, 2025 13:30

fix cursor comments

d160173

add outbox tests for updating

33ca112

vercel bot deployed to Preview December 18, 2025 21:56 View deployment

cursor bot reviewed Dec 18, 2025

View reviewed changes

src/sentry/models/apitoken.py Outdated Show resolved Hide resolved

GabeVillalobos reviewed Dec 18, 2025

View reviewed changes

markstory reviewed Dec 30, 2025

View reviewed changes

Christinarlong added 2 commits January 2, 2026 11:19

Merge branch 'master' into crl/async-outbox

19a637f

move async logic to base outbox class and fix tests

bcb1774

Christinarlong requested a review from a team as a code owner January 5, 2026 22:28

sentry bot reviewed Jan 5, 2026

View reviewed changes

src/sentry/hybridcloud/outbox/base.py Show resolved Hide resolved

vercel bot deployed to Preview January 5, 2026 22:31 View deployment

cursor bot reviewed Jan 5, 2026

View reviewed changes

src/sentry/hybridcloud/outbox/base.py Show resolved Hide resolved

Christinarlong added 2 commits January 5, 2026 14:55

Merge branch 'master' into crl/async-outbox

585863e

fix tests

4f1099b

default flush shenanigans

4ca899e

vercel bot deployed to Preview January 5, 2026 23:45 View deployment

cursor bot reviewed Jan 5, 2026

View reviewed changes

src/sentry/models/apitoken.py Outdated Show resolved Hide resolved

more pr bot fixes

168b891

vercel bot deployed to Preview January 6, 2026 00:08 View deployment

Christinarlong requested review from GabeVillalobos and markstory January 6, 2026 00:29

markstory approved these changes Jan 6, 2026

View reviewed changes

Christinarlong merged commit 39adb91 into master Jan 6, 2026
65 checks passed

Christinarlong deleted the crl/async-outbox branch January 6, 2026 22:14

Uh oh!

fix(tokens): Add async flush outboxes #105264

fix(tokens): Add async flush outboxes #105264

Conversation

Christinarlong commented Dec 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as duplicate.

Uh oh!

codecov bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markstory left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Dec 18, 2025 •

edited

Loading