spans metric alert do not trigger

[xtyinbiao]

Self-Hosted Version

25.8.0

CPU Architecture

x86_64

Docker Version

I am use k8s

Docker Compose Version

i am use k8s

Machine Specification

• My system meets the minimum system requirements of Sentry

Steps to Reproduce

1. i installed sentry 25.8.0 from k8s, and create a span alert, and send test data, but the alert do not trigger
2. this is my alert

3. I see sentry-worker log, just fund this
   01:37:03 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:69dbefdb0c3b401bbd16cb11b06338ee:1' reason='missing_cache')
   01:37:03 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:0b01d45d195f4018b1a97c20c2c9c841:1' reason='missing_cache')
   01:37:03 [INFO] celery.app.trace: Task sentry.issues.tasks.post_process.post_process_group[cbb167be-ac32-4382-ba28-19b9ea0e7b69] succeeded in 0.01703406125307083s: None (data={'id': 'cbb167be-ac32-4382-ba28-19b9ea0e7b69', 'name': 'sentry.issues.tasks.post_process.post_process_group', 'return_value': 'None', 'runtime': 0.01703406125307083, 'args': '()', 'kwargs': "{'is_new': False, 'is_regression': False, 'is_new_group_environment': False, 'primary_hash': None, 'cache_key': 'e:69dbefdb0c3b401bbd16cb11b06338ee:1', 'group_id': None, 'group_states': None, 'occurrence_id': None, 'project_id': 1, 'eventstream_type': 'transaction'}"})
   01:37:03 [INFO] celery.app.trace: Task sentry.issues.tasks.post_process.post_process_group[ea72b836-b163-4d9d-b08c-f415c42e19cc] succeeded in 0.01720268279314041s: None (data={'id': 'ea72b836-b163-4d9d-b08c-f415c42e19cc', 'name': 'sentry.issues.tasks.post_process.post_process_group', 'return_value': 'None', 'runtime': 0.01720268279314041, 'args': '()', 'kwargs': "{'is_new': False, 'is_regression': False, 'is_new_group_environment': False, 'primary_hash': None, 'cache_key': 'e:0b01d45d195f4018b1a97c20c2c9c841:1', 'group_id': None, 'group_states': None, 'occurrence_id': None, 'project_id': 1, 'eventstream_type': 'transaction'}"})
   01:37:05 [INFO] celery.worker.strategy: Task sentry.tasks.options.sync_options_control[71c2c2c5-e7fa-4b94-8a6c-80d996e62308] received (data={'id': '71c2c2c5-e7fa-4b94-8a6c-80d996e62308', 'name': 'sentry.tasks.options.sync_options_control', 'args': None, 'kwargs': None, 'eta': None})
   01:37:05 [DEBUG] celery.pool: TaskPool: Apply <function fast_trace_task at 0x7f81320a6de0> (args:('sentry.tasks.options.sync_options_control', '71c2c2c5-e7fa-4b94-8a6c-80d996e62308', {'lang': None, 'task': 'sentry.tasks.options.sync_options_control', 'id': '71c2c2c5-e7fa-4b94-8a6c-80d996e62308', 'root_id': None, 'parent_id': None, 'group': None, 'meth': None, 'shadow': None, 'eta': None, 'expires': '2025-11-18T01:37:15.326043+00:00', 'retries': 0, 'timelimit': (None, None), 'argsrepr': None, 'kwargsrepr': None, 'origin': None, 'sentry-task-enqueued-time': 1763429825.3257473, 'sentry-trace': 'b2a3ed28f4a54fe9a24f310c03f071a5-b182465495649562', 'baggage': 'sentry-trace_id=b2a3ed28f4a54fe9a24f310c03f071a5,sentry-environment=production,sentry-release=backend%4025.8.0%2Bad3ec8beb30522968010d8899818c9ea3a6c53f0,', 'sentry-monitor-start-timestamp-s': '1763429825.325744867', 'headers': {'sentry-task-enqueued-time': 1763429825.3257473, 'sentry-trace': 'b2a3ed28f4a54fe9a24f310c03f071a5-b182465495649562', 'baggage':... kwargs:{})
   01:37:05 [DEBUG] celery.worker.request: Task accepted: sentry.tasks.options.sync_options_control[71c2c2c5-e7fa-4b94-8a6c-80d996e62308] pid:16
   Traceback (most recent call last):
   File "/usr/src/sentry/src/sentry/options/manager.py", line 224, in lookup_key
   return self.registry[key]
   ~~~~~~~~~~~~~^^^^^
   KeyError: 'sentry:last_worker_ping'
   01:37:05 [DEBUG] sentry: Using legacy key: sentry:last_worker_ping
   Traceback (most recent call last):
   File "/usr/src/sentry/src/sentry/options/manager.py", line 224, in lookup_key
   return self.registry[key]
   ~~~~~~~~~~~~~^^^^^
   KeyError: 'sentry:last_worker_version'
   01:37:05 [DEBUG] sentry: Using legacy key: sentry:last_worker_version
   01:37:05 [INFO] celery.app.trace: Task sentry.tasks.options.sync_options_control[71c2c2c5-e7fa-4b94-8a6c-80d996e62308] succeeded in 0.016090072691440582s: None (data={'id': '71c2c2c5-e7fa-4b94-8a6c-80d996e62308', 'name': 'sentry.tasks.options.sync_options_control', 'return_value': 'None', 'runtime': 0.016090072691440582, 'args': '()', 'kwargs': '{}'})
   01:37:06 [INFO] celery.worker.strategy: Task sentry.tasks.enqueue_outbox_jobs_control[fb41be1b-65c0-44c0-be1b-d2472c7cb87f] received (data={'id': 'fb41be1b-65c0-44c0-be1b-d2472c7cb87f', 'name': 'sentry.tasks.enqueue_outbox_jobs_control', 'args': None, 'kwargs': None, 'eta': None})
   01:37:06 [DEBUG] celery.pool: TaskPool: Apply <function fast_trace_task at 0x7f81320a6de0> (args:('sentry.tasks.enqueue_outbox_jobs_control', 'fb41be1b-65c0-44c0-be1b-d2472c7cb87f', {'lang': None, 'task': 'sentry.tasks.enqueue_outbox_jobs_control', 'id': 'fb41be1b-65c0-44c0-be1b-d2472c7cb87f', 'root_id': None, 'parent_id': None, 'group': None, 'meth': None, 'shadow': None, 'eta': None, 'expires': '2025-11-18T01:38:06.334593+00:00', 'retries': 0, 'timelimit': (None, None), 'argsrepr': None, 'kwargsrepr': None, 'origin': None, 'sentry-task-enqueued-time': 1763429826.3342466, 'sentry-trace': 'c4e6a10256244c80b1619b1c29e9e3d5-a88ea64cdd7b6c1b', 'baggage': 'sentry-trace_id=c4e6a10256244c80b1619b1c29e9e3d5,sentry-environment=production,sentry-release=backend%4025.8.0%2Bad3ec8beb30522968010d8899818c9ea3a6c53f0,', 'sentry-monitor-start-timestamp-s': '1763429826.334243774', 'headers': {'sentry-task-enqueued-time': 1763429826.3342466, 'sentry-trace': 'c4e6a10256244c80b1619b1c29e9e3d5-a88ea64cdd7b6c1b', 'baggage':... kwargs:{})
   01:37:06 [DEBUG] celery.worker.request: Task accepted: sentry.tasks.enqueue_outbox_jobs_control[fb41be1b-65c0-44c0-be1b-d2472c7cb87f] pid:16

4. I test the other alert is success

Expected Result

trigger alert

Actual Result

the alert nog trigger

Event ID

No response

[xtyinbiao]

throughput metric alert is ok, I don't know why the span alarm won't trigger.

Here are the features I enabled——

SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
{
feature: True
for feature in ("organizations:discover",
"organizations:global-views",
"organizations:incidents",
"organizations:integrations-issue-basic",
"organizations:integrations-issue-sync",
"organizations:invite-members",
"organizations:sso-basic",
"organizations:sso-saml2",
"organizations:advanced-search",
"organizations:issue-platform",
"organizations:monitors",
"organizations:dashboards-mep",
"organizations:mep-rollout-flag",
"organizations:dashboards-rh-widget",
"organizations:dynamic-sampling",
"projects:custom-inbound-filters",
"projects:data-forwarding",
"projects:discard-groups",
"projects:plugins",
"projects:rate-limits",
"projects:servicehooks",
)
+ (
# Performance/Tracing/Spans related flags
"organizations:performance-view",
"organizations:visibility-explore-view",
"organizations:transaction-metrics-extraction",
"organizations:indexed-spans-extraction",
"organizations:insights-entry-points",
"organizations:insights-initial-modules",
"organizations:insights-addon-modules",
"organizations:standalone-span-ingestion",
"organizations:starfish-mobile-appstart",
"projects:span-metrics-extraction",
"projects:span-metrics-extraction-addons",

        # flags added in this chart
        "organizations:trace-view-load-more",
        "organizations:trace-tabs-ui",
        "organizations:trace-view-linked-traces",
        "organizations:span-stats",
        "organizations:visibility-explore-range-high",
    )
    + (
        # Session Replay related flags
        "organizations:session-replay",
        
        # flags added in this chart
        "organizations:session-replay-ui",
        "organizations:session-replay-issue-emails",
        "organizations:session-replay-recording-scrubbing",
        "organizations:session-replay-slack-new-issue",
    )
    + (
        # Profiling related flags
        "organizations:profiling",
        "organizations:profiling-view",
        # Continuous Profiling related flags
        "organizations:continuous-profiling",
        "organizations:continuous-profiling-stats",
    )
    + (
        # Flags enabled in this chart but not present in https://github.com/getsentry/self-hosted/blob/master/sentry/sentry.conf.example.py
        "organizations:related-events",
        "organizations:reprocessing-v2",
        "organizations:set-grouping-config",
        "organizations:onboarding",
        "projects:similarity-indexing",
        "projects:similarity-view",
    )
    + (
        # Custom features from values
        "organizations:workflow-engine-metric-alert-processing",
        "organizations:custom-metric",
    )
}

)

[aldy505]

Hmm, is this similar with getsentry/sentry#101581 (comment) ?

[xtyinbiao]

Hmm, is this similar with getsentry/sentry#101581 (comment) ?

I've tried everything here and it doesn't work.
I found the clickhouse table : spans_local has no data. My guess is that spans data snuba is not written to clickhouse

i don't know why