feat(async): Add daily task to archive stale Slack channels

[rgibert]

Add a django-q2 scheduled task that runs daily to find bot-created Slack channels where the workspace retention policy has deleted all message history. For each stale channel, posts an archival notice and archives the channel.

The task queries ExternalLink records (type=SLACK) to find channels Firetower created, checks each via conversations_info and conversations_history(limit=1), and archives any channel with zero remaining messages. Per-channel errors are isolated so one failure doesn't abort the run. If the Slack client is unavailable (no bot token), the task disables its own schedule.

SlackService changes: added archive_channel(), a limit parameter to get_channel_history (avoids paginating all messages when we only need to check existence), and is_archived to get_channel_info.

Resolves RELENG-20

Agent transcript: https://claudescope.sentry.dev/share/onjkTdIu9bD3cgBvGExA2meMCtHFG8S8cpHdLO1DEk8

[linear-code]

RELENG-20

[sentry]

Bug: The paginated path in get_channel_history swallows exceptions and returns [], causing the archive_stale_channels task to incorrectly archive channels on API failure.
_{Severity: HIGH}

Suggested Fix

The try...except block in the paginated loop of get_channel_history should re-raise the exception after logging it. This will allow the calling archive_stale_channels task to catch the exception and correctly skip the channel, preventing it from being archived due to a transient API error. The non-paginated path already propagates exceptions, and the paginated path should behave consistently.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/firetower/integrations/services/slack.py#L464-L467

Potential issue: When `get_channel_history` is called without a `limit`, it uses a
paginated approach. This path contains a `try...except` block that catches all
exceptions from the `conversations_history` API call, logs them, and returns an empty
list of messages. The calling task, `archive_stale_channels`, interprets this empty list
as a sign of inactivity and proceeds to archive the channel. This is incorrect behavior,
as a transient API error (like a timeout or rate limit) could cause a recently active
channel to be archived. The intended behavior is for the exception to propagate so the
task can skip the channel instead of archiving it.

Also affects:

• src/firetower/incidents/tasks.py:134~134
• src/firetower/integrations/services/slack.py:503~506

[rgibert]

Fixed in 2862233. Both get_channel_history and get_thread_replies now re-raise exceptions and raise RuntimeError on not-ok responses instead of silently returning empty results. The archive task's per-channel except Exception handler correctly skips the channel.

— Claude Code

[cursor]

Archiver exceeds worker timeout

Medium Severity

The daily archiver sleeps two seconds before each Slack link after the first, while Q_CLUSTER uses a 180-second task timeout. With dozens of terminal-incident Slack ExternalLink rows, sleep alone can exceed the limit, so django-q may terminate the run before every channel is scanned.

 

^{Reviewed by Cursor Bugbot for commit 3968031. Configure here.}

[rgibert]

Acknowledged. The 2s delay is intentional to stay under Slack's Tier 3 rate limit (~50 req/min). The task is idempotent -- if it times out, the next daily run picks up where it left off since already-archived channels are skipped via is_archived check. If channel volume grows enough to make this a real problem, we can batch into smaller chunks or move to an async queue, but that's premature for the current scale.

— Claude Code

[cursor]

System messages block stale archival

Medium Severity

The stale-channel check treats any history item whose bot_id is not Firetower’s as blocking archival. Slack system events (e.g. channel_join) usually have no bot_id, so they are counted as foreign activity and the channel is skipped even when only Firetower bot posts remain besides those events.

 

^{Reviewed by Cursor Bugbot for commit 033c52d. Configure here.}

[rgibert]

Intentional. System messages (channel_join, channel_purpose, etc.) lack bot_id and are treated as non-own activity. This is the safer direction -- we'd rather skip a channel than wrongly archive one with real activity obscured by system events. If this causes too many false-skips in practice, we can add a subtype filter in a follow-up.

— Claude Code

[sentry]

Bug: The bot_id property does not cache None results, leading to repeated API calls if the initial lookup fails or returns no ID.
_{Severity: LOW}

Suggested Fix

Modify the bot_id property to explicitly set self._bot_id to a sentinel value or None within the try...except block and after a successful API call, even if the result is None. This ensures that failed lookups are cached and not retried.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/firetower/integrations/services/slack.py#L71-L72

Potential issue: The `bot_id` property in the `SlackService` does not cache `None`
results. If the `auth.test` API call fails or returns a response without a `bot_id`, the
property returns `None` but does not update the `self._bot_id` cache attribute.
Consequently, subsequent accesses to the `bot_id` property will re-trigger the same API
call instead of returning a cached `None` value. This is inefficient for long-lived
`SlackService` instances, as it results in repeated, unnecessary network requests upon
failure.

[rgibert]

Valid observation but low impact here. The archive task early-returns when bot_id is None (archive.py:36-38), so subsequent access never happens. The SlackService is also short-lived per task invocation. Can address with a sentinel pattern in a follow-up if needed.

— Claude Code

[sentry-warden]

Transient Slack client failure permanently disables the archive schedule

If slack.client is falsy on any single run (e.g. during a token rotation or config drift), the task sets repeats=0 on the schedule and exits — there is no code path that ever re-enables it. The daily job is permanently killed by a single transient unavailability.

Evidence

• archive_stale_channels sets Schedule.objects.filter(name='archive_stale_channels').update(repeats=0) whenever slack.client is falsy and returns immediately.
• SCHEDULES in __init__.py sets repeats: -1 only at initial registration; no migration, signal, or task re-sets it back after a failure.
• SlackService.__init__ sets self.client = None when BOT_TOKEN is absent from settings, which can happen transiently during deploys or secret rotations.
• test_disables_schedule_when_no_client confirms the repeats=0 write but has no corresponding test for re-enabling the schedule.

_{Identified by Warden find-bugs}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f36375b. Configure here.}

[cursor]

Thread bot replies ignored

Medium Severity

When deciding if a channel is stale, top-level history treats any message whose bot_id is not Firetower’s as activity, but thread follow-up uses get_thread_replies, which drops replies that have a bot_id. A channel whose visible history is only Firetower bot posts can still be archived even when another bot has active thread replies.

Additional Locations (1)

• src/firetower/integrations/services/slack.py#L588-L597

 

^{Reviewed by Cursor Bugbot for commit f36375b. Configure here.}

[rgibert]

Intentional asymmetry. The top-level check is conservative: any non-Firetower message (including other bots) counts as activity to avoid wrongful archival. The thread check uses get_thread_replies which filters to human-only replies -- its purpose is to detect human discussion in threads, not bot-to-bot chatter. A channel with only Firetower top-level messages and only other-bot thread replies has no human activity and is a valid archive candidate.

— Claude Code

[sentry]

Bug: A SlackApiError during channel archival is logged twice: once in the archive_channel function and again in the calling task, leading to redundant log entries.
_{Severity: LOW}

Suggested Fix

Remove the redundant logging. Either the archive_channel function should not log the error and only return False, letting the caller handle all logging, or the caller should not log the exception again after archive_channel has already logged it.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/firetower/incidents/tasks/archive.py#L118-L122

Potential issue: When the `slack.archive_channel` function fails with a `SlackApiError`,
it logs the error and returns `False`. The calling code in `archive.py` detects this
`False` return value, raises a `RuntimeError`, and then immediately catches that
exception in an outer block. This `except Exception` block then logs the exception a
second time, resulting in redundant logging for the same failure event. While this does
not affect functionality, it creates unnecessary noise in the logs.

Also affects:

• src/firetower/integrations/services/slack.py:580~590