Issue #4425 : Use a standard 'killed' error with cause when onMaxAttempts: 'kill' exhausts retries #4425

[neerajchowdary889]

Use a standard 'killed' error with cause when onMaxAttempts: 'kill' exhausts retries #4425

issue: #4425

Problem

When an invocation is killed — either via the Admin API or after exhausting all retries
with onMaxAttempts: 'kill' — no journal event was written to sys_journal_events.

This caused two problems:

1. Admin API kill: no event at all, so the UI had no way to show the invocation was
   deliberately killed.
2. Kill-after-max-retries: the final error stored was the raw service error (e.g.
   HTTP 500). There was no way to distinguish a genuine failure from a kill.

Closes #4425

Solution

Add a Killed journal event type written in both kill paths.

• Admin API kill (kill_invoked_invocation, kill_suspended_or_paused_invocation):
  writes KilledEvent { last_failure: None } after the invocation is ended.
• Kill-after-max-retries (new KilledAfterMaxAttempts invoker effect): the invoker
  packages the last transient error into a KilledEvent { last_failure: Some(...) } and
  sends it as a new effect kind. The partition processor writes it after end_invocation.
  The final invocation result is now always KILLED_INVOCATION_ERROR (code ABORTED,
  message "killed") instead of the raw service error.

The event is always written after end_invocation returns, so it survives the
do_drop_journal cleanup that runs inside end_invocation.

Changes

File	Change
crates/types/protobuf/restate/journal_events.proto	Add KilledEvent proto message
crates/types/src/journal_events/mod.rs	Add Killed = 3 to EventType, Killed(KilledEvent) to Event, KilledEvent struct
crates/types/src/journal_events/raw.rs	Encode/decode and proto conversions for KilledEvent
crates/invoker-api/src/effects.rs	Add KilledAfterMaxAttempts { killed_event: RawEvent } to EffectKind
crates/invoker-impl/src/lib.rs	Send KilledAfterMaxAttempts instead of Failed on OnTaskError::Kill
crates/worker/src/partition/state_machine/mod.rs	Handle KilledAfterMaxAttempts effect; write KilledEvent in both kill functions
crates/storage-query-datafusion/src/journal_events/schema.rs	Document Killed as a valid event_type value
crates/worker/.../tests/kill_cancel.rs	Two new tests covering both kill paths
release-notes/unreleased/4425-killed-journal-event.md	Release note

Querying killed invocations

SELECT id, event_json
FROM sys_journal_events
WHERE event_type = 'Killed'

The event_json column contains:

{ "ty": "Killed", "last_failure": { "error_code": 500, "error_message": "..." } }

last_failure is absent when the invocation was killed via the Admin API.

Breaking change

For onMaxAttempts: 'kill' invocations, the final stored error changes from the raw
service error to KILLED_INVOCATION_ERROR (code ABORTED, message "killed"). Code
that inspects the error payload to detect kills should migrate to querying
sys_journal_events for event_type = 'Killed'.

Testing

cargo nextest run -p restate-worker kill_cancel
cargo nextest run -p restate-invoker-impl

All 11 kill/cancel tests pass, all 31 invoker-impl tests pass.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.