Skip to content

Commit c9f500e

Browse files
committed
feat(events): let a workflow log events without streaming them
A workflow can now opt out of live SSE while keeping a full event record, read back at GET /v2/workflows/{name}/instances/{id}/events. The default is unchanged and remains both: stream and log. public NotificationPolicy Notifications { get; } = new() { Delivery = EventDeliveryMode.LogOnly }; EventEnvelope.Transient becomes EventEnvelope.Delivery, an EventDeliveryMode of StreamAndLog, LogOnly or StreamOnly. A pair of booleans would have made "neither" representable, and neither is not a destination. StreamOnly is what Transient meant; the sinks now take the subset of each batch that named them. Envelopes also carry WorkflowName, denormalised onto the row so the log can be filtered by workflow without joining back to the instance, with a matching column and index on WorkflowEvents. Two consequences the design takes a position on: The SSE endpoint returns 409 for a log-only workflow, with a problem detail naming the v2 route, rather than holding open a stream it will never write to. An empty stream is indistinguishable from a stalled run, and a client waiting on one has no way to find out. A stream-only event under a log-only workflow goes nowhere at all. That is the correct reading — opting out of streaming opts out of streamed tokens — so StreamDeltas plus LogOnly produces no deltas anywhere, and the wiki says so where someone will look for it. StreamOnly as a workflow-level choice is rejected at startup: it would leave the run with no durable record, which is a different feature from the one being asked for and almost certainly a mistake. The v2 route is scoped by workflow name like the instance-state route, so a caller states which workflow they believe they are reading and a mismatch is a 404 rather than a silent success. Payloads are re-emitted as JSON values instead of escaped strings, so nobody parses them twice. Tests: 681 unit, 141 integration, 7 chaos, 21 broker.
1 parent 14b1b2e commit c9f500e

11 files changed

Lines changed: 560 additions & 38 deletions

File tree

README.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,7 @@ Policies are keyed by workflow **version**, since executor ids and gates change
284284
- `GET /instances/{id}/checkpoints`
285285
- `GET /instances/{id}/events/history`
286286
- `GET /instances/{id}/events`
287+
- `GET /v2/workflows/{name}/instances/{id}/events`
287288
- `POST /instances/{id}/cancel`
288289
- `POST /instances/{id}/retry`
289290
- `POST /instances/{id}/rerun`
@@ -351,6 +352,26 @@ A definition controls what its runs emit by implementing `INotifyingWorkflow`
351352
for the catalog API. Terminal events are never suppressible, and filtering happens before a sequence
352353
number is taken, so the gapless sequence that `Last-Event-ID` catch-up depends on stays intact.
353354

355+
The same policy decides *where* events go. The default is both the durable log and the live stream; a
356+
workflow nobody watches as it happens can keep the record and drop the stream:
357+
358+
```csharp
359+
public NotificationPolicy Notifications { get; } = new()
360+
{
361+
Delivery = EventDeliveryMode.LogOnly
362+
};
363+
```
364+
365+
| Mode | Durable log | Live stream |
366+
| --- | --- | --- |
367+
| `StreamAndLog` *(default)* | Yes | Yes |
368+
| `LogOnly` | Yes | No |
369+
370+
A log-only run stays fully observable — events are still sequenced, redacted, and carry the workflow
371+
name — and are read at `GET /v2/workflows/{name}/instances/{id}/events`. Its SSE endpoint returns
372+
`409` pointing at that route rather than holding open a stream that will never produce anything,
373+
because an empty stream is indistinguishable from a stalled run.
374+
354375
An `LlmExecutor` emits one `llm.completed` per call carrying model, prompt version, tokens, cost,
355376
latency and finish reason. Streamed tokens (`llm.delta`, opt-in per node via `StreamDeltas`) are
356377
**transient**: fanned out live, never stored, and written without an SSE `id:`, so a reconnecting

docs/wiki.md

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -751,6 +751,32 @@ public sealed class BulkWorkflow : IWorkflowDefinition<Ctx, Result>, INotifyingW
751751

752752
`ByNode` works in both directions: it can quiet one node in a `Standard` workflow or keep one node loud in a `Minimal` one.
753753

754+
### Turning SSE off for a workflow
755+
756+
Level controls *what* is emitted. `Delivery` controls *where it goes*, and the default is both the durable log and the live stream:
757+
758+
```csharp
759+
public NotificationPolicy Notifications { get; } = new()
760+
{
761+
Delivery = EventDeliveryMode.LogOnly // record everything; stream nothing
762+
};
763+
```
764+
765+
| Mode | Durable log | Live stream |
766+
| --- | --- | --- |
767+
| `StreamAndLog` *(default)* | Yes | Yes |
768+
| `LogOnly` | Yes | No |
769+
| `StreamOnly` | No | Yes — individual events only, not a workflow-level choice |
770+
771+
`LogOnly` suits a run nobody watches as it happens — a nightly batch, or work whose events are read afterwards for reconciliation. The run stays fully observable; only the timing changes. Events are still sequenced, still redacted, still carry the workflow name, and are read at `GET /v2/workflows/{name}/instances/{id}/events`.
772+
773+
Two consequences worth knowing:
774+
775+
- **The SSE endpoint refuses.** `GET /instances/{id}/events` returns `409` with a problem detail pointing at the v2 route, rather than holding a stream open that will never produce anything. An empty stream is indistinguishable from a stalled run, and a client waiting on one has no way to tell.
776+
- **Streamed tokens disappear entirely.** `llm.delta` is stream-only and is never written to the log, so a log-only workflow produces none. That is the correct reading — a workflow that has opted out of streaming has opted out of streamed tokens too — but it does mean `StreamDeltas` and `LogOnly` together is a contradiction the runtime resolves in favour of `LogOnly`.
777+
778+
`StreamOnly` is rejected at startup as a workflow-level choice: it would leave the run with no durable record at all, which is a different feature from the one being asked for and almost certainly a mistake.
779+
754780
Two rules keep the policy safe. **Terminal events are never suppressible** — a subscriber's stream closes on `workflow.terminated`, and `approval.*`, `instance.*` and `event.*` are control-plane and broker facts rather than run chatter. And **filtering happens before the sequence number is taken**: a suppressed event that had consumed one would leave a hole in the gapless sequence, and `Last-Event-ID` catch-up would wait forever for an event that is never coming.
755781

756782
Declared `Emits` names are validated at startup and surfaced on `GET /workflows/{name}`, so a consumer discovers the vocabulary rather than reverse-engineering it.
@@ -934,8 +960,38 @@ Unknown workflow, version, or executor returns `404`. A raw node or an unsupport
934960
| `GET` | `/instances/{id}/checkpoints` | Inspect checkpoint metadata |
935961
| `GET` | `/instances/{id}/events/history` | Read persisted event history |
936962
| `GET` | `/instances/{id}/events` | Subscribe to live SSE events |
963+
| `GET` | `/v2/workflows/{name}/instances/{id}/events` | The event log, scoped and attributed by workflow |
937964
| `GET` | `/workflows/{name}/instances/{id}/state` | Lifecycle status plus the workflow's own audit record |
938965

966+
`/v2/workflows/{name}/instances/{id}/events` is the read path for a workflow configured
967+
[log-only](#turning-sse-off-for-a-workflow), and equally valid for a streaming one — the same rows
968+
either way. Like the instance-state route it is scoped by workflow name, so a caller states which
969+
workflow they believe they are reading and a mismatch returns `404` rather than being silently
970+
accepted. `from`, `to`, `limit` and `types` all apply.
971+
972+
The response carries the instance and workflow context once, and each row repeats `instanceId` and
973+
`workflowName` so events collected across several instances keep their attribution:
974+
975+
```json
976+
{
977+
"instanceId": "01J...", "workflowName": "nightly-reconcile",
978+
"workflowVersion": "1.0.0", "tenantId": "acme",
979+
"total": 12, "nextCursor": null,
980+
"items": [
981+
{ "instanceId": "01J...", "workflowName": "nightly-reconcile", "sequence": 7,
982+
"eventType": "custom.batch.processed", "executorId": "reconcile", "superstep": 2,
983+
"payloadJson": { "rows": 500 }, "occurredAt": "2026-08-17T09:00:00Z" }
984+
]
985+
}
986+
```
987+
988+
Payloads are re-emitted as JSON values rather than escaped strings, so a caller reads them directly
989+
instead of parsing twice.
990+
991+
`GET /instances/{id}/events` returns `409` for a log-only workflow, with a problem detail pointing at
992+
the v2 route. Holding a stream open that will never produce anything is worse than refusing: a client
993+
cannot tell it apart from a stalled run.
994+
939995
`/workflows/{name}/instances/{id}/state` is the one instance route scoped by workflow name, because
940996
what it returns is shaped by that workflow's declaration. A mismatched name is a wrong URL rather
941997
than a different resource, so it returns `404` rather than the instance. `audit` is `null` when the
@@ -1455,7 +1511,13 @@ A wait with no `timeout` waits forever by design. Set one, with `WaitExpiryActio
14551511

14561512
### `llm.delta` events are missing from event history
14571513

1458-
They are not stored, by design. Streamed tokens are transient: live fan-out only, no sequence number, no durable row. Subscribe to `GET /instances/{id}/events` to see them; `GET /instances/{id}/events/history` will never return them.
1514+
They are not stored, by design. Streamed tokens are stream-only: live fan-out, no sequence number, no durable row. Subscribe to `GET /instances/{id}/events` to see them; `GET /instances/{id}/events/history` will never return them.
1515+
1516+
If none arrive on the live stream either, check whether the workflow declares `Delivery = LogOnly`. A stream-only event under a log-only workflow goes nowhere at all — which is the intended reading, not a bug, but it does make `StreamDeltas` and `LogOnly` a combination that produces no tokens anywhere.
1517+
1518+
### The SSE endpoint returns 409
1519+
1520+
The workflow declares `Delivery = EventDeliveryMode.LogOnly`, so it never streams. Read its events at `GET /v2/workflows/{name}/instances/{id}/events` instead — the problem detail carries the exact URL. This is deliberate: an empty stream held open is indistinguishable from a stalled run, so the endpoint refuses rather than misleading a client into waiting.
14591521

14601522
Also confirm `LlmOptions.StreamDeltas` is set on that node — it defaults to `false`, so no deltas are produced at all unless the definition asked for them.
14611523

src/Abacus.Run.Service/Infrastructure/SqlServerStores.cs

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,10 @@ protected override void OnModelCreating(ModelBuilder modelBuilder)
4040
entity.HasKey(row => new { row.InstanceId, row.Sequence });
4141
entity.Property(row => row.Payload).IsRequired();
4242
entity.HasIndex(row => row.EventType);
43+
44+
// Supports reading a workflow's log across instances, which is the query the
45+
// denormalised column exists for.
46+
entity.HasIndex(row => new { row.WorkflowName, row.OccurredAt });
4347
});
4448

4549
modelBuilder.Entity<JsonRow>(entity =>
@@ -72,6 +76,12 @@ public sealed class EventRow
7276
public required string InstanceId { get; set; }
7377
public required long Sequence { get; set; }
7478
public required string EventType { get; set; }
79+
80+
/// <summary>
81+
/// Denormalised from the instance so the log can be filtered by workflow without a join. Nullable
82+
/// because rows written before this column existed do not have it.
83+
/// </summary>
84+
public string? WorkflowName { get; set; }
7585
public string? TenantId { get; set; }
7686
public string? ExecutorId { get; set; }
7787
public int? Superstep { get; set; }
@@ -279,6 +289,7 @@ public async ValueTask AppendBatchAsync(IReadOnlyList<EventEnvelope> events, Can
279289
InstanceId = envelope.InstanceId,
280290
Sequence = envelope.Sequence,
281291
EventType = envelope.EventType,
292+
WorkflowName = envelope.WorkflowName,
282293
TenantId = envelope.TenantId,
283294
ExecutorId = envelope.ExecutorId,
284295
Superstep = envelope.Superstep,

src/Abacus.Run/Abstractions/Instances.cs

Lines changed: 34 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -103,17 +103,44 @@ public sealed record EventEnvelope
103103
public required DateTimeOffset OccurredAt { get; init; }
104104

105105
/// <summary>
106-
/// Fanned out to live subscribers but never appended to the durable store, and carrying no
107-
/// sequence number.
106+
/// The workflow this instance is running. Denormalised onto the event so the log can be read and
107+
/// filtered by workflow without joining back to the instance row.
108+
/// </summary>
109+
public string? WorkflowName { get; init; }
110+
111+
/// <summary>Where this event goes: the durable log, live subscribers, or both.</summary>
112+
public EventDeliveryMode Delivery { get; init; } = EventDeliveryMode.StreamAndLog;
113+
114+
/// <summary>True when the event carries no sequence number and leaves no durable record.</summary>
115+
public bool IsStreamOnly => Delivery == EventDeliveryMode.StreamOnly;
116+
}
117+
118+
/// <summary>
119+
/// Where an event is delivered. One enum rather than a pair of booleans, because "neither" is not a
120+
/// meaningful destination and should not be representable.
121+
/// </summary>
122+
public enum EventDeliveryMode
123+
{
124+
/// <summary>Appended to the durable log and fanned out to live subscribers. The default.</summary>
125+
StreamAndLog,
126+
127+
/// <summary>
128+
/// Appended to the durable log only. For a workflow that wants a queryable event record without
129+
/// a live stream — the run is still fully observable after the fact, just not as it happens.
130+
/// </summary>
131+
LogOnly,
132+
133+
/// <summary>
134+
/// Fanned out to live subscribers only, carrying no sequence number and leaving no record.
108135
/// </summary>
109136
/// <remarks>
110137
/// For data with no replay value — streamed LLM tokens, where the complete text is in the
111-
/// executor's output anyway. A transient event takes no sequence number, so the durable sequence
112-
/// stays gapless and <c>Last-Event-ID</c> catch-up keeps working; it is written to SSE without an
113-
/// <c>id:</c> field, which is what stops a reconnecting client waiting for a chunk that no longer
114-
/// exists. A token stream is not resumable and the transport should say so.
138+
/// executor's output anyway. Taking no sequence number keeps the durable sequence gapless so
139+
/// <c>Last-Event-ID</c> catch-up still works, and the event is written to SSE without an
140+
/// <c>id:</c> field, which stops a reconnecting client waiting for a chunk that no longer exists.
141+
/// A token stream is not resumable and the transport should say so.
115142
/// </remarks>
116-
public bool Transient { get; init; }
143+
StreamOnly
117144
}
118145

119146
/// <summary>

src/Abacus.Run/Api/Endpoints.cs

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
using System.Text.Json;
2+
using System.Text.Json.Serialization;
23
using Abacus.Run.Abstractions;
34
using Abacus.Run.Core;
45
using Microsoft.AspNetCore.Builder;
@@ -41,6 +42,49 @@ public sealed record CancelRequestDto(string? Reason);
4142

4243
public sealed record RerunRequestDto(string? Mode, JsonElement? Context, string? FromCheckpointId, string? Reason);
4344

45+
/// <summary>
46+
/// One row of the event log. <c>PayloadJson</c> is re-emitted as a raw JSON value rather than an
47+
/// escaped string, so a caller reads the payload directly instead of parsing it twice.
48+
/// </summary>
49+
public sealed record EventLogDto(
50+
string InstanceId,
51+
string? WorkflowName,
52+
long Sequence,
53+
string EventType,
54+
string? ExecutorId,
55+
int? Superstep,
56+
string? TenantId,
57+
[property: JsonConverter(typeof(RawJsonConverter))] string PayloadJson,
58+
DateTimeOffset OccurredAt);
59+
60+
/// <summary>Writes an already-serialised JSON string through untouched.</summary>
61+
internal sealed class RawJsonConverter : JsonConverter<string>
62+
{
63+
public override string Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
64+
=> reader.GetString() ?? "{}";
65+
66+
public override void Write(Utf8JsonWriter writer, string value, JsonSerializerOptions options)
67+
{
68+
if (string.IsNullOrWhiteSpace(value))
69+
{
70+
writer.WriteNullValue();
71+
return;
72+
}
73+
74+
try
75+
{
76+
using JsonDocument document = JsonDocument.Parse(value);
77+
document.RootElement.WriteTo(writer);
78+
}
79+
catch (JsonException)
80+
{
81+
// A payload that is not valid JSON is written as the string it is, rather than failing
82+
// the whole response — one malformed row must not hide the rest of the log.
83+
writer.WriteStringValue(value);
84+
}
85+
}
86+
}
87+
4488
public sealed record PublishEventRequestDto(
4589
string? Topic, JsonElement? Payload, string? CorrelationKey, string? Scope);
4690

@@ -391,6 +435,17 @@ private static void MapEvents(IEndpointRouteBuilder app)
391435
return Results.NotFound();
392436
}
393437

438+
// A log-only workflow would hold this stream open and never write to it, which a client
439+
// cannot tell apart from a stalled run. Refuse, and say where the events actually are.
440+
if (NotificationsOf(services, instance) is { IsLogOnly: true })
441+
{
442+
return Results.Problem(
443+
title: "This workflow does not stream events",
444+
detail: $"'{instance.WorkflowName}' is configured for log-only delivery. Read its events at " +
445+
$"/v2/workflows/{instance.WorkflowName}/instances/{id}/events.",
446+
statusCode: StatusCodes.Status409Conflict);
447+
}
448+
394449
long from = 0;
395450
string? lastEventId = http.Request.Headers["Last-Event-ID"].FirstOrDefault();
396451
if (long.TryParse(lastEventId, out long parsed))
@@ -593,6 +648,59 @@ private static void MapApprovals(IEndpointRouteBuilder app)
593648
return Results.Accepted(value: new { messageId = message.MessageId, topic = message.Topic });
594649
});
595650

651+
// The event log as a queryable record rather than a stream. This is the read path for a
652+
// log-only workflow, and works just as well for a streamed one — the same rows either way.
653+
// Scoped by workflow name, matching the instance-state route, so the caller states which
654+
// workflow they believe they are reading and a mismatch is caught rather than assumed.
655+
app.MapGet("/v2/workflows/{name}/instances/{id}/events", async (
656+
string name,
657+
string id,
658+
[FromQuery] long? from,
659+
[FromQuery] long? to,
660+
[FromQuery] int? limit,
661+
[FromQuery] string? types,
662+
IInstanceStore instances,
663+
IEventStore store,
664+
CancellationToken cancellationToken) =>
665+
{
666+
WorkflowInstance? instance = await instances.GetAsync(id, cancellationToken).ConfigureAwait(false);
667+
if (instance is null)
668+
{
669+
return Results.NotFound();
670+
}
671+
672+
if (!string.Equals(instance.WorkflowName, name, StringComparison.OrdinalIgnoreCase))
673+
{
674+
return Results.Problem(
675+
title: "Workflow mismatch",
676+
detail: $"Instance '{id}' belongs to '{instance.WorkflowName}', not '{name}'.",
677+
statusCode: StatusCodes.Status404NotFound);
678+
}
679+
680+
string[]? typeFilter = types?
681+
.Split(',', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);
682+
683+
Page<EventEnvelope> page = await store.QueryAsync(
684+
new EventQuery(id, from ?? 0, to, Math.Clamp(limit ?? 100, 1, 1000), typeFilter), cancellationToken)
685+
.ConfigureAwait(false);
686+
687+
return Results.Ok(new
688+
{
689+
instanceId = id,
690+
workflowName = instance.WorkflowName,
691+
workflowVersion = instance.WorkflowVersion,
692+
tenantId = instance.TenantId,
693+
total = page.Total,
694+
nextCursor = page.NextCursor,
695+
696+
// WorkflowName is denormalised onto each row too, so a caller collecting events from
697+
// several instances keeps the attribution without carrying the envelope's context.
698+
items = page.Items.Select(e => new EventLogDto(
699+
e.InstanceId, e.WorkflowName ?? instance.WorkflowName, e.Sequence, e.EventType,
700+
e.ExecutorId, e.Superstep, e.TenantId, e.PayloadJson, e.OccurredAt))
701+
});
702+
});
703+
596704
// What is listening, and what is waiting. An instance parked on an event with no visible
597705
// reason is the worst version of this feature.
598706
app.MapGet("/subscriptions", async (
@@ -730,6 +838,21 @@ internal static bool TryParseOutcome(string? value, out ApprovalOutcomeKind outc
730838
approval.Assignees, approval.RequiredApprovers, approval.AllowModification, approval.CreatedAt,
731839
approval.ExpiresAt, $"/approvals/{approval.ApprovalId}/decision");
732840

841+
/// <summary>
842+
/// The notification policy the instance's own workflow version declared, or null when it
843+
/// declared none. Resolved by the instance's version, not the newest, so an in-flight run keeps
844+
/// the behaviour it started under.
845+
/// </summary>
846+
private static NotificationPolicy? NotificationsOf(IServiceProvider services, WorkflowInstance instance)
847+
{
848+
var registry = services.GetService<IWorkflowRegistry>();
849+
850+
return registry?.Resolve(instance.WorkflowName, instance.WorkflowVersion)?.Definition
851+
is INotifyingWorkflow notifying
852+
? notifying.Notifications
853+
: null;
854+
}
855+
733856
/// <summary>
734857
/// The delivered payload is deliberately absent: it is domain data that has already been
735858
/// redacted on its way to the event stream, and repeating it unredacted here would undo that.

0 commit comments

Comments
 (0)