Skip to content

[WIP] System.Text.Json and ProtocolBuffers codecs for Orleans.Journaling#9973

Draft
ReubenBond wants to merge 11 commits intodotnet:mainfrom
ReubenBond:rebond/journaling-aot-codecs
Draft

[WIP] System.Text.Json and ProtocolBuffers codecs for Orleans.Journaling#9973
ReubenBond wants to merge 11 commits intodotnet:mainfrom
ReubenBond:rebond/journaling-aot-codecs

Conversation

@ReubenBond
Copy link
Copy Markdown
Member

@ReubenBond ReubenBond commented Mar 25, 2026 •
edited
Loading

Summary

Refactors Orleans.Journaling serialization behind pluggable ILogEntryCodec<TEntry> / ILogDataCodec<T> abstractions and introduces two new NuGet packages — all Native AOT compatible (no MakeGenericType or reflection).

Changes

Core abstractions (Orleans.Journaling)

  • Typed command DTOs in Entries/ — each durable type gets a sealed record hierarchy (DurableDictionaryEntry<TKey, TValue>, DurableListEntry<T>, etc.)
  • ILogEntryCodec<TEntry> — per-entry-hierarchy codec: Write(TEntry, IBufferWriter<byte>) and Read(ReadOnlySequence<byte>)
  • ILogDataCodec<T> — raw value codec for payload serialization
  • Per-type codec provider interfaces (IDurableDictionaryCodecProvider, etc.) — provide closed-generic codecs without reflection
  • OrleansBinary*EntryCodec — preserves the existing Orleans binary wire format as the default codec
  • VarIntHelper — standalone LEB128 helpers with overflow guards (5-byte cap for uint32, 10-byte for uint64)

New: Microsoft.Orleans.Journaling.Json (alpha)

  • System.Text.Json-based codecs for all durable types
  • JSON discriminator (cmd field) for command dispatch
  • Configurable via UseJsonCodec() extension method

New: Microsoft.Orleans.Journaling.Protobuf (alpha)

  • Google Protocol Buffers codecs using generated .proto schema
  • TypedValue oneof for native protobuf encoding of well-known scalar types
  • Falls back to ILogDataCodec<T> for non-native types
  • Configurable via UseProtobufCodec() extension method

Durable type refactoring

  • All durable types (DurableDictionary, DurableList, DurableQueue, DurableSet, DurableState, DurableValue, DurableTaskCompletionSource) now consume ILogEntryCodec<TEntry> through per-type provider injection
  • Removed inline Orleans serialization code from each durable type

Tests

  • Round-trip codec tests for all durable types in both JSON and Protobuf
  • Migration tests (write-and-recover) for binary, JSON, and Protobuf formats
  • LogDataCodecTests for the Orleans binary data codec
  • Existing durable type tests updated and passing

Usage

// Default: Orleans binary format (backward compatible)
builder.AddStateMachineStorage();

// JSON format
builder.AddStateMachineStorage().UseJsonCodec();

// Protobuf format
builder.AddStateMachineStorage().UseProtobufCodec();

Test results

  • Core: 62/62 ✅
  • JSON: 32/32 ✅
  • Protobuf: 31/31 ✅

ReubenBond and others added 11 commits March 25, 2026 11:59
@ReubenBond
feat(Journaling): Typed command DTOs and per-type ILogEntryCodec<TEntry>
f46178e 
Refactor Orleans.Journaling serialization to use typed command objects
instead of positional stream-based ILogEntryWriter/ILogEntryReader.

Core changes:
- Add command DTO hierarchies for all 7 durable types (Entries/)
- Add ILogEntryCodec<TEntry> per-type codec interface
- Add ILogDataCodec<T> value-level serialization interface
- Add binary per-type codecs preserving exact legacy wire format (Codecs/)
- Refactor all durable types to use pattern matching on DTOs
- Update StateMachineManager to accept typed codecs via DI
- Add OrleansLogDataCodec<T> adapter for IFieldCodec<T>
- Scaffold Orleans.Journaling.Json and Orleans.Journaling.Protobuf packages

All 71 existing tests pass on both net8.0 and net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
feat(Journaling.Json): Per-type JSON codecs with JsonElement and 'cmd…
5e2624b 
…' discriminator

Replace old positional-stream JsonEntryCodec/JsonLogDataCodec with per-type
codecs using non-generic JsonElement DTOs and [JsonPolymorphic] for the 'cmd'
type discriminator.

- Add JsonEntryTypes.cs with non-generic JSON DTOs using JsonElement fields
- Add JsonDictionaryEntryCodec<TKey,TValue> bridging generic DTOs ↔ JSON DTOs
- Add JsonListEntryCodec<T>, JsonQueueEntryCodec<T>, JsonSetEntryCodec<T>
- Add JsonValueEntryCodec<T>, JsonStateEntryCodec<T>, JsonTcsEntryCodec<T>
- Add JsonLogEntryCodecResolver<TEntry> for open-generic DI resolution
- Update UseJsonCodec() extension
- JSON codec does not use ILogDataCodec — uses System.Text.Json directly

All 71 tests pass on both net8.0 and net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
feat(Journaling.Protobuf): Per-type Protobuf codecs with protobuf wir…
254790d 
…e format

Replace old positional-stream ProtobufEntryCodec/ProtobufLogDataCodec with
per-type codecs using Google.Protobuf CodedOutputStream/CodedInputStream.

- Add ProtobufCodecHelper for shared serialization utilities
- Add ProtobufDictionaryEntryCodec<TKey,TValue> with tagged protobuf fields
- Add ProtobufListEntryCodec<T>, ProtobufQueueEntryCodec<T>, ProtobufSetEntryCodec<T>
- Add ProtobufValueEntryCodec<T>, ProtobufStateEntryCodec<T>, ProtobufTcsEntryCodec<T>
- Add ProtobufLogEntryCodecResolver<TEntry> for open-generic DI resolution
- Protobuf codec uses ILogDataCodec<T> for user value serialization within
  the protobuf envelope (values as length-delimited bytes fields)

All 71 tests pass on both net8.0 and net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
refactor(Journaling): Remove old positional-stream ILogEntryWriter/Re…
3601075 
…ader/CodecFactory

Remove the superseded positional-stream serialization abstractions:
- Delete ILogEntryCodec.cs (old ILogEntryWriter, ILogEntryReader, ILogEntryCodecFactory)
- Delete OrleansBinaryEntryCodec.cs (old OrleansBinaryEntryWriter, OrleansBinaryEntryReader)
- Rename ILogEntryCodec2.cs → ILogEntryCodec.cs (typed per-type codec is now the only interface)
- Remove ILogEntryCodecFactory DI registration from HostingExtensions

The typed ILogEntryCodec<TEntry> per-type codec approach fully replaces these.
All 71 tests pass on both net8.0 and net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
test(Journaling): Add Protobuf codec round-trip tests for all durable…
e611f4a 
… types

Add comprehensive round-trip tests for all 7 Protobuf per-type codecs:
Dictionary (Set/Remove/Clear/Snapshot), List (Add/Set/Insert/RemoveAt/Clear/Snapshot),
Queue (Enqueue/Dequeue/Clear/Snapshot), Set (Add/Remove/Clear/Snapshot),
Value (Set), State (Set/Clear), TCS (Completed/Faulted/Canceled/Pending).

Total: 96 tests pass on both net8.0 and net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
test(Journaling): Add Json/Protobuf projects to slnx and thorough JSO…
b18a70c 
…N codec tests

- Add Orleans.Journaling.Json and Orleans.Journaling.Protobuf to Orleans.slnx
- Add comprehensive JsonCodecTests.cs with 28 tests covering all 7 durable
  types: Dictionary (Set/Remove/Clear/Snapshot/EmptySnapshot/ComplexTypes),
  List (Add/Set/Insert/RemoveAt/Clear/Snapshot), Queue (Enqueue/Dequeue/Clear/
  Snapshot), Set (Add/Remove/Clear/Snapshot), Value (Set/ComplexType),
  State (Set/Clear), TCS (Completed/Faulted/Canceled/Pending)
- Each test verifies JSON output contains correct 'cmd' discriminator and
  values round-trip correctly

Total: 124 tests pass on both net8.0 and net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
refactor(Journaling): Split Json/Protobuf tests into separate projects
62aaafd 
- Create test/Orleans.Journaling.Json.Tests/ with JsonCodecTests (28 tests)
  and CodecMigrationTests (4 tests)
- Create test/Orleans.Journaling.Protobuf.Tests/ with ProtobufCodecTests (25 tests)
- Remove Json/Protobuf test dependencies from core Orleans.Journaling.Tests
- Add test projects to Orleans.slnx
- Add InternalsVisibleTo for new test projects in src csproj files

Test distribution:
  Orleans.Journaling.Tests: 62 tests (core + binary codec)
  Orleans.Journaling.Json.Tests: 32 tests (JSON codec + migration)
  Orleans.Journaling.Protobuf.Tests: 25 tests (Protobuf codec)
  Total: 119 tests, all passing on both net8.0 and net10.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
Replace manual protobuf wire-format with .proto schema and code gener…
312972e 
…ation

Rewrite Orleans.Journaling.Protobuf codecs to use a formal .proto schema
(JournalingEntries.proto) with Grpc.Tools code generation instead of manual
CodedOutputStream/CodedInputStream wire-format encoding.

- Add Protos/JournalingEntries.proto defining all 7 entry message types
  using oneof for command discrimination
- Add Grpc.Tools package reference and Protobuf compile item to .csproj
- Rewrite all codec implementations as thin mappers between Orleans entry
  records and generated protobuf messages
- Use WriteTo(IBufferWriter<byte>) and ParseFrom(ReadOnlySequence<byte>)
  for zero-intermediate-copy serialization
- Remove manual tag constants, MemoryStream intermediaries, SkipField,
  and CopyToBufferWriter helpers
- Architecture now mirrors the JSON codec pattern exactly

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
Add TypedValue for native protobuf encoding of well-known types
17162ff 
Introduce a TypedValue proto message with oneof for native protobuf scalars
(int32, int64, uint32, uint64, float, double, bool, string) and a bytes
fallback. Replace all raw bytes value fields with TypedValue in the proto
schema.

Add ProtobufValueConverter<T> that resolves the optimal encoding strategy
once at construction time:
- Native scalars (int, string, etc.) use oneof fields directly
- IMessage types use ToByteString()/Parser.ParseFrom() natively
- Other types fall back to ILogDataCodec<T> via bytes_value

The resolver only resolves ILogDataCodec<T> from DI when T is not a
natively supported type, avoiding unnecessary DI resolution for common
cases.

Remove ProtobufCodecHelper (replaced by ProtobufValueConverter).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
Remove reflection from journaling codec resolvers for Native AOT
3844ef6 
Replace open-generic ILogEntryCodec<> DI resolvers that used
GetGenericTypeDefinition, MakeGenericType, and Activator.CreateInstance
with per-type codec provider interfaces. Each durable type now injects
its specific provider (e.g., IDurableDictionaryCodecProvider) and calls
a typed GetCodec<...>() method, letting the compiler see generic
instantiations statically.

- Add 7 per-type codec provider interfaces in ILogEntryCodecProviders.cs
- Add OrleansBinaryLogEntryCodecProvider implementing all 7 interfaces
- Refactor JsonLogEntryCodecResolver into JsonLogEntryCodecProvider
- Refactor ProtobufLogEntryCodecResolver into ProtobufLogEntryCodecProvider
- Update all durable types and StateMachineManager constructors
- Add DynamicallyAccessedMembers annotation on ProtobufValueConverter<T>
- Delete DefaultLogEntryCodecResolver (replaced by provider)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
Fix VarInt overflow, JSON DI registration, and add Protobuf migration…
456d85c 
… tests

- Add byte-count caps to VarIntHelper: 5 bytes max for uint32, 10 for uint64,
  preventing infinite loops on malformed LEB128 input.
- Fix JsonJournalingExtensions to capture JsonSerializerOptions via factory
  lambda instead of registering it as a bare singleton (avoids DI collisions
  with other components that register JsonSerializerOptions).
- Add CodecMigrationTests for the Protobuf codec covering Dictionary, List,
  Value, Set, and Queue write-and-recover round-trips.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond ReubenBond changed the title Native AOT-compatible journaling codecs with JSON and Protobuf support [WIP] System.Text.Json and ProtocolBuffers codecs for Orleans.Journaling Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ReubenBond