Ra v3 #494
Merged
Ra v3 #494
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kjnilsson
force-pushed
the
v3
branch
2 times, most recently
from
dbb90ff to
08610a0
Compare
kjnilsson
force-pushed
the
v3
branch
2 times, most recently
from
70b7e6e to
3e50dd2
Compare
kjnilsson
force-pushed
the
v3
branch
5 times, most recently
from
f8a3bed to
177855c
Compare
kjnilsson
force-pushed
the
v3
branch
2 times, most recently
from
53636ed to
0f65f08
Compare
kjnilsson
force-pushed
the
v3
branch
2 times, most recently
from
b13920d to
360eb77
Compare
kjnilsson
force-pushed
the
v3
branch
3 times, most recently
from
d3a23c9 to
ab1281f
Compare
kjnilsson
force-pushed
the
v3
branch
2 times, most recently
from
e83b688 to
b4c1959
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR brings test suites in line with the v3 work by updating snapshot and log APIs, renaming backends, and removing obsolete tests.
- Rename
ra_log_readertora_log_segmentsin unit tests - Refactor
ra_snapshot_SUITEto use the newbg_workpattern and updatedpending/complete_snapshotsignatures - Update WAL and memory tests to expect lists of indexes instead of tuple ranges, and remove the legacy property-based suite
Reviewed Changes
Copilot reviewed 52 out of 55 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| test/unit_SUITE.erl | Replace ra_log_reader with ra_log_segments |
| test/ra_snapshot_SUITE.erl | Migrate snapshot tests to new begin_snapshot/pending API |
| test/ra_log_wal_SUITE.erl | Adapt await_written to expect lists of indexes |
| test/ra_log_memory.erl | Change install_snapshot/4 to match new live‐indexes signature |
| test/ra_machine_ets_SUITE.erl | Remove Common Test include (ct.hrl) |
| test/ra_log_props_SUITE.erl | Delete entire deprecated property-based suite |
Comments suppressed due to low confidence (2)
test/ra_log_memory.erl:174
- The new
install_snapshot/4clause returns{ok, State, []}but the expected return is{State, List}. Remove theokatom so it matches the spec and callers ({State, []}).
{ra_log_memory_state(), list()}.
test/ra_machine_ets_SUITE.erl:12
- The Common Test macros (e.g.
ct:pal) are used elsewhere in this file. Removing thect.hrlinclude may cause undefined macro errors. Re-add the Common Test include if CT macros are still used.
-export([
611ab22 to
e368de3
Compare
kjnilsson
force-pushed
the
v3
branch
3 times, most recently
from
5f67ef4 to
c525487
Compare
In certain scenarios.
kjnilsson
force-pushed
the
v3
branch
2 times, most recently
from
0873afb to
c54fd86
Compare
Hopefully this will work better on windows.
Else subsequent writes would crash the entire process.
The WAL can lose writer tracking data so if we have unwritten items in the mem table these need to be resent after log init
When resending entries to the WAL after a failure or restart, the previous index must be correctly tracked and passed to each write call. This is essential for sparse writes where entries are not contiguous. Changes: - Consolidate resend_pending and resend_from0 into a single code path that threads PrevIdx through the fold - Update wal_rewrite to accept and pass PrevIdx to ra_log_wal:write - Fix write_sparse to properly update pending sequence with limit/append - Add guard to ra_seq:limit for non-integer ceiling index - Add tests for sparse write resend and set_last_index with pending
Extend the release_cursor effect to support a 4-tuple variant with
options: {release_cursor, Index, MacState, Opts}
Opts can specify conditions that must be met before the snapshot is
taken:
- {written, Idx}: wait until last_written >= Idx
- no_snapshot_sends: wait until no peers are receiving snapshots
This prevents snapshots from being taken before all referenced log
entries have been durably written to disk, which is important when
the machine state contains "live indexes" (references to entries
still in the WAL).
The pending release cursor is stashed in the server state and
re-evaluated when:
- Written events are processed (evaluate_quorum, handle_follower)
- Commit index advances (evaluate_commit_index_follower)
- Snapshot sends complete (install_snapshot_result, handle_down)
Introduce ra_li module implementing a leaky integrator for smoothed
rate estimation with configurable decay time. The integrator handles
irregular update intervals and gracefully handles backwards timestamps.
Track commit rate in ra_server_proc on every tick_timeout across all
states (leader, follower, candidate, pre_vote, await_condition). The
receive_snapshot state resets the integrator to avoid counting snapshot
index jumps as throughput.
Two rate metrics are exposed via ra:member_overview/1:
- commit_rate: smoothed rate using 6x tick_timeout decay (~30s default)
- commit_rate_last: point-in-time rate from the last tick interval
The commit rate state is stored as a tuple {ra_li:state(), LastCommitIndex,
LastIntervalRate} to minimize record fields.
Readers following a symlink during compaction could see stale data if the symlink was created before the .compacting file was renamed to the target. Reverse the order so the compacted data is in place before any symlink points to it. Update recovery logic to use .compacting absence as the commit point.
this is temporary for now to see if we can find the bug.
Else we may end up with pending entries that never will be written and will later trigger a resend that will fail.
As they may be queried later if using release cursor conditions.
- Remove snapshot_next_event field, store deferred event in
snapshot_phase as {awaiting_pending, EventType, Rpc}
- Add ra_log:has_pending/1 for O(1) pending check
- Simplify abort_receive to reset log based on snapshot_has_live_indexes
rather than checking individual phases
This was referenced Feb 2, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR for visibility of v3 work.
See: https://github.com/rabbitmq/ra/milestone/7
Ra v3 Pull Request Description
Overview
Ra v3 is a major release introducing log compaction - a fundamentally new approach to log management that allows state machines to retain specific log entries ("live indexes") beyond the snapshot boundary. This enables significant optimisations for workloads where keeping certain historical entries is more efficient than including them in snapshots.
✨ New Features
Sparse Log Compaction System
ra_machine:live_indexes/1callback - New optional callback allowing state machines to specify which log indexes should be retained beyond the snapshot boundaryra:trigger_compaction/1API - Manually trigger log compaction on a Ra serverNew Modules
ra_kv- Key value store that keeps values on disk and reads them on demand.ra_seq- Efficient sparse sequence data structure for tracking non-contiguous index rangesra_log_segments- Complete rewrite of segment management (replacesra_log_reader) with compaction supportra_log_snapshot_state- Tracks snapshot state including live indexes for WAL coordinationra_worker- Background worker process for each Ra server handling snapshots, checkpoints, and compactionConsistent Aux Commands
ra:consistent_aux/3- New API for executing aux commands after consensus has been achieved (similar toconsistent_querybut for aux commands)Absolute Machine Timers
{timer, Name, Duration, {abs, true}}WAL Sparse Write Support
write/7API acceptingPrevIndexparameterra_mt) refactored to track sparse sequences instead of contiguous rangesRead Entries API
ra_server_proc:read_entries/4for reading specific log entries by index (replaceslog_fold)🔧 Improvements
Snapshotting Enhancements
Performance Optimizations
ra_machine:live_indexescall to appropriate phaseRobustness Improvements
Remote Query API Changes**
ra:leader_query/2,3now only accepts MFA tuples{Module, Function, Args}- anonymous functions are no longer supportedra:consistent_query/2,3now only accepts MFA tuples{Module, Function, Args}- anonymous functions are no longer supportedra:local_query/2,3remains unchanged and still supports both anonymous functions and MFA tuplesRemoved Deprecated APIs
The following deprecated functions have been removed from the
ramodule:ra:start_server/1→ usera:start_server/2ra:restart_server/1→ usera:restart_server/2ra:stop_server/1→ usera:stop_server/2ra:start_cluster/1→ usera:start_cluster/2ra:overview/0→ usera:overview/1ra:register_external_log_reader/1→ removed entirely (external readers no longer supported)WAL Configuration Changes
wal_write_strategyoption removed - Thedefaultando_syncwrite strategies are no longer configurable; the WAL now uses a single optimized strategyModule Removals
ra_log_reader- Replaced byra_log_segmentswith enhanced compaction supportAPI Changes
ra_server_proc:log_fold/4→ replaced byra_server_proc:read_entries/4ra_snapshot:begin_snapshot/4→ra_snapshot:begin_snapshot/5ra_snapshot:complete_snapshot/3→ra_snapshot:complete_snapshot/4accept_chunksignature changed from 4 to 3 parametersra_snapshot:handle_down/3→ra_snapshot:handle_error/3Internal Structure Changes
ra_mt) now tracksra_seq:state()instead of{First, Last}rangessmallest_live_idxandseqinstead ofsnap_idxandrangeparentparameter (supervisor pid)📚 Documentation
docs/internals/COMPACTION.mddescribing the compaction architecture🧪 Testing
ra_kv_SUITE,ra_log_segments_SUITE,ra_seq_SUITEra_kv_harnessfor testing Ra KV under various failure conditionsra_log_props_SUITE(property-based tests)