Implement wal logging and replay for `EdgeView`, `NodeView`, `delete_edge` and graph properties by fabubaker · Pull Request #2493 · Pometry/Raphtory

fabubaker · 2026-02-13T21:45:28Z

Adds wal logging and replay for add_metadata, update_metadata and add_updates on an EdgeView and NodeView.

Additionally adds wal support for delete_edge and all operations on graph properties.

We can reuse existing wal code for add_edge

github-actions

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite	Current: `f957390`	Previous: `9567900`	Ratio
`lotr_graph/num_edges`	`4` ns/iter (`± 0`)	`0` ns/iter (`± 0`)	`+∞`
`lotr_graph/num_nodes`	`432` ns/iter (`± 1`)	`2` ns/iter (`± 0`)	`216`
`lotr_graph/graph_latest`	`3` ns/iter (`± 0`)	`0` ns/iter (`± 0`)	`+∞`
`lotr_graph_materialise/materialize`	`9270087` ns/iter (`± 89816`)	`1522661` ns/iter (`± 3102`)	`6.09`
`lotr_graph_window_100/num_edges`	`29` ns/iter (`± 0`)	`8` ns/iter (`± 0`)	`3.63`
`lotr_graph_window_100/num_nodes`	`486` ns/iter (`± 1`)	`5` ns/iter (`± 0`)	`97.20`
`lotr_graph_window_100_materialise/materialize`	`9545910` ns/iter (`± 64509`)	`1647661` ns/iter (`± 5147`)	`5.79`
`lotr_graph_window_10/has_node_existing`	`133` ns/iter (`± 9`)	`61` ns/iter (`± 12`)	`2.18`
`lotr_graph_window_10/iterate nodes`	`34307` ns/iter (`± 110`)	`10897` ns/iter (`± 52`)	`3.15`
`lotr_graph_window_10/iterate edges`	`104057` ns/iter (`± 228`)	`45567` ns/iter (`± 1365`)	`2.28`
`lotr_graph_window_10_materialise/materialize`	`6155433` ns/iter (`± 37037`)	`1013406` ns/iter (`± 5119`)	`6.07`
`lotr_graph_subgraph_10pc/has_edge_existing`	`520` ns/iter (`± 4`)	`85` ns/iter (`± 1`)	`6.12`
`lotr_graph_subgraph_10pc/num_nodes`	`354` ns/iter (`± 4`)	`4` ns/iter (`± 0`)	`88.50`
`lotr_graph_subgraph_10pc/has_node_existing`	`231` ns/iter (`± 4`)	`30` ns/iter (`± 0`)	`7.70`
`lotr_graph_subgraph_10pc/iterate nodes`	`4679` ns/iter (`± 15`)	`853` ns/iter (`± 1`)	`5.49`
`lotr_graph_subgraph_10pc_windowed/has_node_existing`	`133` ns/iter (`± 7`)	`62` ns/iter (`± 14`)	`2.15`
`lotr_graph_subgraph_10pc_windowed/iterate nodes`	`4685` ns/iter (`± 37`)	`1399` ns/iter (`± 3`)	`3.35`
`lotr_graph_window_50_layered/num_nodes`	`101959` ns/iter (`± 1104`)	`22515` ns/iter (`± 3326`)	`4.53`
`lotr_graph_window_50_layered/has_node_existing`	`1413` ns/iter (`± 206`)	`123` ns/iter (`± 15`)	`11.49`
`lotr_graph_window_50_layered/max_id`	`115275` ns/iter (`± 1542`)	`27871` ns/iter (`± 475`)	`4.14`
`lotr_graph_window_50_layered/max_degree`	`1944492` ns/iter (`± 21262`)	`298581` ns/iter (`± 9998`)	`6.51`
`lotr_graph_window_50_layered/iterate nodes`	`237823` ns/iter (`± 841`)	`18512` ns/iter (`± 205`)	`12.85`
`lotr_graph_window_50_layered/iterate edges`	`214859` ns/iter (`± 463`)	`83040` ns/iter (`± 327`)	`2.59`
`lotr_graph_window_50_layered/max_neighbour_degree`	`3651492` ns/iter (`± 16284`)	`388874` ns/iter (`± 1455`)	`9.39`
`lotr_graph_window_50_layered/graph_latest`	`183360` ns/iter (`± 8579`)	`38418` ns/iter (`± 718`)	`4.77`
`lotr_graph_window_50_layered_materialise/materialize`	`21718186` ns/iter (`± 378074`)	`3378119` ns/iter (`± 12216`)	`6.43`
`lotr_graph_persistent_window_50_layered/num_edges_temporal`	`587197` ns/iter (`± 2834`)	`207488` ns/iter (`± 4439`)	`2.83`
`lotr_graph_persistent_window_50_layered/num_nodes`	`137021` ns/iter (`± 2079`)	`31798` ns/iter (`± 386`)	`4.31`
`lotr_graph_persistent_window_50_layered/has_node_existing`	`1741` ns/iter (`± 432`)	`180` ns/iter (`± 82`)	`9.67`
`lotr_graph_persistent_window_50_layered/max_id`	`151675` ns/iter (`± 2146`)	`39294` ns/iter (`± 1497`)	`3.86`
`lotr_graph_persistent_window_50_layered/max_degree`	`2764795` ns/iter (`± 17389`)	`457666` ns/iter (`± 12739`)	`6.04`
`lotr_graph_persistent_window_50_layered/iterate nodes`	`323659` ns/iter (`± 802`)	`36447` ns/iter (`± 154`)	`8.88`
`lotr_graph_persistent_window_50_layered/iterate edges`	`187474` ns/iter (`± 434`)	`86397` ns/iter (`± 339`)	`2.17`
`lotr_graph_persistent_window_50_layered/max_neighbour_degree`	`4892212` ns/iter (`± 10590`)	`518296` ns/iter (`± 8441`)	`9.44`
`lotr_graph_persistent_window_50_layered/graph_latest`	`274699` ns/iter (`± 16021`)	`58093` ns/iter (`± 1228`)	`4.73`
`lotr_graph_persistent_window_50_layered_materialise/materialize`	`54902456` ns/iter (`± 1598088`)	`5111080` ns/iter (`± 28206`)	`10.74`
`lotr_graph/proto_decode`	`9583564` ns/iter (`± 60049`)	`1875628` ns/iter (`± 40776`)	`5.11`
`lotr_graph/proto_encode`	`9230511` ns/iter (`± 64127`)	`1163713` ns/iter (`± 114520`)	`7.93`

This comment was automatically generated by workflow using github-action-benchmark.

fabubaker · 2026-02-19T21:03:01Z

db4-graph/src/replay.rs

-                match prop_mapper.get_dtype(*prop_id) {
-                    None => {
-                        prop_mapper.set_id_and_dtype(
-                            prop_name.as_str(),
-                            *prop_id,
-                            prop_value.dtype(),
-                        );
-                    }
-                    Some(old_dtype) => {
-                        let dtype = prop_value.dtype();
-                        let mut unified = false;
-                        let new_dtype = unify_types(&old_dtype, &dtype, &mut unified)?;
-                        if unified {
-                            prop_mapper.set_dtype(*prop_id, new_dtype);
-                        }
-                    }
-                }


set_or_unify_id_and_dtype is not exactly the same as this block, since it always sets the id whereas this doesn't, but minor differences aside lmk if it's ok to use this method here.

we can always optimise that later, so using that method is fine.

fabianmurariu

Looks good, just a few changes needed

fabianmurariu · 2026-02-23T14:50:41Z

db4-graph/src/replay.rs

+                )?;
+            }
+
+            let mut edge_writer = self.edges.get_mut(edge_segment_id).unwrap().writer();


unwrap() i'd switch to ok_or_else

fabianmurariu · 2026-02-23T14:57:11Z

db4-graph/src/replay.rs

+                )?;
+            }
+
+            let mut node_writer = self.nodes.get_mut(segment_id).unwrap().writer();


more unwrap

fabianmurariu · 2026-02-23T14:57:35Z

db4-graph/src/replay.rs

+
+            let mut node_writer = self.nodes.get_mut(segment_id).unwrap().writer();
+
+            let props_vec: Vec<_> = props.iter().map(|(_, id, p)| (*id, p.clone())).collect();


I don't think collect is needed

fabianmurariu · 2026-02-23T14:57:50Z

db4-graph/src/replay.rs

+                ))
+            })?;
+
+            let props_vec: Vec<_> = props.iter().map(|(_, id, p)| (*id, p.clone())).collect();


probably needless collect

…dge-view

ljeub-pometry

Some minor things to clean up

ljeub-pometry · 2026-02-24T08:42:23Z

db4-graph/src/replay.rs

            // Insert prop ids into edge meta.
            for (prop_name, prop_id, prop_value) in &props {
                let prop_mapper = edge_meta.temporal_prop_mapper();
+                let mut write_locked_mapper = prop_mapper.write_locked();


this should be outside the loop! you keep locking and unlocking the mapper for no reason

Actually, maybe it would make more sense to have this as part of the initialisation of the write-locked graph itself?

ljeub-pometry · 2026-02-24T08:45:31Z

db4-graph/src/replay.rs

-                match prop_mapper.get_dtype(*prop_id) {
-                    None => {
-                        prop_mapper.set_id_and_dtype(
-                            prop_name.as_str(),
-                            *prop_id,
-                            prop_value.dtype(),
-                        );
-                    }
-                    Some(old_dtype) => {
-                        let dtype = prop_value.dtype();
-                        let mut unified = false;
-                        let new_dtype = unify_types(&old_dtype, &dtype, &mut unified)?;
-                        if unified {
-                            prop_mapper.set_dtype(*prop_id, new_dtype);
-                        }
-                    }
-                }


we can always optimise that later, so using that method is fine.

ljeub-pometry · 2026-02-24T08:47:50Z

db4-graph/src/replay.rs

+
+            for (prop_name, prop_id, prop_value) in &props {
+                let prop_mapper = edge_meta.metadata_mapper();
+                let mut write_locked_mapper = prop_mapper.write_locked();


As before, we should not have this inside the loop. Probably lock these as part of the write-locked graph initialisation.

ljeub-pometry · 2026-02-24T08:51:38Z

db4-graph/src/replay.rs

+        lsn: LSN,
+        _transaction_id: TransactionID,
+        t: EventTime,
+        _src_name: Option<GID>,


It is possible to create new nodes on a delete_edge call, you need to handle these!

ljeub-pometry · 2026-02-24T08:52:25Z

db4-graph/src/replay.rs

-                        }
-                    }
-                }
+                let mut write_locked_mapper = prop_mapper.write_locked();


As before, don't call this in the loop.

ljeub-pometry · 2026-02-24T08:55:56Z

db4-graph/src/replay.rs

+            })?;
+
+            let mut node_writer = node_writer.writer();
+            let props = props.iter().map(|(_, id, p)| (*id, p.clone()));


use into_iter to avoid the clone

ljeub-pometry · 2026-02-24T08:57:46Z

db4-graph/src/replay.rs

+                ))
+            })?;
+
+            let props = props.iter().map(|(_, id, p)| (*id, p.clone()));


use into_iter to avoid the clone

ljeub-pometry · 2026-02-24T08:58:15Z

db4-graph/src/replay.rs

+            })?;
+
+            let mut node_writer = node_writer.writer();
+            let props = props.iter().map(|(_, id, p)| (*id, p.clone()));


use into_iter to avoid the clone

ljeub-pometry · 2026-02-24T09:07:11Z

db4-storage/src/pages/node_store.rs

+                            // The current segment is full, drop its lock and try to find
+                            // another free segment.


We are not trying to find a free segment here, we are going to replace the segment at this slot with a new one (unless another thread managed to replace it before we get to it)

ljeub-pometry · 2026-02-24T09:10:00Z

db4-storage/src/segments/graph_prop/mod.rs

    }

+    fn immut_lsn(&self) -> LSN {
+        self.head.read().lsn()


That doesn't seem like the immut_lsn since you are reading the mutable part?

fabubaker added 4 commits February 13, 2026 16:44

Add logging and replay for add_updates for edges

1a2a8de

Remove separate add_edge_properties wal entry

6263303

We can reuse existing wal code for add_edge

Use set_or_unify_id_and_dtype during replay

eb97fed

Add logging and replay for AddEdgeMetadata

d110fb9

fabubaker changed the title ~~Implement wal logging and replay for write methods in EdgeView~~ Implement wal logging and replay for EdgeView write operations Feb 13, 2026

fabubaker marked this pull request as draft February 13, 2026 21:47

github-actions bot reviewed Feb 13, 2026

View reviewed changes

github-actions bot and others added 7 commits February 13, 2026 22:55

chore: apply tidy-public auto-fixes

4df7734

Remove src_id and dst_id for add_metadata

d5564ee

Use AddEdgeMetadata logging methods

34bba81

Add set_lsn method for writers

2039f22

Use correct mapper for metadata

1fd78f2

Create add_metadata_impl and reuse

74fd32b

Merge branch 'db_v4' into db_v4_/wal-edge-view

2cc6064

fabubaker marked this pull request as ready for review February 18, 2026 19:41

chore: apply tidy-public auto-fixes

86e5d5a

fabubaker requested a review from ljeub-pometry February 18, 2026 21:09

fabubaker added 3 commits February 19, 2026 15:28

Add wal logging for NodeView.add_updates

1d92c7a

Create add_metadata_impl for NodeView and reuse

b672f93

Add wal logging and replay for NodeView.add_metadata

8030746

fabubaker changed the title ~~Implement wal logging and replay for EdgeView write operations~~ Implement wal logging and replay for EdgeView and NodeView operations Feb 19, 2026

fabubaker commented Feb 19, 2026

View reviewed changes

github-actions bot and others added 8 commits February 19, 2026 22:21

chore: apply tidy-public auto-fixes

5d43d37

Add wal for set_node_type

d936726

Add FIXME for actual set_node_type logging

e2842e6

Add logging and replay for graph_props

e1bf5ae

Implement set_lsn for GraphPropWriter

a46cd98

Return GraphPropWriterT

acf70ad

Call wal.log in add_properties

d4a0c53

Call wal.log in add_properties

f9db442

fabianmurariu requested changes Feb 23, 2026

View reviewed changes

fabubaker added 15 commits February 23, 2026 10:35

Remove some unwraps and collects in replay

fd2e3d8

Merge branch 'db_v4' of github.com:Pometry/Raphtory into db_v4_/wal-e…

67e623b

…dge-view

Fix some more merge conflicts

ef2d9f3

Add logging for set_node_type

8873415

Run fmt

734c2df

Run fmt

ebee5da

Cleanup and fix reserve_and_lock_segment

3d45e82

Remove Option from WriteLockedGraphPropPages writer

3f469a2

Check lsn during graph prop replay

d73f3f8

Refactor graph props add_metadata

20a3b77

Implement logging and replay for graph metadata

cca29fb

Use is_update correctly

72200d6

Implement logging and replay for delete_edge

9f95da8

Run fmt

d30f910

Simplify some comments

f69e1bf

fabubaker force-pushed the db_v4_/wal-edge-view branch from b7f08a8 to f69e1bf Compare February 23, 2026 23:53

fabubaker changed the title ~~Implement wal logging and replay for EdgeView and NodeView operations~~ Implement wal logging and replay for EdgeView, NodeView, delete_edge and graph properties Feb 24, 2026

Run fmt

13a9304

ljeub-pometry requested changes Feb 24, 2026

View reviewed changes

fabianmurariu added 2 commits February 24, 2026 11:29

fix some review comments

e17ae0f

fmt

f957390


		let mut node_writer = self.nodes.get_mut(segment_id).unwrap().writer();

		let props_vec: Vec<_> = props.iter().map(\|(_, id, p)\| (*id, p.clone())).collect();

		// The current segment is full, drop its lock and try to find
		// another free segment.

Comments

Conversation

fabubaker commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

⚠️ Performance Alert ⚠️

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fabianmurariu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ljeub-pometry left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fabubaker commented Feb 13, 2026 •

edited

Loading

github-actions bot left a comment •

edited

Loading