Add developer documentation to explain room DAG concepts like outliers and state_groups

docs/development/room-dag-concepts.md

@@ -13,39 +13,43 @@ A (oldest) <---- B <---- C (most recent)
 
 ## Depth and stream ordering
 
-Events are sorted by `(topological_ordering, stream_ordering)` where
+Events are normally sorted by `(topological_ordering, stream_ordering)` where
 `topological_ordering` is just `depth`. In other words, we first sort by `depth`
 and then tie-break based on `stream_ordering`. `depth` is incremented as new
 messages are added to the DAG. Normally, `stream_ordering` is an auto
-incrementing integer but for `backfilled=true` events, it decrements.
-
-`depth` is not re-calculated if a message is inserted into the middle of the DAG.
+incrementing integer, but backfilled events start with `stream_ordering=-1` and decrement.
 
 ---
 
  - `/sync` returns things in the order they arrive at the server (`stream_ordering`).
- - `/backfill` returns them in the order determined by the event graph `(topological_ordering, stream_ordering)`.
+ - `/messages` (and `/backfill in the federation API) return them in the order determined by the event graph `(topological_ordering, stream_ordering)`.
 
-The general idea is that, if you're following a room in real-time (i.e. `/sync`), you probably want to see the messages as they arrive at your server, rather than skipping any that arrived late; whereas if you're looking at a historical section of timeline (i.e. `/messages`), you want to see the best representation of the state of the room as others were seeing it at the time.
+The general idea is that, if you're following a room in real-time (i.e.
+`/sync`), you probably want to see the messages as they arrive at your server,
+rather than skipping any that arrived late; whereas if you're looking at a
+historical section of timeline (i.e. `/messages`), you want to see the best
+representation of the state of the room as others were seeing it at the time.
 
 
 ## Forward extremity
 
-Most-recent-in-time events in the DAG which are not referenced by any `prev_events` yet.
+Most-recent-in-time events in the DAG which are not referenced by any other events' `prev_events` yet.
 
 The forward extremities of a room are used as the `prev_events` when the next event is sent.
 
 
 ## Backwards extremity
 
-The current marker of where we have backfilled up to.
-
-A backwards extremity is a place where the oldest-in-time events of the DAG
+The current marker of where we have backfilled up to and will generally be the
+oldest-in-time events we know of in the DAG.
 
 This is an event where we haven't fetched all of the `prev_events` for.
 
-Once we have fetched all of it's `prev_events`, it's unmarked as backwards
-extremity and those `prev_events` become the new backwards extremities.
+Once we have fetched all of its `prev_events`, it's unmarked as a backwards
+extremity and those `prev_events` become the new backwards extremities (unless
+we have already persisted them). Also in reality, we backfill in batches of 20
+events or so, so only the `prev_events` of the last oldest-in-time event will
+become the backwards extremeties.
 
 
 ## Outliers
@@ -61,12 +65,11 @@ For example, when we fetch the event auth chain or state for a given event, we
 mark all of those claimed auth events as outliers because we haven't done the
 state calculation ourself.
 
+Outliers are sometimes referred to as floating outliers but there is no
+distinction between a normal and floating outlier. The floating descriptor just
+comes from the fact that all outliers are an arbitrary floating event in the DAG
+as opposed to being inline with the current DAG.
 
-### Floating outlier
-
-A floating `outlier` is an arbitrary floating event in the DAG (as opposed to
-being inline with the current DAG). This happens when an event doesn't have
-any `prev_events` or has `prev_events` that don't exist.
 
 
 ## State groups
@@ -81,3 +84,4 @@ mappings of `event_id -> state_group` and `state_group -> state`.
 ### Stage group edges
 
 TODO: `state_group_edges` is a further optimization...
+      notes from @Azrenbeth, https://pastebin.com/seUGVGeT