Explanation

Author: MichaelMacaulay

website/src/pages/en/subgraphs/querying/distributed-systems.mdx

@@ -2,56 +2,60 @@
 title: Distributed Systems
 ---
 
-The Graph is a protocol implemented as a distributed system.
+Distributed systems offer vast capabilities, but they also come with inherent complexities. In **The Graph**, these complexities are amplified at a global scale.
 
-Connections fail. Requests arrive out of order. Different computers with out-of-sync clocks and states process related requests. Servers restart. Re-orgs happen between requests. These problems are inherent to all distributed systems but are exacerbated in systems operating at a global scale.
+This document provides an explanation of why requests can appear inconsistent, how block reorganization (re-org) events affect data delivery, and why certain solutions exist to maintain consistency.
 
-Consider this example of what may occur if a client polls an Indexer for the latest data during a re-org.
+### Why Distributed Systems Can Appear Inconsistent
 
-1. Indexer ingests block 8
-2. Request served to the client for block 8
-3. Indexer ingests block 9
-4. Indexer ingests block 10A
-5. Request served to the client for block 10A
-6. Indexer detects reorg to 10B and rolls back 10A
-7. Request served to the client for block 9
-8. Indexer ingests block 10B
-9. Indexer ingests block 11
+Whenever different computers are working together across the globe, certain problems become unavoidable. Connections fail, computers get restarted, and clocks fall out of sync. In the context of **The Graph**, multiple Indexers ingest blocks at slightly different times, and clients can make requests to any of these Indexers. The result is that requests may arrive out of order or be answered based on different block states.
+
+A perfect example of this is the so-called "block wobble" phenomenon, where a client sees block data appear to jump forward and backward in unexpected ways. This phenomenon becomes especially noticeable during a block re-org — a situation where a previously ingested block is replaced by a different one under consensus.
+
+#### Example of Block Reorganization
+
+To understand the impact, consider a scenario where a client continuously fetches the latest block from an Indexer:
+
+1. Indexer ingests block 8  
+2. Request served to the client for block 8  
+3. Indexer ingests block 9  
+4. Indexer ingests block 10A  
+5. Request served to the client for block 10A  
+6. Indexer detects re-org to 10B and rolls back 10A  
+7. Request served to the client for block 9  
+8. Indexer ingests block 10B  
+9. Indexer ingests block 11  
 10. Request served to the client for block 11
 
-From the point of view of the Indexer, things are progressing forward logically. Time is moving forward, though we did have to roll back an uncle block and play the block under consensus forward on top of it. Along the way, the Indexer serves requests using the latest state it knows about at that time.
+From the **Indexer's viewpoint**, it sees a forward-moving progression with a brief need to roll back an invalid block. But from the **client's viewpoint**, responses seem to arrive in a puzzling order: block 8, block 10, then suddenly block 9, and finally block 11.
 
-From the point of view of the client, however, things appear chaotic. The client observes that the responses were for blocks 8, 10, 9, and 11 in that order. We call this the "block wobble" problem. When a client experiences block wobble, data may appear to contradict itself over time. The situation worsens when we consider that Indexers do not all ingest the latest blocks simultaneously, and your requests may be routed to multiple Indexers.
+This disruption can cause data to appear contradictory. The challenge is magnified when clients are routed to multiple Indexers, each of which may be at different block heights.
 
-It is the responsibility of the client and server to work together to provide consistent data to the user. Different approaches must be used depending on the desired consistency as there is no one right program for every problem.
+### Why Consistency Matters
 
-Reasoning through the implications of distributed systems is hard, but the fix may not be! We've established APIs and patterns to help you navigate some common use-cases. The following examples illustrate those patterns but still elide details required by production code (like error handling and cancellation) to not obfuscate the main ideas.
+In a distributed protocol like **The Graph**, it is the responsibility of both client and server to coordinate on a consistency strategy. Systems often require different approaches based on their tolerance for receiving out-of-date data versus their need for up-to-the-moment accuracy.
 
-## Polling for updated data
+Reasoning through these distributed-system implications is difficult, but solutions exist to reduce confusion. For instance, certain patterns and APIs ensure either forward-only progress or consistent snapshot views of data. Below, two notable methods to maintain a clearer sense of order are explored.
 
-The Graph provides the `block: { number_gte: $minBlock }` API, which ensures that the response is for a single block equal or higher to `$minBlock`. If the request is made to a `graph-node` instance and the min block is not yet synced, `graph-node` will return an error. If `graph-node` has synced min block, it will run the response for the latest block. If the request is made to an Edge & Node Gateway, the Gateway will filter out any Indexers that have not yet synced min block and make the request for the latest block the Indexer has synced.
+### Ensuring Forward-Only Progress with Block Number Constraints
 
-We can use `number_gte` to ensure that time never travels backward when polling for data in a loop. Here is an example:
+Sometimes, your system only needs to avoid "going backward" in time. **The Graph** provides the `block: { number_gte: $minBlock }` argument to help support this by guaranteeing that returned data will always be from a block number equal to or greater than a specified minimum:
 
 ```javascript
-/// Updates the protocol.paused variable to the latest
-/// known value in a loop by fetching it using The Graph.
+/// Updates the protocol.paused variable by always querying
+/// for a block at or beyond the last known block.
 async function updateProtocolPaused() {
-  // It's ok to start with minBlock at 0. The query will be served
-  // using the latest block available. Setting minBlock to 0 is the
-  // same as leaving out that argument.
   let minBlock = 0
 
   for (;;) {
-    // Schedule a promise that will be ready once
-    // the next Ethereum block will likely be available.
+    // Wait until the next block is likely available.
     const nextBlock = new Promise((f) => {
       setTimeout(f, 14000)
     })
 
     const query = `
         query GetProtocol($minBlock: Int!) {
-            protocol(block: { number_gte: $minBlock }  id: "0") {
+            protocol(block: { number_gte: $minBlock }, id: "0") {
               paused
             }
             _meta {
@@ -65,30 +69,27 @@ async function updateProtocolPaused() {
     const response = await graphql(query, variables)
     minBlock = response._meta.block.number
 
-    // TODO: Do something with the response data here instead of logging it.
+    // This ensures time never travels backward.
     console.log(response.protocol.paused)
 
-    // Sleep to wait for the next block
+    // Wait before fetching the next block.
     await nextBlock
   }
 }
 ```
 
-## Fetching a set of related items
+In an environment with multiple Indexers, the Gateway can filter out Indexers not yet at `minBlock`, ensuring you consistently move forward in time.
 
-Another use-case is retrieving a large set or, more generally, retrieving related items across multiple requests. Unlike the polling case (where the desired consistency was to move forward in time), the desired consistency is for a single point in time.
+### Achieving a Consistent View with Block Hash Constraints
 
-Here we will use the `block: { hash: $blockHash }` argument to pin all of our results to the same block.
+In other scenarios, you need to retrieve multiple related data points or perform pagination without risking differences in blocks. This is where pinning a query to a single block hash becomes critical:
 
 ```javascript
-/// Gets a list of domain names from a single block using pagination
+/// Gets a list of domain names from a single block using pagination.
 async function getDomainNames() {
-  // Set a cap on the maximum number of items to pull.
   let pages = 5
   const perPage = 1000
 
-  // The first query will get the first page of results and also get the block
-  // hash so that the remainder of the queries are consistent with the first.
   const listDomainsQuery = `
     query ListDomains($perPage: Int!) {
         domains(first: $perPage) {
@@ -106,29 +107,33 @@ async function getDomainNames() {
   let result = data.domains.map((d) => d.name)
   let blockHash = data._meta.block.hash
 
-  let query
-  // Continue fetching additional pages until either we run into the limit of
-  // 5 pages total (specified above) or we know we have reached the last page
-  // because the page has fewer entities than a full page.
   while (data.domains.length == perPage && --pages) {
     let lastID = data.domains[data.domains.length - 1].id
-    query = `
+    let query = `
         query ListDomains($perPage: Int!, $lastID: ID!, $blockHash: Bytes!) {
-            domains(first: $perPage, where: { id_gt: $lastID }, block: { hash: $blockHash }) {
+            domains(
+              first: $perPage,
+              where: { id_gt: $lastID },
+              block: { hash: $blockHash }
+            ) {
                 name
                 id
             }
         }`
 
     data = await graphql(query, { perPage, lastID, blockHash })
 
-    // Accumulate domain names into the result
     for (domain of data.domains) {
       result.push(domain.name)
     }
   }
+
   return result
 }
 ```
 
-Note that in case of a re-org, the client will need to retry from the first request to update the block hash to a non-uncle block.
+By pinning queries to a single block hash, all responses relate to the same point in time, eliminating any variation caused by a re-org that might occur if different blocks were referenced. However, if a re-org does affect that chosen block, the client must repeat the process with a new canonical block hash.
+
+### Final Thoughts on Distributed Consistency
+
+Distributed systems can seem unpredictable, but understanding the root causes of events like out-of-order requests and block reorganizations helps clarify why the data may appear contradictory. By using the patterns described above, both crossing block boundaries forward in time and maintaining a single consistent block snapshot become possible strategies in managing this inherent complexity.