NEAR Indexer concept page and tutorial

docs/data-infrastructure/near-indexer.md

@@ -0,0 +1,51 @@
+---
+id: near-indexer
+title: NEAR Indexer
+description: "NEAR Indexer is a micro-framework that provides a stream of blocks recorded on the NEAR network. It is designed to handle real-time events on the blockchain."
+---
+
+# NEAR Indexer
+
+As scaling dApps enter NEAR’s mainnet, an issue may arise: how do they quickly and efficiently access state from our deployed smart contracts, and cut out the cruft? Contracts may grow to have complex data structures and querying the network RPC may not be the optimal way to access state data.
+
+The [NEAR Indexer](https://github.com/near/nearcore/tree/master/chain/indexer) is a micro-framework specifically designed to handle real-time events on the blockchain, allowing to capture and index streams of blocks in a customized manner.
+
+With the NEAR Indexer, developers can perform both high-level data aggregation and low-level introspection of blockchain events.
+
+:::tip
+
+For those searching to not build their own indexer, the [NEAR Lake Framework](./near-lake-framework.md) provides a simpler way to access blockchain data in real-time
+
+:::
+
+---
+
+## How It Works
+
+The NEAR Indexer works by **running a node** and processing blocks as they are added to the blockchain. The framework provides a stream of blocks, allowing developers to subscribe and process these blocks in real-time.
+
+:::tip
+
+Learn how to run it following the [tutorial](./tutorials/near-indexer.md).
+
+:::
+
+---
+
+## Comparison with [NEAR Indexer Framework](near-indexer.md)
+
+Comparing to [NEAR Lake Framework](./near-lake-framework.md) in terms of latency the NEAR Indexer is significantly faster as it reads data directly from the blockchain the same way as RPC nodes do.
+
+Feature | Indexer Framework | Lake Framework
+------- | ----------------- | --------------
+Allows to follow the blocks and transactions in the NEAR Protocol | **Yes** | **Yes**<br />(but only mainnet and testnet networks)
+Decentralized | **Yes** | No<br />(Pagoda Inc dumps the blocks to AWS S3)
+Reaction time (end-to-end) | minimum 3.8s (estimated average 5-7s) | [minimum 3.9s (estimated average 6-8s)](#latency)
+Reaction time (framework overhead only) | 0.1s | 0.2-2.2s
+Estimated cost of infrastructure | [$500+/mo](https://near-nodes.io/rpc/hardware-rpc) | [**$20/mo**](#cost)
+Ease of maintenance | Advanced<br />(need to follow every nearcore upgrade, and sync state) | **Easy**<br />(deploy once and forget)
+How long will it take to start? | days (on mainnet/testnet) | **seconds**
+Ease of local development | Advanced<br />(localnet is a good option, but testing on testnet/mainnet is too heavy) | **Easy**<br />(see [tutorials](./tutorials/near-lake-state-changes-indexer.md))
+Programming languages that a custom indexer can be implemented with | Rust only | **Any**<br />(currently, helper packages are released in [Python](http://pypi.org/project/near-lake-framework), [JavaScript](https://www.npmjs.com/package/near-lake-framework), and [Rust](https://crates.io/crates/near-lake-framework))
+
+---

docs/data-infrastructure/near-lake-framework.md

@@ -88,7 +88,7 @@ $17,20 + $21,60 = $30,16
 
 ---
 
-## Comparison with [NEAR Indexer Framework](https://github.com/near/nearcore/tree/master/chain/indexer)
+## Comparison with [NEAR Indexer Framework](near-indexer.md)
 
 NEAR Lake Framework is reading data from AWS S3, while the NEAR Indexer is running a full node and reading data from the blockchain directly in real time.
 

docs/data-infrastructure/tutorials/near-indexer.md

@@ -0,0 +1,231 @@
+---
+id: listen-to-realtime-events
+title: Listen to Realtime Events
+description: "This tutorial will guide you through building an indexer using the NEAR Indexer Framework. The indexer will listen for FunctionCalls on a specific contract and log the details of each call."
+---
+
+import {Github} from "@site/src/components/UI/Codetabs"
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import MovingForwardSupportSection from '@site/src/components/MovingForwardSupportSection';
+
+In this tutorial, we will build an indexer using the NEAR Indexer Framework. The indexer will listen realtime events from NEAR blockchain for FunctionCalls on a specific contract and log the details of each call.
+
+To get our indexer up and running we will need two steps:
+
+1. To [initialize](#initialization) the indexer
+2. To [start it](#run)
+
+The full source code for the indexer example is available in the [GitHub repository](https://github.com/near-examples/near-indexer?tab=readme-ov-file).
+
+---
+
+## Initialization
+
+In order for our indexer to process blocks it needs to join the NEAR network as a node. To do that, we need first to initialize it, which will download the blockchain `genesis` config, and create a `key` for our node to communicate with other nodes:
+
+<Tabs groupId="code-tabs">
+    <TabItem value="localnet" label="Localnet" default>
+      ```bash
+        cargo run --release -- --home-dir ~/.near/localnet init
+      ```
+    </TabItem>
+    <TabItem value="testnet" label="Testnet" default>
+      ```bash
+        cargo run --release -- --home-dir ~/.near/testnet init --chain-id testnet --download-config rpc --download-genesis
+      ```
+    </TabItem>
+    <TabItem value="mainnet" label="Mainnet" default>
+      ```bash
+        cargo run --release -- --home-dir ~/.near/mainnet init --chain-id mainnet --download-config rpc --download-genesis
+      ```
+    </TabItem>
+</Tabs>
+
+Depending on the network we want to connect, the keys will be created in different folders (`~/.near/<network>`).
+
+#### Config File
+
+A configuration file (`~/.near/<network>/config.json`) is created automatically, whoever, it is recommended to replace with one of the following ones, intended for RPC nodes:
+
+- [testnet config.json](https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/testnet/rpc/config.json)
+- [mainnet config.json](https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/mainnet/rpc/config.json)
+
+:::note Configuration Options
+
+See the [Custom Configuration](#custom-configuration) section below to learn more about further configuration options.
+
+:::
+
+---
+
+## Starting the Indexer
+
+After we finish initializing the indexer, and configuring it, we can start it by running the following command:
+
+<Tabs groupId="code-tabs">
+    <TabItem value="localnet" label="Localnet" default>
+      ```bash
+        cargo run --release -- --home-dir ~/.near/localnet --accounts bob.near --block-height 137510 run
+      ```
+    </TabItem>
+    <TabItem value="testnet" label="Testnet" default>
+      ```bash
+        cargo run --release -- --home-dir ~/.near/testnet --accounts bob.testnet --block-height 218137510 run
+      ```
+    </TabItem>
+    <TabItem value="mainnet" label="Mainnet" default>
+      ```bash
+        cargo run --release -- --home-dir ~/.near/mainnet --accounts bob.near --block-height 167668637 run
+      ```
+    </TabItem>
+</Tabs>
+
+---
+
+## Parsing the Block Data
+
+From the block data, we can access the transactions, their receipts and actions. In this example, we will look for FunctionCall actions on a specific contract and log the details of each call.
+
+<Github fname="main.rs" language="rust"
+        url="https://github.com/near-examples/near-indexer/blob/main/src/main.rs"
+        start="71" end="154" />
+
+---
+
+## Custom Configuration
+
+By default, nearcore is configured to do as little work as possible while still operating on an up-to-date state. Indexers may have different requirements, so you might need to tweak the configuration based on yours.
+
+### Shards/Accounts to Track
+
+We need to ensure that NEAR Indexer follows all the necessary shards, so by default the `"tracked_shards_config"` is set to `"AllShards"`. The most common tweak you might need to apply is listing to specific shards; to do that, lists all the shard UIDs you want to track in the `"tracked_shards_config"` section (`~/.near/<network>/config.json` file):
+
+```json
+...
+"tracked_shards_config": {
+       "Shards": [
+              "s3.v3",
+              "s4.v3"
+       ]
+},
+...
+```
+Or, if you want to track specific accounts:
+
+```json
+...
+"tracked_shards_config": {
+       "Accounts": [
+              "account_a",
+              "account_b"
+       ]
+},
+...
+```
+
+<hr class="subsection" />
+
+### Sync Mode
+You can choose Indexer Framework sync mode by setting what to stream:
+
+- LatestSynced - Real-time syncing, always taking the latest finalized block to stream
+- FromInterruption - Starts syncing from the block NEAR Indexer was interrupted last time
+- BlockHeight(u64) - Specific block height to start syncing from.
+
+<Github fname="main.rs" language="rust"
+        url="https://github.com/near-examples/near-indexer/blob/main/src/main.rs"
+        start="34" end="34" />
+
+<hr class="subsection" />
+
+### Streaming Mode
+You can choose Indexer Framework streaming mode by setting what to stream:
+
+- StreamWhileSyncing - Stream while node is syncing
+- WaitForFullSync - Don't stream until the node is fully synced
+
+<Github fname="main.rs" language="rust"
+        url="https://github.com/near-examples/near-indexer/blob/main/src/main.rs"
+        start="35" end="35" />
+
+<hr class="subsection" />
+
+### Finality
+You can choose finality level at which blocks are streamed:
+
+- None - `optimistic`, a block that (though unlikely) might be skipped
+- DoomSlug - `near-final`, a block that is irreversible, unless at least one block producer is slashed
+- Final - `final`, the block is final and irreversible.
+
+<Github fname="main.rs" language="rust"
+        url="https://github.com/near-examples/near-indexer/blob/main/src/main.rs"
+        start="36" end="36" />
+
+<hr class="subsection" />
+
+### Boot Nodes
+If your node can't find any peers to connect to, you can manually specify some boot nodes in the `config.json` file. You can get a list of active peers for your network by running the following command:
+
+<Tabs groupId="code-tabs">
+    <TabItem value="testnet" label="Testnet" default>
+      ```bash
+      curl -X POST https://rpc.testnet.near.org \
+        -H "Content-Type: application/json" \
+        -d '{
+              "jsonrpc": "2.0",
+              "method": "network_info",
+              "params": [],
+              "id": "dontcare"
+            }' | \
+      jq '.result.active_peers as $list1 | .result.known_producers as $list2 |
+      $list1[] as $active_peer | $list2[] |
+      select(.peer_id == $active_peer.id) |
+      "\(.peer_id)@\($active_peer.addr)"' |\
+      awk 'NR>2 {print ","} length($0) {print p} {p=$0}' ORS="" | sed 's/"//g'
+      ```
+    </TabItem>
+    <TabItem value="mainnet" label="Mainnet" default>
+      ```bash
+      curl -X POST https://rpc.mainnet.near.org \
+        -H "Content-Type: application/json" \
+        -d '{
+              "jsonrpc": "2.0",
+              "method": "network_info",
+              "params": [],
+              "id": "dontcare"
+            }' | \
+      jq '.result.active_peers as $list1 | .result.known_producers as $list2 |
+      $list1[] as $active_peer | $list2[] |
+      select(.peer_id == $active_peer.id) |
+      "\(.peer_id)@\($active_peer.addr)"' |\
+      awk 'NR>2 {print ","} length($0) {print p} {p=$0}' ORS="" | sed 's/"//g'
+      ```
+    </TabItem>
+</Tabs>
+
+And then add the output to the `boot_nodes` section of your `config.json` file as a string:
+
+```json
+...
+"network": {
+    "addr": "0.0.0.0:24567",
+    "boot_nodes": "ed25519:8oVENgBp6zJfnwXFe...",
+    ...
+},
+...
+```
+
+<hr class="subsection" />
+
+### Historical Data
+
+Indexer Framework also exposes access to the internal APIs (see Indexer::client_actors method), so you can fetch data about any block, transaction, etc, yet by default, nearcore is configured to remove old data (garbage collection), so querying the data that was observed a few epochs before may return an error saying that the data is not found. If you only need blocks streaming, you don't need this tweak, but if you need access to the historical data right from your Indexer, consider updating "archive" setting in config.json to true:
+
+```json
+...
+"archive": true,
+...
+```
+
+<MovingForwardSupportSection />

website/sidebars.js

@@ -478,6 +478,7 @@ const sidebar = {
               'Indexers': [
                 'data-infrastructure/indexers',
                 'data-infrastructure/near-lake-framework',
+                'data-infrastructure/near-indexer',
               ],
             },
           ]
@@ -504,8 +505,9 @@ const sidebar = {
                     'data-infrastructure/tutorials/running-near-lake/run-lake-indexer',
                     'data-infrastructure/tutorials/running-near-lake/lake-start-options',
                     'data-infrastructure/tutorials/running-near-lake/credentials',
-                  ]
-                }
+                  ],
+                },
+                "data-infrastructure/tutorials/listen-to-realtime-events"
               ],
             },
           ]