A modular indexer system for ethereum and other blockchain following the same RPC standard.
Git Repo: https://github.com/jolly-roger-eth/ethereum-indexer
You can find some demoes in the examples folder
And here is the Documentation Website
- written in typescript, run both in a browser context and node
- modular : you can use the part you want
- designed to run in-browser and relies only on EIP-1193
- when run on a server, you can hook your own database module to store the indexer process's result
- A json object can be used as DB (useful for in-browser indexing).
- Supports Reorg
- Supports caching
The main reason for building etherfold is to have the indexing be performed in a fully decentralised manner: in the client.
This obviously does not scale for all use-case: try indexing all ERC20/ERC721 and the amount of log to fetch is too big to be useful, in a browser context.
But for some use case it is actually possible and efficient. This is the case where the amount of event is bounded or its scale rate is limited.
It is for example possible to instead of indexing all ERC721, to simply index the ERC721 of the current account.
Due to the limitation of EIP-1193 (no batch request), anything that needs an extra request per block or per transaction is expensive in the browser, so indexer processors are expected to not make use of such features.
Using them would work in a server environment where results can be cached across load-balanced instances, but in a browser environment where each user would have its own instance, they would slow down the indexing too much.
Block timestamps are no longer one of those features. blockTimestamp is now part of the log object itself, standardised in ethereum/execution-apis#639 and served by go-ethereum (>= 1.16.0), reth, besu, erigon, anvil and ethereumjs, so a processor can read event.blockTimestamp for free. It is not universal — Hardhat's EDR does not emit it as of hardhat 3.14.0 (edr#1643) — so the field stays optional, and stream.alwaysFetchTimestamps remains as the fallback: it now fetches only the blocks whose logs arrived without one, and costs nothing on a node that supplies them.
Having said that an hybrid approach is possible where a server index and the in-browser indexer exists only as a backup when every server instances are unavailable expect for a cache (which could even be shared across user in p2p manner).
It is also worth noting that for an indexer to work, it needs to index all events and depending on the games or applications, this might not fit in memory or in browser storage qutoa. For such case, there is no other option to have that handled by a remote service.
install @etherfold/browser and @etherfold/js-processor
npm i @etherfold/browser @etherfold/js-processor
If you use react, here is a mostly self-contained example from App.tsx
import './App.css';
import {fromJSProcessor, JSProcessor} from '@etherfold/js-processor';
import {createIndexerState, keepStateOnIndexedDB} from '@etherfold/browser';
import {connect} from './utils/web3';
import react from 'react';
// we need the contract info
// the abi will be used by the processor to have its type generated, allowing you to get type-safety
// the adress will be given to the indexer, so it index only this contract
// the startBlock field allow to tell the indexer to start indexing from that point only
// here it is the block at which the contract was deployed
const contract = {
abi: [
{
anonymous: false,
inputs: [
{
indexed: true,
name: 'user',
type: 'address',
},
{
indexed: false,
name: 'message',
type: 'string',
},
],
name: 'MessageChanged',
type: 'event',
},
],
address: '0x21d366ee3BbF67AB057c517380D37E54fFd9dfC0',
startBlock: 3040661,
} as const;
// we define the type of the state computed by the processor
// we can also declare it inline in the generic type of JSProcessor
type State = {greetings: {account: `0x${string}`; message: string}[]};
// the processor is given the type of the ABI as Generic type to get generated
// it also specify the type which represent the current state
const processor: JSProcessor<typeof contract.abi, State> = {
// the version is REQUIRED, and ideally you would generate it so that it changes for each change
// when a version changes, the indexer will detect that and clear the state
// if it has the event stream cached, it will repopulate the state automatically
// if you edit a handler and forget to bump it, the indexer says so at load time (an error-level
// drift report, plus the `onProcessorDrift` callback); set `strictProcessorDrift: true` in the
// indexer config to refuse to start instead of merely reporting
version: '1.0.1',
// this function set the starting state
// this allow the app to always have access to a state, no undefined needed
construct() {
return {greetings: []};
},
// each event has an associated on<EventName> function which is given both the current state and the typed event
// each event's argument can be accessed via the `args` field
// it then modify the state as it wishes
// behind the scene, the JSProcessor will handle reorg by reverting and applying new events automatically
onMessageChanged(state, event) {
const greetingFound = state.greetings.find((v) => v.account === event.args.user);
if (greetingFound) {
greetingFound.message = event.args.message;
} else {
state.greetings.push({
message: event.args.message,
account: event.args.user,
});
}
},
};
// we setup the indexer via a call to `createIndexerState`
// this setup a set of observable (subscribe pattern)
// including one for the current state (computed by the processor above)
// and one for the syncing
// we then call `.withHooks(react)` to transform these observable in react hooks ready to be used.
const {init, useState, useSyncing, startAutoIndexing} = createIndexerState(fromJSProcessor(processor)(), {
keepState: keepStateOnIndexedDB('basic') as any,
}).withHooks(react);
// we now need to get a handle on a ethereum provider
// for this app we are simply using window.ethereum
const ethereum = (window as any).ethereum;
// but to not trigger a metamask popup right away we wrap that in a function to be called via a click of a button
function start() {
if (ethereum) {
// here we first connect it to the chain of our choice and then initialise the indexer
// see ./utils/web3
connect(ethereum, {
chain: {
chainId: '11155111',
chainName: 'Sepolia',
rpcUrls: ['https://rpc.sepolia.org'],
nativeCurrency: {name: 'Sepolia Ether', symbol: 'SEP', decimals: 18},
blockExplorerUrls: ['https://sepolia.etherscan.io'],
},
}).then(({ethereum}) => {
// we already setup the processor
// now we need to initialise the indexer with
// - an EIP-1193 provider (window.ethereum here)
// - source config which includes the chainId and the list of contracts (abi,address. startBlock)
init({
provider: ethereum,
source: {chainId: '11155111', contracts: [contract]},
}).then(() => {
// this automatically index on a timer
// alternatively you can call `indexMore` or `indexMoreAndCatchupIfNeeded`, both available from the return value of `createIndexerState`
// startAutoIndexing is easier but manually calling `indexMore` or `indexMoreAndCatchupIfNeeded` is better
// this is because you can call them for every `newHeads` eth_subscribe message
startAutoIndexing();
});
});
}
}
function App() {
// we use the hooks to get the latest state and make sure react update the values as they changes
const $state = useState();
const $syncing = useSyncing();
if (!ethereum) {
return (
<div className="App">
<h1>Indexing a basic example</h1>
<p>To test this app, you need to have a ethereum wallet installed</p>
</div>
);
}
// we have various variable to check the status of the indexer
// here we can act on whether the indexer is still waiting to be provided an EIP-1193 provider
if ($syncing.waitingForProvider) {
return (
<div className="App">
<h1>Indexing a basic example</h1>
<button onClick={start} style={{backgroundColor: '#45ffbb', color: 'black'}}>
Start
</button>
</div>
);
}
// here we add a progress bar indicating the progress of the indexer
return (
<div className="App">
<h1>Indexing a basic example</h1>
<p>{$syncing.lastSync?.syncPercentage || 0}</p>
{$syncing.lastSync ? (
<progress value={($syncing.lastSync.syncPercentage || 0) / 100} style={{width: '100%'}} />
) : (
<p>Please wait...</p>
)}
<div>
{$state.greetings.map((greeting) => (
<p key={greeting.account}>{greeting.message}</p>
))}
</div>
</div>
);
}
export default App;