Skip to content

README.md

Latest commit

 

History

History
114 lines (80 loc) · 8.22 KB

README.md

File metadata and controls

114 lines (80 loc) · 8.22 KB

Console Indexer

How to run

  1. Make sure you have a valid Akash database first.
  2. Create a .env file with the necessary environment variables.
  3. Run npm install to install dependencies.
  4. Start the app with npm start.

You can make sure the api is working by accessing the status endpoint: http://localhost:3079/status

Environment Variables

Name Value Note
HEALTH_CHECKS_ENABLED true or false Specify if the Scheduler should send health check pings.
SENTRY_DSN ex: "https://1234...789@z645.ingest.sentry.io/1234" Sentry DSN used when initializing Sentry
HEALTHCHECKS_SYNC_BLOCKS ex: 041b2561-be28-4a36-bb3f-36a68f86224e HealthChecks.io check ID for the SyncBlocks task.
HEALTHCHECKS_SYNC_AKT_PRICE_HISTORY ex: 041b2561-be28-4a36-bb3f-36a68f86224e HealthChecks.io check ID for the SyncAKTPriceHistory task.
HEALTHCHECKS_SYNC_PROVIDER_INFO ex: 041b2561-be28-4a36-bb3f-36a68f86224e HealthChecks.io check ID for the SyncProviderInfo task.
HEALTHCHECKS_SYNC_KEYBASE_INFO ex: 041b2561-be28-4a36-bb3f-36a68f86224e HealthChecks.io check ID for the SyncKeybaseInfo task.
AKASH_DATABASE_CS ex: postgres://user:password@localhost:5432/console-akash Akash Database Connection String
ACTIVE_CHAIN ex: akash Chain code from chainDefinitions.ts
KEEP_CACHE true or false Specify if the block & block response cache should be kept on drive. Takes a lot of space, but allow rebuilding the database without redownloading every blocks.
STANDBY true or false If true, indexer will not start. Useful for stopping an indexer deployed on akash without needing to close the lease.
DATA_FOLDER ex: ./data/ Directory where block cache and node statuses should be saved. Defaults to ./data/.

Scheduled Tasks

In the startScheduler method we register all the task that must run at an interval.
The scheduler is responsible for running the different tasks at the correct interval and keeping track of any errors that may occurs.
Tasks can be configured to report their execution to healthcheck endpoints automatically (healthchecks.io or sentry.io)

Task Interval Description
Sync Price History 1 hour Responsible for fetching the latest token price information from CoinGecko.
Sync Providers Info 15 minutes Responsible for querying the /status endpoint of every akash provider to track their uptime and available resources.
Provider IP Lookup 30 minutes Responsible for updating the akash providers location based on the node's ip address.
Sync Keybase Info 6 hours Responsible for fetching validator names and picture from keybase.io.
Address Balance Monitor 10 minutes Responsible for updating tracked address balances (Blockspy Specific)
Deployment Balance Monitor 10 minutes Responsible for updating tracked deployment balances. This is blockspy specific and not used in the deploy tool.
Sync Blocks 7 seconds Responsible for downloading new blocks and passing them through the correct indexers.

Data Flow

Index Data Flow

Database Migrations

If you are running from an old database snapshot, you may need to run database migrations to ensure your schema is up to date. See UPGRADE.md for migration instructions.

Step #1 - Download Blocks

Using our nodeAccessor missing blocks are downloaded from RPC nodes. A list of rpc node is setup in the chainDefinition file and their status is then kept updated (earliest/latest available block, rate limiting, etc). The blocks are saved on disk inside a leveldb database (see File Structure).

See downloadBlocks().

Step #2 - Insert Blocks

Blocks are parsed and inserted into their corresponding tables with basic information (block,transaction,message). The isProcessed flag is set to false marking them for processing.

See insertBlocks().

Step #3 - Processing

The processing step loops through blocks that have not been processed yet and run indexers. For every messages it will run any message handler that needs to be run for this message type. Then the afterEveryBlock and afterEveryTransaction hooks are executed and the isProcessed flags are set to true.

See processMessages()

Indexers

Indexers are defined in the /src/indexers directory. Some are cosmos generic and some are akash specific. See Create a new indexer for more info on how to add a new indexer.

  • Validator Indexer is used to keep track of validator informations. It listens to MsgCreateValidator and MsgEditValidator validators and keep track of the every on-chain details about a validator.
  • Message Addresses Indexer is used to keep track of addresses involved in a transaction. This includes every tx signers and sender/receiver of tokens. In the case of token movement this also saves the transferred amount on the message entity.
  • Akash Stats Indexer is used to track every akash specific stats.
    • Bids
    • Leases
    • Deployments
    • Providers

Create a new indexer

To track a new data point based on incoming blocks you need to create a new indexer file and extand the Indexer class.

Here are the properties/methods that can be overriden :

  • msgHandlers - Property mapping msg type (ex: /akash.market.v1beta1.MsgCreateLease) to a message handler. During syncing the message handlers will be called with the decoded message, the height and the Message db entity.
  • initCache() - Can be used to load data in cache before syncing. It will be called once at each start of the application.
  • createTables()/dropTables() - This is for initializing the tables associated with your indexer. It will be called when creating or recreating the database only.
  • seed() - Used to index data from the genesis file. Will be called with the parsed genesis file once when syncing a new chain from block 1.
  • afterEveryBlock() - Called after every new blocks with the current block info and the previous block info.
  • afterEveryTransaction() - Called after every new transactions with the raw decoded tx object and the Transaction db entity.

Block Cache Structure

/data/[chain]/

  • blockResults.db - LevelDB Database containing block results information from RPC nodes
  • blocks.db - LevelDB Database containing block information from RPC nodes
  • genesis.json - If syncing from block 1 the chain's genesis file will be downloaded here.
  • nodeStatus.json - Saved status of every rpc node so that invalid/rate limited nodes are remembered after a restart (if there is persistent storage)

Drizzle Migrations

You may find drizzle present in the app as a development dependency. Although we want to migrate to it eventually, at the moment it is added to ease migrations for the Indexer database. It was used to introspect existing schema and is used to add more migrations and apply them in test environment.