diff --git a/README.md b/README.md index 3e210d5..b90d7e1 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Insight -Insight is a blockchain data processing tool designed to fetch, process, and store on-chain data. It provides a solution for indexing blockchain data, facilitating efficient querying of transactions, logs through a simple API. +Insight is a blockchain data processing tool designed to fetch, process, and store on-chain data. It provides a solution for indexing blockchain data, facilitating efficient querying of transactions and logs through a simple API. Insight's architecture consists of five main components: architecture-diagram @@ -15,15 +15,14 @@ Insight's architecture consists of five main components: 5. **Orchestrator**: The Orchestrator is responsible for coordinating and managing the poller, failure recoverer, and committer. It initializes these components based on configuration settings and starts them concurrently, ensuring they run independently while waiting for all of them to complete their tasks. -Insight's modular architecture and configuration options allow for adaptation to various evm chains and use cases. +Insight's modular architecture and configuration options allow for adaptation to various EVM chains and use cases. ## Getting started ### Pre-requites -1. Install Golang v1.23 -2. Run the DB scripts (change the db name) -3. Create a clickhouse instance +1. Golang v1.23 +2. Clickhouse instance (`docker-compose` includes it for local development) ### Usage To run insight and the associated API, follow these steps: @@ -31,7 +30,8 @@ To run insight and the associated API, follow these steps: ``` git clone https://github.com/thirdweb-dev/insight.git ``` -2. Run the migration scripts here +2. ~~Run the migration scripts here~~ +2. Apply the migrations from [here](internal/tools/) 3. Create `config.yml` from `config.example.yml` and set the values by following the [config guide](#supported-configurations) 4. Create `secrets.yml` from `secrects.example.yml` and set the needed credentials 5. Build an instance @@ -383,26 +383,37 @@ failureRecoverer: ``` #### Storage -This application has 3 kinds of strorage. Main, Staging and Orchestrator. -Each of them takes similar configuration depending on the driver you want to use. - +This application has 3 kinds of storage: `main`, `staging` and `orchestrator`. +Each of them takes similar configuration, slightly depending on the driver you want to use. There are no defaults, so this needs to be configured. +For example, this can be a part of `config.yml`: ```yaml storage: main: - driver: "clickhouse" clickhouse: - host: "localhost" port: 3000 - user: "admin" - password: "admin" database: "base" - disableTLS: false + disableTLS: true staging: - driver: "memory" - memory: - maxItems: 1000000 + clickhouse: + port: 3000 + database: "staging" orchestrator: - driver: "memory" + memory: + maxItems: 10000 +``` +With the corresponding `secrets.yml`: +```yaml +storage: + main: + clickhouse: + host: localhost + user: admin + password: password + staging: + clickhouse: + host: localhost + username: admin + password: password ``` \ No newline at end of file diff --git a/configs/config.example.yml b/configs/config.example.yml index 47909a3..7fc79b1 100644 --- a/configs/config.example.yml +++ b/configs/config.example.yml @@ -33,11 +33,13 @@ storage: main: clickhouse: port: 9440 - database: "base" + database: "default" + disableTLS: true staging: clickhouse: port: 9440 - database: "staging" + database: "default" + disableTLS: true orchestrator: memory: maxItems: 10000 \ No newline at end of file diff --git a/configs/secrets.example.yml b/configs/secrets.example.yml index 80c12b5..f0e168d 100644 --- a/configs/secrets.example.yml +++ b/configs/secrets.example.yml @@ -6,9 +6,9 @@ storage: clickhouse: host: localhost username: admin - password: admin + password: password staging: clickhouse: host: localhost username: admin - password: admin \ No newline at end of file + password: password \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index 580ad48..ab4e55a 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -17,6 +17,10 @@ services: - "9440:9000" volumes: - clickhouse_data:/var/lib/clickhouse + environment: + CLICKHOUSE_DB: default + CLICKHOUSE_USER: admin + CLICKHOUSE_PASSWORD: password ulimits: nofile: soft: 262144