LI-306: Containerization of Local Subgraph Deployment (#1)

* Initial containerization refactor

* Move scripts out of docker folder since independent of Docker

* Update README with subgraph endpoint access instructions

* Refactored containerization to include Hardhat and graph-node deployments, and default env var file

Author: comc

.eslintrc.js

@@ -17,8 +17,9 @@ node_modules/
 # Yarn Integrity file
 .yarn-integrity
 
-# dotenv environment variables file
+# dotenv environment variable file
 .env
+!docker/.env
 
 # VS Code config
 .vscode/

.gitignore

@@ -2,150 +2,74 @@
 
 Indexer of Set Protocol v2 events. Built on [The Graph](https://thegraph.com/).
 
-## Setup
+## SETUP
 
-Requirements:
+### Requirements:
 
-- Bash >= 5.0 [Mac](https://gist.github.com/Rican7/44081a9806595704fa7b289c32fcd62c) / [Win](https://nickjanetakis.com/blog/a-linux-dev-environment-on-windows-with-wsl-2-docker-desktop-and-more)
-- [Node.js >= 14.0](https://nodejs.org/en/download/)
-- Yarn 1.x (`npm install -g yarn`)
-- [Docker >= 19.0](https://www.docker.com/get-started)
+- Docker >= 20.10
 
-Steps:
+### Local Deployment (Hardhat)
 
-1. `git clone https://github.com/SetProtocol/set-protocol-v2-subgraph.git && cd set-protocol-v2-subgraph`
-2. `yarn install`
-3. `yarn gen-deployment <NETWORK_NAME>` hardhat or mainnet
-4. (If deploying to hosted service) `yarn graph auth https://api.thegraph.com/deploy/ <ACCESS_TOKEN>`
-5. `graph deploy --product hosted-service justinkchen/set-protocol-v2-matic` (replace set-protocol-v2-matic with whatever graph is being deployed)
+1. Build the subgraph Docker image
 
-Current subgraphs:
-- justinkchen/set-protocol-v2 (Ethereum Mainnet)
-- justinkchen/set-protocol-v2-staging (Ethereum Mainnet)
-- justinkchen/set-protocol-v2-matic (Matic)
-- justinkchen/set-protocol-v2-matic-staging (Matic)
-- justinkchen/set-protocol-v2-arbitrum (Arbitrum)
-- justinkchen/set-protocol-v2-arbitrum-staging (Arbitrum)
-- justinkchen/set-protocol-v2-avalanche (Avalanche)
-- justinkchen/set-protocol-v2-avax-staging (Avalanche) 
-- justinkchen/set-protocol-v2-optimism-staging (Optimism)
+    `task docker-build`
 
-Using Graph Subgraph Studio (Future steps once Subgraph studio supports more networks):
-1. `graph auth --studio <API KEY>` (see more at https://thegraph.com/studio/subgraph/set-protocol-v2/)
-2. `yarn gen-deployment <NETWORK_NAME>` hardhat or mainnet or matic or staging-mainnet
-3. `cp generated/addresses.ts .` Copy generated addresses file to root location
-3. `graph codegen && graph build`
-4. `graph deploy --studio set-protocol-v2` for mainnet or `graph deploy --studio set-protocol-v2-matic` for Matic/Polygon or `graph deploy --studio set-protocol-v2-staging` for staging mainnet
+1. Deploy a Hardhat node (use `task deploy-hardhat -- detach` to start container in detached mode)
 
+    `task deploy-hardhat`
 
-## Commands
+1. Monitor the Hardhat node until fully deployed and tests are executed, e.g., if running detached use
 
-Usage:
+    `docker logs docker-hardhat-1 --follow`
 
-`yarn <COMMAND>`
+1. Compile the Set Protocol ABIs
 
-Commands:
+    `task gen-abi`
 
-`build` - Compile subgraph
+1. Deploy local subgraph
 
-`codegen` - Generate types (if schema or ABI changed)
+    `task deploy-local`
 
-`deploy-local` - Deploy subgraph to localhost
+1. Once deployed, query the subgraph in the browser at (by default) http://127.0.0.1:8000/subgraphs/name/SetProtocol/set-protocol-v2
 
-`deploy-to <IP>` - Deploy subgraph to Graph Node by IP
+### [TO-DO] External Deployment (Graph Hosted Service / Subgraph Studio)
 
-`deploy-hosted` - Deploy subgraph to hosted service
+TBD
 
-`gen-abis` - Pull contract ABIs from Set Protocol V2 repo (only needed if changed)
+## USAGE
 
-`gen-deployment <NETWORK_NAME>` - Generate deployment-specific files
+Available tasks for this project:
 
-`lint` - Format code
+| COMMAND [OPTS]               | DESCRIPTION |
+|------------------------------|---------------------------------------------------------------------------------|
+| `clean [-- all]`             | Clean up local subgraph deployment; `all` arg additionally removes all volumes and the Hardhat node. |
+| `deploy-hardhat [-- detach]` | Deploy a Hardhat node and subgraph tests; `detach` runs container detached. |
+| `deploy-hosted [-- detach]`  | Build and deploy subgraph on Hosted Service; `detach` runs container detached. |
+| `deploy-local [-- detach]`   | Build and deploy subgraph on local network; `detach` runs container detached. |
+| `destroy-hardhat`            | Tear down the deployed Hardhat node. |
+| `docker-build`               | Build subgraph Docker image on defined node version base (default: 16-slim). |
+| `gen-abi`                    | Pull latest Set Protocol ABIs into the build environment. |
 
-## Local development (hardhat)
+## [TO-DO] ADVANCED DEPLOYMENT GUIDES
 
-### Clone Set Protocol v2 fork
+TBD: Things to be covered in this section
 
-In separate directory:
+- custom override of args (requires custom untracked .env configs or CLI arg overrides)
+- the [Set Protocol V2 repo](https://github.com/SetProtocol/set-protocol-v2.git) currently requires node <= 16; therefore, Node 16 is the default target base image used in the Subgraph Docker image.
+- node dependencies installation into a named Docker volume, build the subgraph mappings, create the subgraph, then deploy it to the target graph node and IPFS database. Not that consecutive runs of this command will use the existing named volume for the `node_modules` unless it is manually removed.
+- the subgraph is deployed to the local graph-node and IPFS containers defined in the `subgraph.env` file. The subgraph endpoints given by the deployment are relative to the Docker container and not accessible externally as given. To access the subgraph, instead navigate to: http://127.0.0.1:8000/subgraphs/name/SetProtocol/setprotocolv2.
 
-1. `git clone https://github.com/jgrizzled/set-protocol-v2.git -b subgraph-dev && cd set-protocol-v2`
-2. `cp .env.default .env`
-3. `yarn install`
+## [TO-DO] SUBGRAPH DEVELOPMENT
 
-To run the hardhat node:
+### Tutorial
 
-1. `yarn chain --hostname 0.0.0.0`
-2. Wait for node to start
-3. (in separate terminal) `yarn deploy-mock`
+TBD
 
-### Install Graph Node
+### Reference Guide
 
-In separate directory:
+To Be Completed
 
-1. `git clone -q --depth=1 https://github.com/graphprotocol/graph-node.git && cd graph-node/docker`
-2. Edit line 20 of docker-compose.yml to `ethereum: hardhat:http://host.docker.internal:8545` (May need to replace host.docker.internal with LAN IP)
-3. Run with `sudo docker-compose up`
-
-`rm -rf ./data` and restart containers if blockchain changes.
-
-`sudo docker-compose build` if updated via `git pull`
-
-### Deploy subgraph locally
-
-From subgraph repo:
-
-1. `yarn gen-deployment hardhat`
-2. `yarn deploy-local`
-
-Graph-node may take a few minutes to sync the subgraph.
-
-Visit `http://localhost:8000/subgraphs/name/desert-defi/setprotocolv2/graphql` to view subgraph data
-
-## Syncing to mainnet
-
-Syncing the subgraph to mainnet requires an Ethereum archive node. We recommend [Turbogeth](https://github.com/ledgerwatch/turbo-geth) as it syncs fast and will fit on a 2TB SSD at present. Note that it takes a few hours for the subgraph to sync and re-deploying the subgraph will re-sync from scratch.
-
-### Turbogeth
-
-In separate directory:
-
-1. `git clone -q --depth=1 https://github.com/ledgerwatch/turbo-geth.git && cd turbo-geth`
-2. `sudo docker-compose build` (re-run if updated via `git pull`)
-3. `sudo XDG_DATA_HOME=/preferred/data/folder docker-compose up -d`
-
-Watch logs with:
-
-`sudo docker logs $(sudo docker container ls | grep tg | cut -d' ' -f1) -f --since 10m`
-
-### Graph Node
-
-From graph-node repo:
-
-Ensure `docker/docker-compose.yml` is configured for mainnet on line 20: `ethereum: mainnet:`
-
-If you previously synced to hardhat, `rm -rf docker/data`.
-
-Don't start Graph Node until Turbogeth is fully synced.
-
-Start with:
-
-1. `cd docker`
-2. `sudo docker-compose up -d`
-
-Watch logs with:
-
-`sudo docker logs $(sudo docker container ls | grep graph-node | cut -d' ' -f1) -f --since 10m`
-
-### Deploy to mainnet
-
-From subgraph repo:
-
-1. `yarn gen-deployment mainnet`
-2. `yarn deploy-local` or `yarn deploy-to <IP>` if Graph Node on another machine.
-
-Watch graph-node logs for sync status and errors. Subgraph URL same as above.
-
-## Files
+#### Key Files
 
 `schema.graphql` - Subgraph schema
 
@@ -159,25 +83,23 @@ Watch graph-node logs for sync status and errors. Subgraph URL same as above.
 
 `src/entities/` - Entity helper functions
 
-## Design
-
-### Historical Entities
+#### Historical Entities
 
-#### Events
+##### Events
 
 Individual transaction log events referenced by TxID and log index. Can have multiple events of the same type with the same timestamp (IE same block). Ex TradeEvent.
 
-#### States
+##### States
 
 Final state of an entity at the end of a block. Referenced by block number. Multiple events in the same block will be consolidated to one state update. Ex TotalSupplyState.
 
 Events are better for tracking important actions while States are better for timeseries data in which multiple data points per timestamp would be inconveinent.
 
-### Current Entities
+##### Current Entities
 
 Tracks the most recent state of a contract. Usually references the latest event or state update entity.
 
-### Entity Helpers
+##### Entity Helpers
 
 Most of the logic regarding manipulation of entities should be in helper functions. Create helper functions based on the following nomenclature as needed.
 
@@ -190,16 +112,16 @@ Prefixes:
 - ensure: create or update existing entity. Useful for states
 - track: execute contract call and store result in new entity. Useful for data that is not provided via event logs.
 
-### Event Handlers
+#### Event/Call/Block Handlers
 
-Process event data and call entity helper functions to update the subgraph. Must register event handlers in `templates/subgraph.yaml`.
+Process events, function calls, and block data to update the subgraph. Must register handlers in `templates/subgraph.yaml`.
 
-### Template spawners
+#### Template spawners
 
 Not all contract addresses are known at the time of subgraph deployment. To track contracts as they are deployed, use contract templates.
 Tell the subgraph to watch a newly created contract by calling create() on imports from `generated/templates`. Ex the SetToken factory contract (SetTokenCreator) emits an event when a new SetToken is created, so we register that address as a new SetToken contract to watch. Templates are defined in `templates/subgraph.yaml`.
 
-## Reference
+## References
 
 [Discord: Index Co-op #set-subgraph](https://discord.gg/8FYPP7ebbw)