Lev 64 update subgraph schema to capture new (#2)

* Subgraph schema update to capture new events, and deployment refactor to support rapid subgraph development and testing

* Resolved local Hardhat network state deployment and updated subgraph schema to index new StreamingFeeModule and TradeModule events

* Add subgraph query example

* [lev-173-subgraph-migration-rollout] Added old subgraph schema entities/fields to prevent breaking changes to front-end on new subgraph sync. Deprecated entries must be removed after new subgraph sync and test.

Author: comc

README.md

@@ -2,6 +2,25 @@
 
 Indexer of Set Protocol v2 events. Built on [The Graph](https://thegraph.com/).
 
+<!--
+TO-DO:
+- Tutorials
+    - Deploy a local subgraph
+    - Deploy a subgraph to Hosted Service
+    - Deploy a subgraph to Subgraph Studio
+- How-To Guides
+    - Update the subgraph schema
+    - Update the subgraph mappings
+    - Test the subgraph locally
+    - Set up a Postman query
+- Technical Reference
+    - Docker compose usage
+    - Task usage
+- Background Information
+    - Schema structure
+    - Query structure
+-->
+
 ## SETUP
 
 ### Requirements:
@@ -10,19 +29,21 @@ Indexer of Set Protocol v2 events. Built on [The Graph](https://thegraph.com/).
 
 ### Local Deployment (Hardhat)
 
-1. Build the subgraph Docker image
+1. Build the Set Protocol Docker base and hardhat images
 
     `task docker-build`
 
-1. Deploy a Hardhat node (use `task deploy-hardhat -- detach` to start container in detached mode)
+1. Deploy a Hardhat node and custom script to the network
+
+    `task deploy-hardhat -- /full/path/to/test/script.ts`
 
-    `task deploy-hardhat`
+    e.g., deploy the subgraph test state setup script with
 
-1. Monitor the Hardhat node until fully deployed and tests are executed, e.g., if running detached use
+    `task deploy-hardhat -- $(pwd)/test/deploy-state.ts`
 
-    `docker logs docker-hardhat-1 --follow`
+1. Monitor the Hardhat node until fully deployed and tests are executed
 
-1. Compile the Set Protocol ABIs
+1. In a new terminal, compile the Set Protocol ABIs
 
     `task gen-abi`
 
@@ -32,6 +53,8 @@ Indexer of Set Protocol v2 events. Built on [The Graph](https://thegraph.com/).
 
 1. Once deployed, query the subgraph in the browser at (by default) http://127.0.0.1:8000/subgraphs/name/SetProtocol/set-protocol-v2
 
+    Example query to run can be found in `test/sample-query.txt`
+
 ### [TO-DO] External Deployment (Graph Hosted Service / Subgraph Studio)
 
 TBD
@@ -40,19 +63,19 @@ TBD
 
 Available tasks for this project:
 
-| COMMAND [OPTS]               | DESCRIPTION |
-|------------------------------|---------------------------------------------------------------------------------|
-| `clean [-- all]`             | Clean up local subgraph deployment; `all` arg additionally removes all volumes and the Hardhat node. |
-| `deploy-hardhat [-- detach]` | Deploy a Hardhat node and subgraph tests; `detach` runs container detached. |
-| `deploy-hosted [-- detach]`  | Build and deploy subgraph on Hosted Service; `detach` runs container detached. |
-| `deploy-local [-- detach]`   | Build and deploy subgraph on local network; `detach` runs container detached. |
-| `destroy-hardhat`            | Tear down the deployed Hardhat node. |
-| `docker-build`               | Build subgraph Docker image on defined node version base (default: 16-slim). |
-| `gen-abi`                    | Pull latest Set Protocol ABIs into the build environment. |
+| COMMAND [OPTS]                       | DESCRIPTION |
+|--------------------------------------|---------------------------------------------------------------------------------|
+| `clean [-- all\|subgraph\|hardhat]`  | Clean up local subgraph deployment; `all` arg additionally removes all volumes and the Hardhat node. |
+| `deploy-hardhat -- /path/to/file.ts` | Deploy a local Hardhat node and run a test script. Must specify full path to file as task input argument. |
+| `deploy-hosted [-- detach]`          | Build and deploy subgraph on Hosted Service; `detach` runs container detached. |
+| `deploy-local [-- detach]`           | Build and deploy subgraph on local network; `detach` runs container detached. |
+| `docker-build`                       | Build subgraph Docker image on defined node version base (default: 16-slim). |
+| `gen-abi`                            | Pull latest Set Protocol ABIs into the build environment. |
+| `gen-schema`                         | Compile the subgraph schema but do not deploy the subgraph. |
 
 ## [TO-DO] ADVANCED DEPLOYMENT GUIDES
 
-TBD: Things to be covered in this section
+TBD: Ideas to be covered in this section
 
 - custom override of args (requires custom untracked .env configs or CLI arg overrides)
 - the [Set Protocol V2 repo](https://github.com/SetProtocol/set-protocol-v2.git) currently requires node <= 16; therefore, Node 16 is the default target base image used in the Subgraph Docker image.
@@ -61,9 +84,12 @@ TBD: Things to be covered in this section
 
 ## [TO-DO] SUBGRAPH DEVELOPMENT
 
-### Tutorial
+### [TEMP] Dev Notes
 
-TBD
+- Each named dataSource or template entry should be in its own mappings/<entity>.ts file
+- Entity names cannot end with "s" due to conflict with query API (not currently documented)
+- Use `setToken` for schema fields, not `set` as will conflict will built-in callers
+- Templates must be initialized appropriately (see `ModuleInitialize` event handler for example)
 
 ### Reference Guide
 
@@ -81,7 +107,7 @@ To Be Completed
 
 `src/mappings/` - Event handlers
 
-`src/entities/` - Entity helper functions
+`src/utils/` - Entity helper functions and other utilities
 
 #### Historical Entities
 
@@ -114,12 +140,12 @@ Prefixes:
 
 #### Event/Call/Block Handlers
 
-Process events, function calls, and block data to update the subgraph. Must register handlers in `templates/subgraph.yaml`.
+Process events, smart contract function calls, and block data to update the subgraph. Must register handlers in `templates/subgraph.yaml`.
 
 #### Template spawners
 
 Not all contract addresses are known at the time of subgraph deployment. To track contracts as they are deployed, use contract templates.
-Tell the subgraph to watch a newly created contract by calling create() on imports from `generated/templates`. Ex the SetToken factory contract (SetTokenCreator) emits an event when a new SetToken is created, so we register that address as a new SetToken contract to watch. Templates are defined in `templates/subgraph.yaml`.
+Tell the subgraph to watch a newly created contract by calling `create()` on imports from `generated/templates`. For example, the SetToken factory contract (SetTokenCreator) emits an event when a new SetToken is created, so we register that address as a new SetToken contract to watch. Templates are defined in `templates/subgraph.yaml`.
 
 ## References
 

Taskfile.yml

@@ -5,22 +5,35 @@ dotenv: ["./docker/.env"]
 tasks:
 
   clean:
-    desc: "Clean up local subgraph deployment; deletes all locally deployed subgraph data."
+    desc: "Clean up deployments. Options: 'subgraph' (default), 'hardhat', or 'all'."
     dir: docker/
     cmds:
-      - docker-compose -f docker-compose.local.yml down --volumes {{.ARGS}}
+      - docker-compose -f {{.TARGET}} down --volumes {{.ARGS}}
+      - rm -rf ../generated ../build
     vars:
+      TARGET:
+        sh: if [ "{{.CLI_ARGS}}" = "hardhat" ]; then
+              echo "docker-compose.hardhat.yml";
+            else
+              echo "docker-compose.local.yml";
+            fi
       ARGS:
-        sh: if [ "{{.CLI_ARGS}}" = "all" ]; then echo "--remove-orphans"; else echo ""; fi
+        sh: if [ "{{.CLI_ARGS}}" = "all" ]; then
+              echo "--remove-orphans";
+            else
+              echo "";
+            fi
 
   deploy-hardhat:
-    desc: "Deploy a Hardhat node and subgraph tests."
+    desc: "Deploy a local Hardhat node and test script. Must specify full path to file as task input argument."
     dir: docker/
     cmds:
-      - docker-compose -f docker-compose.hardhat.yml up {{.ARGS}}
+      - docker-compose -f docker-compose.hardhat.yml run --name {{.HARDHAT_IP}} --rm -w /app -v {{.CLI_ARGS}}:/tmp/{{.DEPLOY_SCRIPT}} hardhat-build bash /app/scripts/deploy-hardhat.sh
     vars:
-      ARGS:
-        sh: if [ "{{.CLI_ARGS}}" = "detach" ]; then echo "-d"; else echo ""; fi
+      DEPLOY_SCRIPT:
+        sh: basename "{{.CLI_ARGS}}"
+    env:
+      DEPLOY_SCRIPT: "{{.DEPLOY_SCRIPT}}"
 
   deploy-hosted:
     # TO-DO: untested
@@ -56,21 +69,22 @@ tasks:
       - sh: if [ -z "$(ls -A ../abis)" ]; then exit -1; fi
         msg: "ERROR: No ABIs found. Run 'task gen-abi' first."
 
-  destroy-hardhat:
-    desc: "Tear down the deployed Hardhat node."
-    dir: docker/
-    cmds:
-      - docker-compose -f docker-compose.hardhat.yml down --volumes
-
   docker-build:
-    desc: "Build subgraph Docker image on defined node version base (default: 16-slim)."
+    desc: "Build Docker images based on defined node version base (default: 16-slim)."
     cmds:
-      - docker build -t {{.SET_IMAGE_NAME}}:node-${NODE_VER} -f docker/Dockerfile --build-arg NODE_VER=$NODE_VER .
-    env:
-      NODE_VER: "{{.NODE_VER}}"
+      - docker build -t {{.SET_IMAGE_NAME}}:node-{{.NODE_VER}} -f docker/Dockerfile.base --build-arg NODE_VER={{.NODE_VER}} .
+      - docker build -t {{.HARDHAT_IMAGE_NAME}}:node-{{.NODE_VER}} -f docker/Dockerfile.hardhat --build-arg SET_IMAGE_NAME={{.SET_IMAGE_NAME}} --build-arg NODE_VER={{.NODE_VER}} .
 
   gen-abi:
     desc: "Pull latest Set Protocol ABIs into the build environment."
     cmds:
       # Note the following command requires node ver 16 or below (only 12 and 16 tested working)
       - docker run --rm -v $(pwd)/abis:/subgraph/abis -v $(pwd)/scripts:/subgraph/scripts -w /subgraph {{.SET_IMAGE_NAME}}:node-{{.NODE_VER}} bash scripts/update-abis.sh
+
+  gen-schema:
+    desc: "Generate schema intermediate artifacts for development without a full deploy."
+    cmds:
+      - rm -rf generated/ build/
+      - docker run --rm -e NETWORK_NAME={{.NETWORK_LOCAL}} -v $(pwd)/:/subgraph -v setprotocol-subgraph-node_modules:/subgraph/node_modules -w /subgraph {{.SET_IMAGE_NAME}}:node-{{.NODE_VER}} sh scripts/gen-schema.sh
+    vars:
+      NETWORK_NAME: "{{.NETWORK_LOCAL}}"

docker/.env

@@ -1,8 +1,9 @@
 # Node version to use for deployment
 NODE_VER=16
 
-# Docker image name to target for builds and container deployments
-SET_IMAGE_NAME=setprotocol/subgraph
+# Docker image names to target for builds and container deployments
+HARDHAT_IMAGE_NAME=setprotocol/hardhat
+SET_IMAGE_NAME=setprotocol/base
 
 # Subgraph version label to deploy
 # See: https://thegraph.com/docs/en/studio/deploy-subgraph-studio#deploying-a-subgraph-to-subgraph-studio
@@ -13,7 +14,8 @@ GRAPH_NAME=set-protocol-v2
 
 # HARDHAT NODE (LOCAL)
 # --------------------
-HARDHAT_IP=http://docker-hardhat-1
+HARDHAT_IP=docker-hardhat-1
+# HARDHAT_IP=docker_hardhat_run_45d1437ad006
 HARDHAT_PORT=8545
 
 # LOCAL DEPLOYMENT PARAMS

docker/Dockerfile renamed to docker/Dockerfile.base

@@ -1,9 +1,9 @@
-# Build SetProtocol Subgraph Image
+# Build SetProtocol Base Image
 #
-# Builds a minimal image with dependencies required for The Graph deployments.
+# Builds a minimal image with dependencies required for various deployments.
 # Expectation is to build in the repo root directory by running, e.g.,
 #
-#   docker build -t setprotocol/subgraph:${NODE_VER}-slim -f docker/Dockerfile .
+#   docker build -t setprotocol/base:${NODE_VER}-slim -f docker/Dockerfile.base .
 #
 # This is handled by the task `docker-build` in the Taskfile.
 

docker/Dockerfile.hardhat

@@ -0,0 +1,21 @@
+# Build SetProtocol Hardhat Image
+#
+# Builds a minimal image with dependencies required for a Hardhat node deployment.
+# Expectation is to build in the repo root directory by running, e.g.,
+#
+#   docker build -t setprotocol/hardhat:${NODE_VER}-slim -f docker/Dockerfile.hardhat .
+#
+# This is handled by the task `docker-build` in the Taskfile.
+
+ARG NODE_VER=16
+ARG SET_IMAGE_NAME=setprotoco/base
+
+FROM ${SET_IMAGE_NAME}:node-${NODE_VER} as base
+
+COPY ./docker/scripts/deploy-set-protocol-v2.sh /tmp
+
+WORKDIR /app
+
+RUN /bin/bash /tmp/deploy-set-protocol-v2.sh \
+    && rm -rf /tmp/*
+

docker/docker-compose.hardhat.yml

@@ -2,15 +2,15 @@ version: '3'
 
 services:
 
-  hardhat:
-    image: ${SET_IMAGE_NAME}:node-${NODE_VER}
-    command: ["sh", "/scripts/deploy-hardhat.sh"]
+  hardhat-build:
+    image: ${HARDHAT_IMAGE_NAME}:node-${NODE_VER}
     environment:
-      HARDHAT_PORT: ${HARDHAT_PORT}
-    working_dir: /hardhat
+      HARDHAT_PORT: ${HARDHAT_PORT:-"8545"}
+      DEPLOY_SCRIPT: ${DEPLOY_SCRIPT:-""}
+    working_dir: /app
     volumes:
-      - ../scripts:/scripts
-      - hardhat:/hardhat
+      - ../scripts:/app/scripts
+      - hardhat:/app
 
 volumes:
   hardhat:

docker/docker-compose.hosted.yml

@@ -5,14 +5,14 @@ services:
   setprotocol-subgraph:
     image: ${SET_IMAGE_NAME}:node-${NODE_VER}
     environment:
-      DEPLOYMENT: ${DEPLOYMENT}
-      IPFS_IP: ${IPFS_HOSTED}
-      GRAPH_NODE_IP: ${GRAPH_NODE_HOSTED}
-      ACCESS_TOKEN: ${ACCESS_TOKEN}
-      SUBGRAPH_VERSION: ${SUBGRAPH_VERSION}
-      NETWORK_NAME: ${NETWORK_NAME}
-      GITHUB_REPO: ${GITHUB_REPO}
-      GRAPH_NAME: ${GRAPH_NAME}
+      DEPLOYMENT: ${DEPLOYMENT:-""}
+      IPFS_IP: ${IPFS_HOSTED:-""}
+      GRAPH_NODE_IP: ${GRAPH_NODE_HOSTED:-""}
+      ACCESS_TOKEN: ${ACCESS_TOKEN:-""}
+      SUBGRAPH_VERSION: ${SUBGRAPH_VERSION:-""}
+      NETWORK_NAME: ${NETWORK_NAME:-""}
+      GITHUB_REPO: ${GITHUB_REPO:-""}
+      GRAPH_NAME: ${GRAPH_NAME:-""}
     volumes:
       - ../:/subgraph
       - setprotocol-subgraph-node_modules:/subgraph/node_modules

docker/docker-compose.local.yml

@@ -16,7 +16,7 @@ services:
       postgres_db: graph-node
       ipfs: 'ipfs:${IPFS_PORT}'
       GRAPH_LOG: info
-      ethereum: hardhat:${HARDHAT_IP}:${HARDHAT_PORT}
+      ethereum: hardhat:http://${HARDHAT_IP}:${HARDHAT_PORT}
 
   ipfs:
     image: ipfs/go-ipfs:v0.4.23
@@ -38,14 +38,14 @@ services:
     depends_on:
       - graph-node
     environment:
-      DEPLOYMENT: ${DEPLOYMENT}
-      IPFS_IP: ${IPFS_IP}
-      GRAPH_NODE_IP: ${GRAPH_NODE_IP}
-      ACCESS_TOKEN: ${ACCESS_TOKEN}
-      SUBGRAPH_VERSION: ${SUBGRAPH_VERSION}
-      NETWORK_NAME: ${NETWORK_NAME}
-      GITHUB_REPO: ${GITHUB_REPO}
-      GRAPH_NAME: ${GRAPH_NAME}
+      DEPLOYMENT: ${DEPLOYMENT:-""}
+      IPFS_IP: ${IPFS_IP:-""}
+      GRAPH_NODE_IP: ${GRAPH_NODE_IP:-""}
+      ACCESS_TOKEN: ${ACCESS_TOKEN:-""}
+      SUBGRAPH_VERSION: ${SUBGRAPH_VERSION:-""}
+      NETWORK_NAME: ${NETWORK_NAME:-""}
+      GITHUB_REPO: ${GITHUB_REPO:-""}
+      GRAPH_NAME: ${GRAPH_NAME:-""}
     volumes:
       - ../:/subgraph
       - setprotocol-subgraph-node_modules:/subgraph/node_modules

docker/scripts/deploy-set-protocol-v2.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+set -e
+
+if [ ! -d "set-protocol-v2" ]; then
+    git clone -q https://github.com/SetProtocol/set-protocol-v2.git
+fi
+
+cd set-protocol-v2
+
+# Set up default env vars as required
+if [ ! -f ".env" ]; then
+    cp .env.default .env
+fi
+
+# Install the dependencies
+yarn install

package.json

@@ -3,12 +3,5 @@
   "version": "0.1.0",
   "repository": "https://github.com/SetProtocol/set-protocol-v2-subgraph",
   "description": "Set Protocol subgraph deployment repo",
-  "license": "MIT",
-  "dependencies": {
-    "@graphprotocol/graph-cli": "^0.26.0",
-    "@graphprotocol/graph-ts": "^0.24.1",
-    "handlebars": "^4.7.7",
-    "ts-node": "^10.5.0",
-    "typescript": "^4.5.5"
-  }
+  "license": "MIT"
 }