Lev 116 add support for external deployments (#3)

* Initial refactor for supporting external deployment to Hosted Service.

* Minor refactoring of SetTokenTemplate creation

* Minor code format fixes

* Update gitignore context

* [lev-192-dynamically-populate-deprecated] Dynamically populate the subgraph.yaml targets based on dotenv subgraph network to handle deprecated Ethereum mainnet+staging SetTokenCreator contracts

* [lev-191-auto-populate-subgraph-name-in] Dynamically populate subgraph name based on target network and clean up deploy scripts

* Fixed hosted deployment commands and improved error handling

* [sim-164-add-delegatemanager-abi-to-subgraph] Add DelegatedManager contract from set-v2-strategies repo to subgraph ABI generator

* Clean up deployments.json unneeded contract addresses and add deprecated SetTokenCreator contract for Polygon staging

* Add documentation about Hosted Subgraph inactivity policy

* [lev-204-diagnose-and-fix-polygon-deployments]:
    - Add documentation about Hosted Subgraph inactivity policy
    - Clean up deployments.json unneeded contract addresses and add deprecated SetTokenCreator contract for Matic staging

* Resolve issue with SetTokenCreator start block in Matic deployment

* [Sim 163 create local hardhat node testable state (#6)]
    - add DelegatedManager abi
    - add template for delegated manager deployment script
    - Refactored Hardhat node state deployment to target set-v2-strategies repo
    - Fixed deploy-state script import
    - fix imports in existing scripts
    - add events for delegated manager role changes
    - fix header comment
    - add additional manager test cases

Co-authored-by: Colin McRae <[email protected]>
Co-authored-by: Pranav Bhardwaj <[email protected]>

Author: 3 people

.gitignore

@@ -11,8 +11,11 @@ pids
 *.seed
 *.pid.lock
 
-# Dependency directories
+# Dependency files and directories
 node_modules/
+# These should be explicitly committed if we want to lock versions
+package.json
+package-lock.json
 
 # Yarn Integrity file
 .yarn-integrity
@@ -38,4 +41,10 @@ cache/
 
 # Generated subgraph manifest and addresses
 subgraph.yaml
-addresses.ts
+addresses.ts
+
+# User dotenv
+docker/custom.env
+
+# Non-explicitly-committed tests
+test/

README.md

@@ -3,7 +3,7 @@
 Indexer of Set Protocol v2 events. Built on [The Graph](https://thegraph.com/).
 
 <!--
-TO-DO:
+[TO-DO] CONTENTS (herein or external):
 - Tutorials
     - Deploy a local subgraph
     - Deploy a subgraph to Hosted Service
@@ -55,23 +55,41 @@ TO-DO:
 
     Example query to run can be found in `test/sample-query.txt`
 
-### [TO-DO] External Deployment (Graph Hosted Service / Subgraph Studio)
+### External Deployment to Hosted Service
+
+1. Build the Set Protocol Docker base and hardhat images
+
+    `task docker-build`
+
+1. Deploy hosted subgraph to network specified by the `NETWORK_HOSTED` argument
+
+    `task deploy-hosted [-- NETWORK_HOSTED [SUBGRAPH_ACCESS_TOKEN]]`
+
+    Note: `NETWORK_HOSTED` and `SUBGRAPH_ACCESS_TOKEN` must be provided as input arguments or defined in the dotenv configuration. Input arguments take precendence over dotenv configurations. For input arguments, you can specify just the network, or both the network and the access token, but you cannot provide the access token alone.
+
+**NOTE**
+
+Per the [documentation](https://thegraph.com/docs/en/hosted-service/deploy-subgraph-hosted/#subgraph-archive-policy) on the Hosted Service:
+
+> A subgraph is defined as "inactive" if it was deployed to the Hosted Service more than 45 days ago, and if it has received 0 queries in the last 30 days.
+
+### [TO-DO] External Deployment to Subgraph Studio
 
 TBD
 
 ## USAGE
 
 Available tasks for this project:
 
-| COMMAND [OPTS]                       | DESCRIPTION |
-|--------------------------------------|---------------------------------------------------------------------------------|
-| `clean [-- all\|subgraph\|hardhat]`  | Clean up local subgraph deployment; `all` arg additionally removes all volumes and the Hardhat node. |
-| `deploy-hardhat -- /path/to/file.ts` | Deploy a local Hardhat node and run a test script. Must specify full path to file as task input argument. |
-| `deploy-hosted [-- detach]`          | Build and deploy subgraph on Hosted Service; `detach` runs container detached. |
-| `deploy-local [-- detach]`           | Build and deploy subgraph on local network; `detach` runs container detached. |
-| `docker-build`                       | Build subgraph Docker image on defined node version base (default: 16-slim). |
-| `gen-abi`                            | Pull latest Set Protocol ABIs into the build environment. |
-| `gen-schema`                         | Compile the subgraph schema but do not deploy the subgraph. |
+| COMMAND [OPTS]                        | DESCRIPTION |
+|---------------------------------------|---------------------------------------------------------------------------------|
+| `clean [-- all\|subgraph\|hardhat]`   | Clean up local subgraph deployment; `all` arg additionally removes all volumes and the Hardhat node. |
+| `deploy-hardhat -- /path/to/file.ts`  | Deploy a local Hardhat node and run a test script. Must specify full path to file as task input argument. |
+| `deploy-hosted [-- NETWORK_HOSTED [SUBGRAPH_ACCESS_TOKEN]]` | Build and deploy subgraph to `NETWORK_HOSTED` on Hosted Service. `SUBGRAPH_ACCESS_TOKEN` must be provided or defined in a private dotenv. |
+| `deploy-local [-- detach]`            | Build and deploy subgraph on local network; `detach` runs container detached. |
+| `docker-build`                        | Build subgraph Docker image on defined node version base (default: 16-slim). |
+| `gen-abi`                             | Pull latest Set Protocol ABIs into the build environment. |
+| `gen-schema [-- hosted]`              | Compile the subgraph schema but do not deploy the subgraph; default target subgraph network is hardhat. |
 
 ## [TO-DO] ADVANCED DEPLOYMENT GUIDES
 
@@ -91,6 +109,13 @@ TBD: Ideas to be covered in this section
 - Use `setToken` for schema fields, not `set` as will conflict will built-in callers
 - Templates must be initialized appropriately (see `ModuleInitialize` event handler for example)
 
+
+`ModuleInitialized` Event Notes
+- `event.address` - `ModuleInitialized` module contract address
+- `event.transaction.hash` - hash of the call transaction that triggered the event
+- `event.params._module` - the initialized module contract address
+
+
 ### Reference Guide
 
 To Be Completed

Taskfile.yml

@@ -1,6 +1,9 @@
 version: "3"
 
-dotenv: ["./docker/.env"]
+dotenv: [
+  "./docker/custom.env",
+  "./docker/.env",
+]
 
 tasks:
 
@@ -40,13 +43,16 @@ tasks:
     desc: "Build and deploy subgraph on Hosted Service."
     dir: docker/
     cmds:
-      - docker-compose -f docker-compose.hosted.yml up {{.ARGS}}
+      - docker-compose -f docker-compose.hosted.yml up
     vars:
-      ARGS:
-        sh: if [ "{{.CLI_ARGS}}" = "detach" ]; then echo "-d"; else echo ""; fi
+      NETWORK_NAME:
+        sh: if [ $(echo "{{.CLI_ARGS}}" | cut -d" " -f1) ]; then echo "{{.CLI_ARGS}}" | cut -d" " -f1; else echo "{{.NETWORK_HOSTED}}"; fi
+      SUBGRAPH_ACCESS_TOKEN:
+        sh: if [ $(echo "{{.CLI_ARGS}}" | cut -d" " -s -f2) ]; then echo "{{.CLI_ARGS}}" | cut -d" " -s -f2; else echo "{{.SUBGRAPH_ACCESS_TOKEN}}"; fi
     env:
+      ACCESS_TOKEN: "{{.SUBGRAPH_ACCESS_TOKEN}}"
       DEPLOYMENT: "hosted"
-      NETWORK_NAME: "{{.NETWORK_HOSTED}}"
+      NETWORK_NAME: "{{.NETWORK_NAME}}"
     preconditions:
       - sh: if [ -z "$(ls -A ../abis)" ]; then exit -1; fi
         msg: "ERROR: No ABIs found. Run 'task gen-abi' first."
@@ -61,7 +67,7 @@ tasks:
         sh: if [ "{{.CLI_ARGS}}" = "detach" ]; then echo "-d"; else echo ""; fi
     env:
       DEPLOYMENT: "local"
-      ACCESS_TOKEN: "{{.ACCESS_TOKEN}}"
+      ACCESS_TOKEN: "{{.LOCAL_ACCESS_TOKEN}}"
       IPFS_IP: "{{.IPFS_CONTAINER}}:{{.IPFS_PORT}}"
       GRAPH_NODE_IP: "{{.GRAPH_NODE_CONTAINER}}:{{.GRAPH_NODE_PORT}}"
       NETWORK_NAME: "{{.NETWORK_LOCAL}}"
@@ -85,6 +91,9 @@ tasks:
     desc: "Generate schema intermediate artifacts for development without a full deploy."
     cmds:
       - rm -rf generated/ build/
-      - docker run --rm -e NETWORK_NAME={{.NETWORK_LOCAL}} -v $(pwd)/:/subgraph -v setprotocol-subgraph-node_modules:/subgraph/node_modules -w /subgraph {{.SET_IMAGE_NAME}}:node-{{.NODE_VER}} sh scripts/gen-schema.sh
+      - docker run --rm -e NETWORK_NAME={{.NETWORK_NAME}} -v $(pwd)/:/subgraph -v setprotocol-subgraph-node_modules:/subgraph/node_modules -w /subgraph {{.SET_IMAGE_NAME}}:node-{{.NODE_VER}} sh scripts/gen-schema.sh
     vars:
-      NETWORK_NAME: "{{.NETWORK_LOCAL}}"
+      NETWORK_NAME:
+        sh: if [ "{{.CLI_ARGS}}" = "hosted" ]; then echo "{{.NETWORK_HOSTED}}"; else echo "{{.NETWORK_LOCAL}}"; fi
+    env:
+      NETWORK_NAME: "{{.NETWORK_NAME}}"

deployments.json

@@ -1,62 +1,56 @@
 {
   "mainnet": {
     "network": "mainnet",
+    "subgraphName": "set-protocol-v2",
     "startBlock": 10830495,
     "SetTokenCreator": "0xef72d3278dc3eba6dc2614965308d1435ffd748a"
   },
-  "hardhat": {
-    "network": "hardhat",
-    "startBlock": 0,
-    "Controller": "0x1dc4c1cefef38a777b15aa20260a54e584b16c48",
-    "SetTokenCreator": "0x8d61158a366019ac78db4149d75fff9dda51160d",
-    "IntegrationRegistry": "0xb69e673309512a9d726f87304c6984054f87a93b",
-    "PriceOracle": "0x131855dda0aaff096f6854854c55a4debf61077a",
-    "SetValuer": "0xe86bb98fcf9bff3512c74589b78fb168200cc546",
-    "BasicIssuanceModule": "0x1d7022f5b17d2f8b695918fb48fa1089c9f85401",
-    "StreamingFeeModule": "0xdc688d29394a3f1e6f1e5100862776691afaf3d2",
-    "NavIssuanceModule": "0xb7c9b454221e26880eb9c3101b3295ca7d8279ef",
-    "TradeModule": "0x59adefa01843c627ba5d6aa350292b4b7ccae67a"
-  },
-  "matic": {
-    "network": "matic",
-    "startBlock": 0,
-    "Controller": "0x75FBBDEAfE23a48c0736B2731b956b7a03aDcfB2",
-    "SetTokenCreator": "0x14f0321be5e581abF9d5BC76260bf015Dc04C53d",
-    "IntegrationRegistry": "0x4c4C649455c6433dC48ff1571C9e50aC58f0CeFA",
-    "PriceOracle": "0x9378Ad514c00E4869656eE27b634d852DD48feAD",
-    "SetValuer": "0x3700414Bb6716FcD8B14344fb10DDd91FdEA59eC",
-    "BasicIssuanceModule": "0x38E5462BBE6A72F79606c1A0007468aA4334A92b",
-    "StreamingFeeModule": "0x8440f6a2c42118bed0D6E6A89Bf170ffd13e21c0",
-    "TradeModule": "0xd04AabadEd11e92Fefcd92eEdbBC81b184CdAc82"
-  },
   "staging-mainnet": {
     "network": "mainnet",
+    "subgraphName": "set-protocol-v2-staging",
     "startBlock": 10819027,
     "SetTokenCreator": "0x026d25C2B70Ddbb8D759f1f38d6fD6e23b60B6DF"
   },
+  "matic": {
+    "network": "matic",
+    "subgraphName": "set-protocol-v2-matic",
+    "startBlock": 0,
+    "SetTokenCreator": "0x14f0321be5e581abF9d5BC76260bf015Dc04C53d"
+  },
   "matic-staging-mainnet": {
     "network": "matic",
+    "subgraphName": "set-protocol-v2-matic-staging",
     "startBlock": 16655167,
     "SetTokenCreator": "0x2907206b253665eef4776b8e684d61afd957b974"
   },
   "arbitrum-staging-mainnet": {
     "network": "arbitrum-one",
+    "subgraphName": "set-protocol-v2-arbitrum-staging",
     "startBlock": 1168154,
     "SetTokenCreator": "0xCF786472d37f557A80fE6daFF6f2672bfDa728a3"
   },
   "avalanche-staging-mainnet": {
     "network": "avalanche",
+    "subgraphName": "set-protocol-v2-avax-staging",
     "startBlock": 6097319,
     "SetTokenCreator": "0xE99447aBbD5A7730b26D2D16fCcB2086319e4bC3"
   },
+  "optimism-mainnet": {
+    "network": "optimism",
+    "subgraphName": "set-protocol-v2-optimism",
+    "startBlock": 900529,
+    "SetTokenCreator": "0x0bc84D31f11D90156c30B4f19509Ede969A0B840"
+  },
   "optimism-staging-mainnet": {
     "network": "optimism",
+    "subgraphName": "set-protocol-v2-optimism-staging",
     "startBlock": 593299,
     "SetTokenCreator": "0xCF786472d37f557A80fE6daFF6f2672bfDa728a3"
   },
-  "optimism-mainnet": {
-    "network": "optimism",
-    "startBlock": 900529,
-    "SetTokenCreator": "0x0bc84D31f11D90156c30B4f19509Ede969A0B840"
+  "hardhat": {
+    "network": "hardhat",
+    "subgraphName": "set-protocol-v2",
+    "startBlock": 0,
+    "SetTokenCreator": "0x8d61158a366019ac78db4149d75fff9dda51160d"
   }
 }

docker/.env

@@ -1,36 +1,69 @@
-# Node version to use for deployment
-NODE_VER=16
+#---------------------------------
+# SUBGRAPH CONFIGURATION SETTINGS
+#---------------------------------
+# This file defines the default and optional configuration parameters for
+# subgraph development and both local and external deployment targets.
+#
+# It is suggested to create a copy of this file with, e.g.,
+#   cp docker/.env docker/custom.env
+# and uncomment and configure custom parameters and settings in the copy rather
+# than directly editing this file. The custom dotenv file sohuld be prepended
+# into the Taskfile.yml dotenv array to take precedence over any settings in
+# this file.
 
-# Docker image names to target for builds and container deployments
-HARDHAT_IMAGE_NAME=setprotocol/hardhat
-SET_IMAGE_NAME=setprotocol/base
+# USER CONFIGURABLE SETTINGS
+#----------------------------
 
-# Subgraph version label to deploy
+# Subgraph Version Label for Deployment
 # See: https://thegraph.com/docs/en/studio/deploy-subgraph-studio#deploying-a-subgraph-to-subgraph-studio
-SUBGRAPH_VERSION=test-0.0.1
+# SUBGRAPH_VERSION="0.0.1"
+
+# Hosted Service Target Network (see deployments.json for options)
+#NETWORK_HOSTED="mainnet"
+
+# Hosted Service Subgraph Access Token
+# This private access key is required to deploy a subgraph to the Hosted
+# Service and is tied to the original Github account (GITHUB_REPO) used to
+# deploy the subgraph. It must not be committed or shared publicly.
+# See: https://thegraph.com/docs/en/hosted-service/deploy-subgraph-hosted/#store-the-access-token
+#SUBGRAPH_ACCESS_TOKEN=
+
+# Github Repository Name for Hosted Service Deployments
+# Defaults to `justinkchen` as the original deployment linked repo tied to
+# the Set company ACCESS_TOKEN. Uncomment to override only if necessary.
+#GITHUB_REPO="justinkchen"
+
+# SetProtocol Subgraph Name for Deployment
+# Defaults to the existing name based on the target graph network if empty.
+# Uncomment and specify a name to override only if necessary.
+#SUBGRAPH_NAME=
 
-GITHUB_REPO=SetProtocol
-GRAPH_NAME=set-protocol-v2
 
-# HARDHAT NODE (LOCAL)
-# --------------------
+# DEFAULT CONFIGURATION SETTINGS
+#--------------------------------
+# Do not change settings below unless you understand the consequences.
+
+# Node Version for Docker Images
+NODE_VER=16
+
+# SetProtocol Base Docker Image Name
+SET_IMAGE_NAME=setprotocol/base
+# SetProtocol Hardhat Node Docker Image Name
+HARDHAT_IMAGE_NAME=setprotocol/hardhat
+
+# Hardhat Node Configuration
 HARDHAT_IP=docker-hardhat-1
-# HARDHAT_IP=docker_hardhat_run_45d1437ad006
 HARDHAT_PORT=8545
 
-# LOCAL DEPLOYMENT PARAMS
-# -----------------------
-ACCESS_TOKEN=test
+# Local Deployment Configuration
+LOCAL_ACCESS_TOKEN=test # note: only used locally, ignored in hosted deployments
 IPFS_CONTAINER=docker-ipfs-1
 IPFS_PORT=5001
 GRAPH_NODE_CONTAINER=docker-graph-node-1
 GRAPH_NODE_PORT=8020
 NETWORK_LOCAL=hardhat
-GRAPHQL_PORT=8000 # port on host for TCP access to Subgraph GraphQL
+GRAPHQL_PORT=8000 # port on host for TCP access to Subgraph GraphQL query tool
 
-# HOSTED DEPLOYMENT PARAMS
-# ------------------------
+# Hosted Service Deployment Configuration
 IPFS_HOSTED=https://api.thegraph.com/ipfs/
 GRAPH_NODE_HOSTED=https://api.thegraph.com/deploy/
-# External network to deploy on (see deployments.json for options)
-NETWORK_HOSTED=mainnet

docker/Dockerfile.base

@@ -12,5 +12,5 @@ ARG NODE_VER=16
 FROM node:${NODE_VER}-slim as base
 
 RUN apt update \
-    && apt install -y python3 make gcc g++ openssl git libsecret-1-dev \
+    && apt install -y python3 make gcc g++ openssl git libsecret-1-dev jq \
     && rm -rf /var/lib/apt/lists/* /tmp/*

docker/docker-compose.hosted.yml

@@ -5,20 +5,19 @@ services:
   setprotocol-subgraph:
     image: ${SET_IMAGE_NAME}:node-${NODE_VER}
     environment:
-      DEPLOYMENT: ${DEPLOYMENT:-""}
+      DEPLOYMENT: ${DEPLOYMENT:-hosted}
       IPFS_IP: ${IPFS_HOSTED:-""}
       GRAPH_NODE_IP: ${GRAPH_NODE_HOSTED:-""}
-      ACCESS_TOKEN: ${ACCESS_TOKEN:-""}
+      ACCESS_TOKEN: ${ACCESS_TOKEN}
       SUBGRAPH_VERSION: ${SUBGRAPH_VERSION:-""}
-      NETWORK_NAME: ${NETWORK_NAME:-""}
-      GITHUB_REPO: ${GITHUB_REPO:-""}
-      GRAPH_NAME: ${GRAPH_NAME:-""}
+      NETWORK_NAME: ${NETWORK_NAME}
+      GITHUB_REPO: ${GITHUB_REPO:-justinkchen}
+      SUBGRAPH_NAME: ${SUBGRAPH_NAME}
     volumes:
       - ../:/subgraph
       - setprotocol-subgraph-node_modules:/subgraph/node_modules
     working_dir: /subgraph
     command: "sh ./scripts/deploy.sh"
-    # restart: on-failure:max-retries 5
 
 volumes:
   setprotocol-subgraph-node_modules:

docker/docker-compose.local.yml

@@ -31,21 +31,21 @@ services:
       POSTGRES_PASSWORD: let-me-in
       POSTGRES_DB: graph-node
     volumes:
-      - graph-node-postgres:/var/lib/postgresql/data  
+      - graph-node-postgres:/var/lib/postgresql/data
 
   setprotocol-subgraph:
     image: ${SET_IMAGE_NAME}:node-${NODE_VER}
     depends_on:
       - graph-node
     environment:
-      DEPLOYMENT: ${DEPLOYMENT:-""}
+      DEPLOYMENT: ${DEPLOYMENT:-local}
       IPFS_IP: ${IPFS_IP:-""}
       GRAPH_NODE_IP: ${GRAPH_NODE_IP:-""}
-      ACCESS_TOKEN: ${ACCESS_TOKEN:-""}
+      ACCESS_TOKEN: ${ACCESS_TOKEN:-test}
       SUBGRAPH_VERSION: ${SUBGRAPH_VERSION:-""}
-      NETWORK_NAME: ${NETWORK_NAME:-""}
-      GITHUB_REPO: ${GITHUB_REPO:-""}
-      GRAPH_NAME: ${GRAPH_NAME:-""}
+      NETWORK_NAME: ${NETWORK_NAME:-hardhat}
+      GITHUB_REPO: ${GITHUB_REPO:-SetProtocol}
+      SUBGRAPH_NAME: ${SUBGRAPH_NAME}
     volumes:
       - ../:/subgraph
       - setprotocol-subgraph-node_modules:/subgraph/node_modules