Sim 165 update subgraph schema and mappings (#10)

* Clean up Hardhat Docker build script

* add DelegatedManager migration to EOAmanager test case

* update imports

* update calls to initializeModuleAndExtension with module parameters

* Refactor contract ABI deployment in support set-protocol-v2 v0.4.0 and set-v2-strategies v0.0.7

* add WIP deployment script for DelegatedManagerSystem test environment

* add updated abi generation for DelegatedManagerSystem contracts

* * Restructure subgraph in support of DelegatedManager integration

* Refactor local deployments to improve developer experience

* Refactor Taskfile tasks to use docker-compose

* Update deployments.json with new contract addresses

* Replace setTokens String type with SetToken on Owner, Methodologist, and Operator for expanded querying capability

* Add entity to track template instantiation to eliminate warnings on duplicates

* Clean up deployments.json target names in line with set-v2-deployments repo; add Controller and ManagerCore addresses for available deployments

* Minor update to README and update sample query examples

Author: comc

.gitignore

@@ -41,10 +41,13 @@ cache/
 
 # Generated subgraph manifest and addresses
 subgraph.yaml
+!templates/subgraph.yaml
 addresses.ts
 
 # User dotenv
 docker/custom.env
 
 # Non-explicitly-committed tests
-test/
+test/
+
+abis/

README.md

@@ -2,25 +2,6 @@
 
 Indexer of Set Protocol v2 events. Built on [The Graph](https://thegraph.com/).
 
-<!--
-[TO-DO] CONTENTS (herein or external):
-- Tutorials
-    - Deploy a local subgraph
-    - Deploy a subgraph to Hosted Service
-    - Deploy a subgraph to Subgraph Studio
-- How-To Guides
-    - Update the subgraph schema
-    - Update the subgraph mappings
-    - Test the subgraph locally
-    - Set up a Postman query
-- Technical Reference
-    - Docker compose usage
-    - Task usage
-- Background Information
-    - Schema structure
-    - Query structure
--->
-
 ## SETUP
 
 ### Requirements:
@@ -67,7 +48,7 @@ Indexer of Set Protocol v2 events. Built on [The Graph](https://thegraph.com/).
 
     Note: `NETWORK_HOSTED` and `SUBGRAPH_ACCESS_TOKEN` must be provided as input arguments or defined in the dotenv configuration. Input arguments take precendence over dotenv configurations. For input arguments, you can specify just the network, or both the network and the access token, but you cannot provide the access token alone.
 
-**NOTE**
+**IMPORTANT NOTE**
 
 Per the [documentation](https://thegraph.com/docs/en/hosted-service/deploy-subgraph-hosted/#subgraph-archive-policy) on the Hosted Service:
 
@@ -81,111 +62,40 @@ TBD
 
 Available tasks for this project:
 
-| COMMAND [OPTS]                        | DESCRIPTION |
-|---------------------------------------|---------------------------------------------------------------------------------|
-| `clean [-- all\|subgraph\|hardhat]`   | Clean up local subgraph deployment; `all` arg additionally removes all volumes and the Hardhat node. |
-| `deploy-hardhat -- /path/to/file.ts`  | Deploy a local Hardhat node and run a test script. Must specify full path to file as task input argument. |
+| COMMAND [OPTS]                       | DESCRIPTION |
+|--------------------------------------|---------------------------------------------------------------------------------|
+| `clean [-- all\|subgraph\|hardhat]`  | Clean up local subgraph deployment; `all` arg additionally removes all volumes and the Hardhat node. |
+| `deploy-hardhat -- /path/to/file.ts` | Deploy a local Hardhat node and run a test script. Must specify full path to file as task input argument. |
 | `deploy-hosted [-- NETWORK_HOSTED [SUBGRAPH_ACCESS_TOKEN]]` | Build and deploy subgraph to `NETWORK_HOSTED` on Hosted Service. `SUBGRAPH_ACCESS_TOKEN` must be provided or defined in a private dotenv. |
-| `deploy-local [-- detach]`            | Build and deploy subgraph on local network; `detach` runs container detached. |
-| `docker-build`                        | Build subgraph Docker image on defined node version base (default: 16-slim). |
-| `gen-abi`                             | Pull latest Set Protocol ABIs into the build environment. |
-| `gen-schema [-- hosted]`              | Compile the subgraph schema but do not deploy the subgraph; default target subgraph network is hardhat. |
-
-## [TO-DO] ADVANCED DEPLOYMENT GUIDES
-
-TBD: Ideas to be covered in this section
-
-- custom override of args (requires custom untracked .env configs or CLI arg overrides)
-- the [Set Protocol V2 repo](https://github.com/SetProtocol/set-protocol-v2.git) currently requires node <= 16; therefore, Node 16 is the default target base image used in the Subgraph Docker image.
-- node dependencies installation into a named Docker volume, build the subgraph mappings, create the subgraph, then deploy it to the target graph node and IPFS database. Not that consecutive runs of this command will use the existing named volume for the `node_modules` unless it is manually removed.
-- the subgraph is deployed to the local graph-node and IPFS containers defined in the `subgraph.env` file. The subgraph endpoints given by the deployment are relative to the Docker container and not accessible externally as given. To access the subgraph, instead navigate to: http://127.0.0.1:8000/subgraphs/name/SetProtocol/setprotocolv2.
-
-## [TO-DO] SUBGRAPH DEVELOPMENT
-
-### [TEMP] Dev Notes
-
-- Each named dataSource or template entry should be in its own mappings/<entity>.ts file
-- Entity names cannot end with "s" due to conflict with query API (not currently documented)
-- Use `setToken` for schema fields, not `set` as will conflict will built-in callers
-- Templates must be initialized appropriately (see `ModuleInitialize` event handler for example)
-
+| `deploy-local [-- detach]`           | Build and deploy graph-node and ipfs services on local network followed by deployment of the subgraph; `detach` runs containers in detached mode. |
+| `deploy-local-graphnode [-- detach]` | Build and deploy graph-node and ipfs services on local network; `detach` runs containers in detached mode. |
+| `deploy-local-subgraph [-- refresh]` | Compile and deploy the subgraph onto running graph-node and ipfs services on local network. `refresh` first deletes all IPFS data and restarts the service. |
+| `docker-build`                       | Build subgraph Docker image on defined node version base (default: 16-slim). |
+| `gen-abi`                            | Pull latest Set Protocol ABIs into the build environment. |
+| `gen-schema [-- hosted]`             | Compile the subgraph schema but do not deploy the subgraph; default target subgraph network is local unless `hosted` argument is provided. |
 
-`ModuleInitialized` Event Notes
-- `event.address` - `ModuleInitialized` module contract address
-- `event.transaction.hash` - hash of the call transaction that triggered the event
-- `event.params._module` - the initialized module contract address
-
-
-### Reference Guide
-
-To Be Completed
-
-#### Key Files
+## KEY FILES
 
 `schema.graphql` - Subgraph schema
 
 `templates/subgraph.yaml` - configure watched contracts and events
 
-`deployments.json` - configure deployed contract addresses
+`deployments.json` - configure deployed contract addresses on each chain
 
 `src/` - [AssemblyScript](https://www.assemblyscript.org) code for subgraph handlers
 
 `src/mappings/` - Event handlers
 
-`src/utils/` - Entity helper functions and other utilities
-
-#### Historical Entities
-
-##### Events
-
-Individual transaction log events referenced by TxID and log index. Can have multiple events of the same type with the same timestamp (IE same block). Ex TradeEvent.
-
-##### States
+`src/utils/` - Entity mappings functions and other helper utilities
 
-Final state of an entity at the end of a block. Referenced by block number. Multiple events in the same block will be consolidated to one state update. Ex TotalSupplyState.
+## REFERENCES
 
-Events are better for tracking important actions while States are better for timeseries data in which multiple data points per timestamp would be inconveinent.
+[The Graph Docs](https://thegraph.com/docs/)
 
-##### Current Entities
-
-Tracks the most recent state of a contract. Usually references the latest event or state update entity.
-
-##### Entity Helpers
-
-Most of the logic regarding manipulation of entities should be in helper functions. Create helper functions based on the following nomenclature as needed.
-
-Prefixes:
-
-- create: create new entity
-- get: lookup entity by ID and return entity or throw error
-- update: update entity with new properties
-- delete: remove entity from subgraph store
-- ensure: create or update existing entity. Useful for states
-- track: execute contract call and store result in new entity. Useful for data that is not provided via event logs.
-
-#### Event/Call/Block Handlers
-
-Process events, smart contract function calls, and block data to update the subgraph. Must register handlers in `templates/subgraph.yaml`.
-
-#### Template spawners
-
-Not all contract addresses are known at the time of subgraph deployment. To track contracts as they are deployed, use contract templates.
-Tell the subgraph to watch a newly created contract by calling `create()` on imports from `generated/templates`. For example, the SetToken factory contract (SetTokenCreator) emits an event when a new SetToken is created, so we register that address as a new SetToken contract to watch. Templates are defined in `templates/subgraph.yaml`.
-
-## References
-
-[Discord: Index Co-op #set-subgraph](https://discord.gg/8FYPP7ebbw)
-
-[Subgraph Outline Sheet](https://docs.google.com/spreadsheets/d/1I3sk1kvfCPnnrUUCiBa35DZneeTx0vtGk04B-rKCJVE/edit?usp=sharing)
-
-[Subgraph notes doc](https://docs.google.com/document/d/1inFbQiskHoEKaNYdaHx69-quy8Y2xIva6N3673qw2jA/edit)
-
-[TheGraph Docs](https://thegraph.com/docs/)
+[Set Protocol Subgraph Docs](https://app.gitbook.com/o/-MGdl9Y5UCSpZPXC3ad7/s/-MGdlDDRsIRuOJOl7btN/function-tools-and-guides/engineering/development-guides/the-graph)
 
 [Set Protocol V2 Docs](https://docs.tokensets.com/)
 
 [Set Protocol V2 Contracts](https://github.com/SetProtocol/set-protocol-v2)
 
-[Set Protocol System Diagram](https://drive.google.com/file/d/15ETEqxkjkR29GmWH4gg4ob_OW9lb_Nly/view)
-
-[Hosted API](https://thegraph.com/explorer/subgraph/desert-defi/setprotocolv2)
+[Set Protocol V2 Strategies Contracts](https://github.com/SetProtocol/set-v2-strategies)

Taskfile.yml

@@ -8,7 +8,7 @@ dotenv: [
 tasks:
 
   clean:
-    desc: "Clean up deployments. Options: 'subgraph' (default), 'hardhat', or 'all'."
+    desc: "Clean up deployments. Options: 'subgraph' (default), 'graphnode', 'hardhat', or 'all'."
     dir: docker/
     cmds:
       - docker-compose -f {{.TARGET}} down --volumes {{.ARGS}}
@@ -17,6 +17,8 @@ tasks:
       TARGET:
         sh: if [ "{{.CLI_ARGS}}" = "hardhat" ]; then
               echo "docker-compose.hardhat.yml";
+            elif [ "{{.CLI_ARGS}}" = "graphnode" ]; then
+              echo "docker-compose.graphnode.yml";
             else
               echo "docker-compose.local.yml";
             fi
@@ -37,33 +39,63 @@ tasks:
         sh: basename "{{.CLI_ARGS}}"
     env:
       DEPLOY_SCRIPT: "{{.DEPLOY_SCRIPT}}"
+    preconditions:
+      - sh: if [ -z "{{.CLI_ARGS}}" ]; then
+              echo "No test deployment script provided.";
+              exit 1;
+            fi
 
   deploy-hosted:
-    # TO-DO: untested
     desc: "Build and deploy subgraph on Hosted Service."
     dir: docker/
     cmds:
       - docker-compose -f docker-compose.hosted.yml up
     vars:
       NETWORK_NAME:
-        sh: if [ $(echo "{{.CLI_ARGS}}" | cut -d" " -f1) ]; then echo "{{.CLI_ARGS}}" | cut -d" " -f1; else echo "{{.NETWORK_HOSTED}}"; fi
+        sh: if [ $(echo "{{.CLI_ARGS}}" | cut -d" " -f1) ]; then
+              echo "{{.CLI_ARGS}}" | cut -d" " -f1;
+            else
+              echo "{{.NETWORK_HOSTED}}";
+            fi
       SUBGRAPH_ACCESS_TOKEN:
-        sh: if [ $(echo "{{.CLI_ARGS}}" | cut -d" " -s -f2) ]; then echo "{{.CLI_ARGS}}" | cut -d" " -s -f2; else echo "{{.SUBGRAPH_ACCESS_TOKEN}}"; fi
+        sh: if [ $(echo "{{.CLI_ARGS}}" | cut -d" " -s -f2) ]; then
+              echo "{{.CLI_ARGS}}" | cut -d" " -s -f2;
+            else
+              echo "{{.SUBGRAPH_ACCESS_TOKEN}}";
+            fi
     env:
       ACCESS_TOKEN: "{{.SUBGRAPH_ACCESS_TOKEN}}"
       DEPLOYMENT: "hosted"
       NETWORK_NAME: "{{.NETWORK_NAME}}"
     preconditions:
-      - sh: if [ -z "$(ls -A ../abis)" ]; then exit -1; fi
+      - sh: if [ -z "$(ls -A ../abis)" ]; then exit 1; fi
         msg: "ERROR: No ABIs found. Run 'task gen-abi' first."
 
   deploy-local:
-    desc: "Build and deploy subgraph on local network."
+    desc: "Build graph-node and ipfs services then deploy subgraph on local network."
     dir: docker/
     cmds:
-      - docker-compose -f docker-compose.local.yml up {{.ARGS}}
+      - docker-compose -f docker-compose.graphnode.yml -f docker-compose.local.yml up {{.DETACH}}
     vars:
-      ARGS:
+      DETACH:
+        sh: if [ "{{.CLI_ARGS}}" = "detach" ]; then echo "-d"; else echo ""; fi
+    env:
+      DEPLOYMENT: "local"
+      ACCESS_TOKEN: "{{.LOCAL_ACCESS_TOKEN}}"
+      IPFS_IP: "{{.IPFS_CONTAINER}}:{{.IPFS_PORT}}"
+      GRAPH_NODE_IP: "{{.GRAPH_NODE_CONTAINER}}:{{.GRAPH_NODE_PORT}}"
+      NETWORK_NAME: "{{.NETWORK_LOCAL}}"
+    preconditions:
+      - sh: if [ -z "$(ls -A ../abis)" ]; then exit 1; fi
+        msg: "ERROR: No ABIs found. Run 'task gen-abi' first."
+
+  deploy-local-graphnode:
+    desc: "Build and deploy graph-node and ipfs services on local network."
+    dir: docker/
+    cmds:
+      - docker-compose -f docker-compose.graphnode.yml up {{.DETACH}}
+    vars:
+      DETACH:
         sh: if [ "{{.CLI_ARGS}}" = "detach" ]; then echo "-d"; else echo ""; fi
     env:
       DEPLOYMENT: "local"
@@ -72,7 +104,34 @@ tasks:
       GRAPH_NODE_IP: "{{.GRAPH_NODE_CONTAINER}}:{{.GRAPH_NODE_PORT}}"
       NETWORK_NAME: "{{.NETWORK_LOCAL}}"
     preconditions:
-      - sh: if [ -z "$(ls -A ../abis)" ]; then exit -1; fi
+      - sh: if [ -z "$(ls -A ../abis)" ]; then exit 1; fi
+        msg: "ERROR: No ABIs found. Run 'task gen-abi' first."
+
+  deploy-local-subgraph:
+    desc: "Deploy subgraph on running local graph-node and ipfs network."
+    dir: docker/
+    cmds:
+      - if [ "{{.REFRESH}}" = "true" ]; then echo "Deleting IPFS database and re-building..."; else echo "Skipping IPFS refresh..."; fi
+      - if [ "{{.REFRESH}}" = "true" ]; then docker-compose -f docker-compose.graphnode.yml exec ipfs rm -rf /data/ipfs/*; fi
+      - if [ "{{.REFRESH}}" = "true" ]; then docker-compose -f docker-compose.graphnode.yml restart ipfs; fi
+      - if [ "{{.REFRESH}}" = "true" ]; then docker-compose -f docker-compose.graphnode.yml restart graph-node; fi
+      - docker-compose -f docker-compose.local.yml run setprotocol-subgraph
+      - docker-compose -f docker-compose.local.yml rm -f setprotocol-subgraph
+    vars:
+      REFRESH:
+        sh: if [ "{{.CLI_ARGS}}" = "refresh" ]; then echo "true"; else echo "false"; fi
+    env:
+      DEPLOYMENT: "local"
+      ACCESS_TOKEN: "{{.LOCAL_ACCESS_TOKEN}}"
+      IPFS_IP: "{{.IPFS_CONTAINER}}:{{.IPFS_PORT}}"
+      GRAPH_NODE_IP: "{{.GRAPH_NODE_CONTAINER}}:{{.GRAPH_NODE_PORT}}"
+      NETWORK_NAME: "{{.NETWORK_LOCAL}}"
+    preconditions:
+      - sh: if [ $( docker container inspect -f '{{.State.Status}}' {{.GRAPH_NODE_CONTAINER}} ) != "running" ]; then exit 1; fi
+        msg: "ERROR: Docker container '{{.GRAPH_NODE_CONTAINER}}' is not running."
+      - sh: if [ $( docker container inspect -f '{{.State.Status}}' {{.IPFS_CONTAINER}} ) != "running" ]; then exit 1; fi
+        msg: "ERROR: Docker container '{{.IPFS_CONTAINER}}' is not running."
+      - sh: if [ -z "$(ls -A ../abis)" ]; then exit 1; fi
         msg: "ERROR: No ABIs found. Run 'task gen-abi' first."
 
   docker-build:
@@ -96,9 +155,13 @@ tasks:
     desc: "Generate schema intermediate artifacts for development without a full deploy."
     cmds:
       - rm -rf generated/ build/
-      - docker run --rm -e NETWORK_NAME={{.NETWORK_NAME}} -v $(pwd)/:/subgraph -v setprotocol-subgraph-node_modules:/subgraph/node_modules -w /subgraph {{.SET_IMAGE_NAME}}:node-{{.NODE_VER}} sh scripts/gen-schema.sh
+      - docker-compose -f docker/docker-compose.schema.yml run gen-schema
+      - docker-compose -f docker/docker-compose.schema.yml rm -f gen-schema
     vars:
       NETWORK_NAME:
         sh: if [ "{{.CLI_ARGS}}" = "hosted" ]; then echo "{{.NETWORK_HOSTED}}"; else echo "{{.NETWORK_LOCAL}}"; fi
     env:
       NETWORK_NAME: "{{.NETWORK_NAME}}"
+    preconditions:
+      - sh: if [ -z "$(ls -A ./abis)" ]; then exit 1; fi
+        msg: "ERROR: No ABIs found. Run 'task gen-abi' first."