Generate Hasura metadata from Tortoise models (#2)

* Generate Hasura metadata from Tortoise models

* CLI command configure-graphql

* Generate select permissions in Hasura metadata

* Readme improvements

* README improvements

* Fix __version__

* Catch ModuleNotFoundError

Author: droserasprout

.gitignore

@@ -143,4 +143,6 @@ cython_debug/
 
 *sqlite3*
 
-.noseids
+.noseids
+
+secrets.env

README.md

@@ -78,7 +78,9 @@ This command will create a new package with the following structure (some lines
 dipdup_hic_et_nunc/
 ├── handlers
 │   ├── on_mint.py
+│   ├── on_rollback.py
 │   └── on_transfer.py
+├── hasura-metadata.json
 ├── models.py
 ├── schemas
 │   ├── KT1Hkg5qeNhfwpKW4fXvq7HGZB9z2EnmCCA9
@@ -98,6 +100,8 @@ dipdup_hic_et_nunc/
 
 `schemas` directory is JSON schemas describing parameters of corresponding contract entrypoints. `types` are Pydantic dataclasses of these schemas. These two directories are autogenerated, you don't need to modify them. `models` and `handlers` modules will be discussed later.
 
+You could invoke `init` command on existing project (must be in your `PYTHONPATH`. Do it each time you update contract addresses or models. Code you've wrote won't be overwritten.
+
 ### Define models
 
 Dipdup uses [Tortoise](https://tortoise-orm.readthedocs.io/en/latest/) under the hood, fast asynchronous ORM supporting all major database engines. Check out [examples](https://tortoise-orm.readthedocs.io/en/latest/examples.html) to learn how to use is.
@@ -145,9 +149,21 @@ async def on_mint(
         await token.save()
 ```
 
+Handler name `on_rollback` is reserved by dipdup, this special handler will be discussed later.
+
+### Atomicity and persistency
+
+Here's a few important things to know before running your indexer:
+
+* __WARNING!__ Make sure that database you're connecting to is used by dipdup exclusively. When index configuration or models change the whole database will be dropped and indexing will start from scratch.
+* Do not rename existing indexes in config file without cleaning up database first, didpup won't handle this renaming automatically and will consider renamed index as a new one.
+* Multiple indexes pointing to different contracts must not reuse the same models because synchronization is performed by index first and then by block.
+* Reorg messages signal about chain reorganizations, when some blocks, including all operations, are rolled back in favor of blocks with higher weight. Chain reorgs happen quite often, so it's not something you can ignore. You have to handle such messages correctly, otherwise you will likely accumulate duplicate data or, worse, invalid data. By default Dipdup will start indexing from scratch on such messages. To implement your own rollback logic edit generated `on_rollback` handler.
+
 ### Run your dapp
 
 Now everything is ready to run your indexer:
+
 ```shell
 $ dipdup -c config.yml run
 ```
@@ -165,14 +181,21 @@ $ docker-compose up dipdup
 
 Note that can use `DIPDUP_DATABASE_PASSWORD` environment variable to avoid storing database password in `dipdup.yml`.
 
-### Atomicity and persistency
+### Optional: configure Hasura GraphQL Engine
 
-Here's a few important things to keep in mind while running your indexer:
+`init` command generates Hasura metadata JSON in the package root. You can use `configure-graphql` command to apply it to the running GraphQL Engine instance:
+
+```shell
+$ dipdup -c config.yml configure-graphql --url http://127.0.0.1:8080 --admin-secret changeme
+```
+
+Or if using included `docker-compose.yml` example:
+
+```shell
+$ docker-compose up -d graphql-engine
+$ docker-compose up configure-graphql
+```
 
-* Do not rename existing indexes in config file without cleaning up database first, didpup won't handle this renaming automatically and will consider renamed index as a new one.
-* Make sure that database you're connecting to is used by dipdup exclusively. When index configuration or models will change the whole database will be dropped and indexing will start from scratch.
-* Multiple indexes pointing to different contracts must not reuse the same models because synchronization is performed by index first and then by block.
-* Reorg messages signal about chain reorganizations, when some blocks, including all operations, are rolled back in favor of blocks with higher weight. Chain reorgs happen quite often, so it's not something you can ignore. You have to handle such messages correctly, otherwise you will likely accumulate duplicate data or, worse, invalid data. By default Dipdup will start indexing from scratch on such messages. To implement your own rollback logic edit generated `on_rollback` handler.
 
 ### Optional: configure logging
 

docker-compose.yml

@@ -30,5 +30,11 @@ services:
     restart: always
     env_file: secrets.env
 
+  configure-graphql:
+    <<: *x-dipdup
+    command: ["-c", "dipdup.yml", "configure-graphql", "--url", "http://graphql-engine:8080", "--admin-secret", "changeme"]
+    depends_on:
+      - graphql-engine
+  
 volumes:
   db:

src/dipdup/__init__.py

@@ -1 +1 @@
-__version__ = '3.0.4'
+__version__ = '0.0.0'

src/dipdup/cli.py

@@ -1,13 +1,15 @@
 import asyncio
 import hashlib
+import json
 import logging
 import os
 import sys
 from dataclasses import dataclass
 from functools import wraps
 from os.path import dirname, join
-from typing import Dict, List
+from typing import Dict, List, Optional
 
+import aiohttp
 import click
 from tortoise import Tortoise
 from tortoise.exceptions import OperationalError
@@ -147,3 +149,35 @@ async def init(ctx):
     await codegen.fetch_schemas(config)
     await codegen.generate_types(config)
     await codegen.generate_handlers(config)
+    await codegen.generate_hasura_metadata(config)
+
+
+@cli.command(help='Configure Hasura GraphQL Engine')
+@click.option('--url', type=str, help='Hasura GraphQL Engine URL', default='http://127.0.0.1:8080')
+@click.option('--admin-secret', type=str, help='Hasura GraphQL Engine admin secret', default=None)
+@click.pass_context
+@click_async
+async def configure_graphql(ctx, url: str, admin_secret: Optional[str]):
+    config: DipDupConfig = ctx.obj.config
+
+    url = url.rstrip("/")
+    hasura_metadata_path = join(config.package_path, 'hasura_metadata.json')
+    with open(hasura_metadata_path) as file:
+        hasura_metadata = json.load(file)
+    headers = {}
+    if admin_secret:
+        headers['X-Hasura-Admin-Secret'] = admin_secret
+    async with aiohttp.ClientSession() as session:
+        async with session.post(
+            url=f'{url}/v1/query',
+            data=json.dumps(
+                {
+                    "type": "replace_metadata",
+                    "args": hasura_metadata,
+                },
+            ),
+            headers=headers,
+        ) as resp:
+            result = await resp.json()
+            if not result.get('message') == 'success':
+                raise Exception(result)

src/dipdup/codegen.py

@@ -1,13 +1,15 @@
+import importlib
 import json
 import logging
 import os
 import subprocess
 from contextlib import suppress
 from os import mkdir
 from os.path import dirname, exists, join
-from typing import Any, Dict
+from typing import List
 
 from jinja2 import Template
+from tortoise import Model, fields
 
 from dipdup.config import ROLLBACK_HANDLER, DipDupConfig
 from dipdup.datasources.tzkt.datasource import TzktDatasource
@@ -18,7 +20,7 @@
 async def create_package(config: DipDupConfig):
     try:
         package_path = config.package_path
-    except ImportError:
+    except (ImportError, ModuleNotFoundError):
         package_path = join(os.getcwd(), config.package)
         mkdir(package_path)
         with open(join(package_path, '__init__.py'), 'w'):
@@ -147,3 +149,108 @@ async def generate_handlers(config: DipDupConfig):
             if not exists(handler_path):
                 with open(handler_path, 'w') as file:
                     file.write(handler_code)
+
+
+def _format_array_relationship(related_name: str, table: str, column: str):
+    return {
+        "name": related_name,
+        "using": {
+            "foreign_key_constraint_on": {
+                "column": column,
+                "table": {
+                    "schema": "public",
+                    "name": table,
+                },
+            },
+        },
+    }
+
+
+def _format_object_relationship(table: str, column: str):
+    return {
+        "name": table,
+        "using": {
+            "foreign_key_constraint_on": column,
+        },
+    }
+
+
+def _format_select_permissions(columns: List[str]):
+    return {
+        "role": "user",
+        "permission": {
+            "columns": columns,
+            "filter": {},
+            "allow_aggregations": True,
+        },
+    }
+
+
+def _format_table(name: str):
+    return {
+        "table": {
+            "schema": "public",
+            "name": name,
+        },
+        "object_relationships": [],
+        "array_relationships": [],
+        "select_permissions": [],
+    }
+
+
+def _format_metadata(tables):
+    return {
+        "version": 2,
+        "tables": tables,
+    }
+
+
+async def generate_hasura_metadata(config: DipDupConfig):
+    _logger.info('Generating Hasura metadata')
+    metadata_tables = {}
+    model_tables = {}
+    models = importlib.import_module(f'{config.package}.models')
+
+    for attr in dir(models):
+        model = getattr(models, attr)
+        if isinstance(model, type) and issubclass(model, Model) and model != Model:
+
+            table_name = model._meta.db_table or model.__name__.lower()
+            model_tables[f'models.{model.__name__}'] = table_name
+
+            table = _format_table(table_name)
+            metadata_tables[table_name] = table
+
+    for attr in dir(models):
+        model = getattr(models, attr)
+        if isinstance(model, type) and issubclass(model, Model) and model != Model:
+            table_name = model_tables[f'models.{model.__name__}']
+
+            metadata_tables[table_name]['select_permissions'].append(
+                _format_select_permissions(list(model._meta.db_fields)),
+            )
+
+            for field in model._meta.fields_map.values():
+                if isinstance(field, fields.relational.ForeignKeyFieldInstance):
+                    if not isinstance(field.related_name, str):
+                        raise Exception(f'`related_name` of `{field}` must be set')
+                    related_table_name = model_tables[field.model_name]
+                    metadata_tables[table_name]['object_relationships'].append(
+                        _format_object_relationship(
+                            table=model_tables[field.model_name],
+                            column=field.model_field_name + '_id',
+                        )
+                    )
+                    metadata_tables[related_table_name]['array_relationships'].append(
+                        _format_array_relationship(
+                            related_name=field.related_name,
+                            table=table_name,
+                            column=field.model_field_name + '_id',
+                        )
+                    )
+
+    metadata = _format_metadata(tables=list(metadata_tables.values()))
+
+    metadata_path = join(config.package_path, 'hasura_metadata.json')
+    with open(metadata_path, 'w') as file:
+        json.dump(metadata, file, indent=4)

src/dipdup/config.py

@@ -4,14 +4,15 @@
 import logging.config
 import os
 import sys
+from os import environ as env
 from os.path import dirname
 from typing import Any, Callable, Dict, List, Optional, Type, Union
 
 from attr import dataclass
 from cattrs_extras.converter import Converter
 from ruamel.yaml import YAML
-from tortoise import Model, Tortoise
-from os import environ as env
+from tortoise import Tortoise
+
 from dipdup.models import IndexType, State
 
 ROLLBACK_HANDLER = 'on_rollback'
@@ -62,7 +63,6 @@ def connection_string(self):
         return f'{self.driver}://{self.user}:{self.password}@{self.host}:{self.port}/{self.database}'
 
 
-
 @dataclass(kw_only=True)
 class TzktDatasourceConfig:
     """TzKT datasource config

src/dipdup/datasources/tzkt/datasource.py

@@ -28,7 +28,7 @@ def __init__(
         operation_index_configs: List[OperationIndexConfig],
     ):
         super().__init__()
-        self._url = url
+        self._url = url.rstrip('/')
         self._operation_index_configs = {config.contract: config for config in operation_index_configs}
         self._synchronized = asyncio.Event()
         self._callback_lock = asyncio.Lock()

src/dipdup_hic_et_nunc/handlers/on_mint.py

@@ -11,4 +11,4 @@ async def on_mint(
     mint: HandlerContext[Mint],
     operations: List[OperationData],
 ) -> None:
-    ...
+    await Address.get_or_create(address=mint.parameter.address)

src/dipdup_hic_et_nunc/hasura_metadata.json

@@ -0,0 +1,67 @@
+{
+    "version": 2,
+    "tables": [
+        {
+            "table": {
+                "schema": "public",
+                "name": "address"
+            },
+            "object_relationships": [],
+            "array_relationships": [
+                {
+                    "name": "tokens",
+                    "using": {
+                        "foreign_key_constraint_on": {
+                            "column": "holder_id",
+                            "table": {
+                                "schema": "public",
+                                "name": "token"
+                            }
+                        }
+                    }
+                }
+            ],
+            "select_permissions": [
+                {
+                    "role": "user",
+                    "permission": {
+                        "columns": [
+                            "address"
+                        ],
+                        "filter": {},
+                        "allow_aggregations": true
+                    }
+                }
+            ]
+        },
+        {
+            "table": {
+                "schema": "public",
+                "name": "token"
+            },
+            "object_relationships": [
+                {
+                    "name": "address",
+                    "using": {
+                        "foreign_key_constraint_on": "holder_id"
+                    }
+                }
+            ],
+            "array_relationships": [],
+            "select_permissions": [
+                {
+                    "role": "user",
+                    "permission": {
+                        "columns": [
+                            "token_id",
+                            "token_info",
+                            "id"
+                        ],
+                        "filter": {},
+                        "allow_aggregations": true
+                    }
+                }
+            ]
+        }
+    ]
+}