schema wipe confirmation, update hashes on schema approve and reason: ignore, update docs and help (#175)

* Update builtin logging configs

* Docs, tiny optimization

* Always update hashes on approval, schema wipe confirmation

* Lint

* Fix --immune argument

Author: droserasprout

CHANGELOG.md

@@ -6,17 +6,16 @@ Please use [this](https://docs.gitlab.com/ee/development/changelog.html) documen
 
 ### ⚠ Migration
 
-* Run `dipdup schema approve --hashes` command on every database you want to use with 4.0.0-rc1.
+* Run `dipdup schema approve` command on every database you want to use with 4.0.0-rc1. Running `dipdup migrate` is not necessary since `spec_version` hasn't changed in this release.
 
 ### Added
 
-* cli: Added `dipdup run --skip-hasura` flag to skip updating Hasura metadata.
-* cli: Added `dipdip run --early-realtime` flag to establish a realtime connection before all indexes are synchronized.
-* cli: Added`dipdup run --merge-subscriptions`  flag to subscribe to all operations/big map diffs during realtime indexing. This flag helps to avoid reaching TzKT subscriptions limit (currently 10000 channels). 
-* cli: Added `dipdup status` command to print the current status of indexes from database
-* cli: Added `dipdup config export [--unsafe]` command to print config after resolving all links and variables. Add `--unsafe` option to substitute environment variables.
-* cli: Added `dipdup cache show` command to get information about file caches used by DipDup.
-* cli: Added `dipdup schema approve --hashes` flag to recalculate schema and index config hashes on the next run.
+* cli: Added `run --skip-hasura` flag to skip updating Hasura metadata.
+* cli: Added `run --early-realtime` flag to establish a realtime connection before all indexes are synchronized.
+* cli: Added`run --merge-subscriptions`  flag to subscribe to all operations/big map diffs during realtime indexing. This flag helps to avoid reaching TzKT subscriptions limit (currently 10000 channels).
+* cli: Added `status` command to print the current status of indexes from the database.
+* cli: Added `config export [--unsafe]` command to print config after resolving all links and variables. Add `--unsafe` option to substitute environment variables.
+* cli: Added `cache show` command to get information about file caches used by DipDup.
 * config: Added `first_level` and `last_level` optional fields to `TemplateIndexConfig`. These limits are applied after ones from the template itself.
 * config: Added `daemon` boolean field to `JobConfig` to run a single callback indefinitely. Conflicts with `crontab` and `interval` fields.
 * config: Added `advanced` top-level section with following fields:
@@ -36,11 +35,12 @@ advanced:
   skip_hasura: False
 ```
 
-`ReindexingRequiredError` exception raised by default when reindexing is triggered. CLI flags have priority over self-titled `AdvancedConfig` fields.
+`ReindexingRequiredError` exception is raised by default when reindexing is triggered. CLI flags have priority over self-titled `AdvancedConfig` fields.
 
 ### Fixed
 
 * cli: Fixed crashes and output inconsistency when piping DipDup commands.
+* cli: Fixed `schema wipe --immune` flag being ignored.
 * codegen: Fixed missing imports in handlers generated during init.
 * coinbase: Fixed possible data inconsistency caused by caching enabled for method `get_candles`.
 * http: Fixed increasing sleep time between failed request attempts.
@@ -51,6 +51,12 @@ advanced:
 * tzkt: Fixed `get_originated_contracts` and `get_similar_contracts` methods whose output was limited to `HTTPConfig.batch_size` field.
 * tzkt: Fixed lots of SignalR bugs by replacing `aiosignalrcore` library with `pysignalr`.
 
+## Changed
+
+* cli: `dipdup schema wipe` command now requires confirmation when invoked in the interactive shell.
+* cli: `dipdup schema approve` command now also causes a recalculation of schema and index config hashes.
+* index: DipDup will recalculate respective hashes if reindexing is triggered with `config_modified: ignore` or `schema_modified: ignore` in advanced config.
+
 ### Deprecated
 
 * cli: `run --oneshot` option is deprecated and will be removed in the next major release. The oneshot mode applies automatically when `last_level` field is set in the index config.
@@ -59,7 +65,7 @@ advanced:
 ### Performance
 
 * config: Configuration files are loaded 10x times faster.
-* index: Number of operations processed by matcher reduced by 40%-95% depending on number of addresses and entrypoints used.
+* index: Number of operations processed by matcher reduced by 40%-95% depending on the number of addresses and entrypoints used.
 * tzkt: Rate limit was increased. Try to set `connection_timeout` to a higher value if requests fail with `ConnectionTimeout` exception.
 * tzkt: Improved performance of response deserialization. 
 

src/dipdup/cli.py

@@ -3,6 +3,7 @@
 import os
 import signal
 import subprocess
+import sys
 from contextlib import suppress
 from dataclasses import dataclass
 from functools import wraps
@@ -341,26 +342,27 @@ async def schema(ctx):
     ...
 
 
-@schema.command(name='approve', help='Continue indexing with the same schema after crashing with `ReindexingRequiredError`')
-@click.option('--hashes', is_flag=True, help='Recalculate hashes of schema and index configs saved in database')
+@schema.command(name='approve', help='Continue to use existing schema after reindexing was triggered')
 @click.pass_context
 @cli_wrapper
 async def schema_approve(ctx, hashes: bool):
     config: DipDupConfig = ctx.obj.config
     url = config.database.connection_string
     models = f'{config.package}.models'
 
+    _logger.info('Approving schema `%s`', url)
+
     async with tortoise_wrapper(url, models):
-        schema = await Schema.filter().get()
-        if not schema.reindex:
-            return
+        # FIXME: Non-nullable fields
+        await Schema.filter().update(
+            reindex=None,
+            hash='',
+        )
+        await Index.filter().update(
+            config_hash='',
+        )
 
-        schema.reindex = None  # type: ignore
-        if hashes:
-            # FIXME: Non-nullable fields
-            schema.hash = ''  # type: ignore
-            await Index.filter().update(config_hash='')
-        await schema.save()
+    _logger.info('Schema approved')
 
 
 @schema.command(name='wipe', help='Drop all database tables, functions and views')
@@ -372,13 +374,32 @@ async def schema_wipe(ctx, immune: bool):
     url = config.database.connection_string
     models = f'{config.package}.models'
 
+    try:
+        assert sys.__stdin__.isatty()
+        click.confirm(f'You\'re about to wipe schema `{url}`. All indexed data will be irreversibly lost, are you sure?', abort=True)
+    except AssertionError:
+        click.echo('Not in a TTY, skipping confirmation')
+    # FIXME: Can't catch asyncio.CancelledError here
+    except click.Abort:
+        click.echo('Aborted')
+        return
+
+    _logger.info('Wiping schema `%s`', url)
+
     async with tortoise_wrapper(url, models):
         conn = get_connection(None)
         if isinstance(config.database, PostgresDatabaseConfig):
-            await wipe_schema(conn, config.database.schema_name, config.database.immune_tables)
+            await wipe_schema(
+                conn=conn,
+                name=config.database.schema_name,
+                # NOTE: Don't be confused by the name of `--immune` flag, we want to drop all tables if it's set.
+                immune_tables=config.database.immune_tables if not immune else (),
+            )
         else:
             await Tortoise._drop_databases()
 
+    _logger.info('Schema wiped')
+
 
 @schema.command(name='export', help='Print schema SQL including `on_reindex` hook')
 @click.pass_context

src/dipdup/configs/debug.yml

@@ -1,25 +1,23 @@
-  version: 1
-  disable_existing_loggers: false
-  formatters:
-    brief:
-      format: "%(levelname)-8s %(name)-20s %(message)s"
-  handlers:
-    console:
-      level: DEBUG
-      formatter: brief
-      class: logging.StreamHandler
-      stream : ext://sys.stdout
-  loggers:
-    dipdup:
-      level: DEBUG
+version: 1
+disable_existing_loggers: false
+formatters:
+  brief:
+    format: "%(levelname)-8s %(name)-20s %(message)s"
+handlers:
+  console:
+    level: DEBUG
+    formatter: brief
+    class: logging.StreamHandler
+    stream: ext://sys.stdout
+loggers:
+  dipdup:
+    level: DEBUG
 
-    SignalRCoreClient:
-      formatter: brief
-    aiosqlite:
-      level: DEBUG
-    db_client:
-      level: DEBUG
-  root:
+  aiosqlite:
     level: DEBUG
-    handlers:
-      - console
+  db_client:
+    level: DEBUG
+root:
+  level: DEBUG
+  handlers:
+    - console

src/dipdup/configs/logging.yml

@@ -1,25 +1,23 @@
-  version: 1
-  disable_existing_loggers: false
-  formatters:
-    brief:
-      format: "%(levelname)-8s %(name)-20s %(message)s"
-  handlers:
-    console:
-      level: INFO
-      formatter: brief
-      class: logging.StreamHandler
-      stream : ext://sys.stdout
-  loggers:
-    dipdup:
-      level: INFO
+version: 1
+disable_existing_loggers: false
+formatters:
+  brief:
+    format: "%(levelname)-8s %(name)-20s %(message)s"
+handlers:
+  console:
+    level: INFO
+    formatter: brief
+    class: logging.StreamHandler
+    stream: ext://sys.stdout
+loggers:
+  dipdup:
+    level: INFO
 
-    SignalRCoreClient:
-      formatter: brief
-    aiosqlite:
-      level: INFO
-    db_client:
-      level: INFO
-  root:
+  aiosqlite:
     level: INFO
-    handlers:
-      - console
+  db_client:
+    level: INFO
+root:
+  level: INFO
+  handlers:
+    - console

src/dipdup/configs/warning.yml

@@ -1,25 +1,23 @@
-  version: 1
-  disable_existing_loggers: false
-  formatters:
-    brief:
-      format: "%(levelname)-8s %(name)-20s %(message)s"
-  handlers:
-    console:
-      level: WARNING
-      formatter: brief
-      class: logging.StreamHandler
-      stream : ext://sys.stdout
-  loggers:
-    dipdup:
-      level: WARNING
+version: 1
+disable_existing_loggers: false
+formatters:
+  brief:
+    format: "%(levelname)-8s %(name)-20s %(message)s"
+handlers:
+  console:
+    level: WARNING
+    formatter: brief
+    class: logging.StreamHandler
+    stream: ext://sys.stdout
+loggers:
+  dipdup:
+    level: WARNING
 
-    SignalRCoreClient:
-      formatter: brief
-    aiosqlite:
-      level: WARNING
-    db_client:
-      level: WARNING
-  root:
+  aiosqlite:
     level: WARNING
-    handlers:
-      - console
+  db_client:
+    level: WARNING
+root:
+  level: WARNING
+  handlers:
+    - console

src/dipdup/context.py

@@ -118,6 +118,10 @@ async def reindex(self, reason: Optional[Union[str, ReindexingReason, Reindexing
         self.logger.warning('Reindexing initialized, reason: %s, action: %s', reason.value, action.value)
 
         if action == ReindexingAction.ignore:
+            if reason == ReindexingReasonC.schema_modified:
+                await Schema.filter().update(hash='')
+            elif reason == ReindexingReasonC.config_modified:
+                await Index.filter().update(config_hash='')
             return
 
         elif action == ReindexingAction.exception:

src/dipdup/datasources/bcd/datasource.py

@@ -33,9 +33,6 @@ def __init__(
     async def run(self) -> None:
         pass
 
-    async def resync(self) -> None:
-        pass
-
     async def get_tokens(self, address: str) -> List[Dict[str, Any]]:
         tokens, offset = [], 0
         while True: