Fix some Websocket issues, remove Index.head relation (#144)

* Fix action order in callbacks, refactor datasource class

* Fix race in emitter

* typo

* Another typo

* Another typo

* Another typo

* Another typo

* Let callbacks handle locks by themselves

* FFFFUUU

* Fix missing level on emit

* Another stupid typo

* Emit timeout

* Capture exceptions early

* Log dead tasks

* Drop pyee ☠

* Rename callbacks

* Unlink head

* Cleanup

* Cleanup, fix ugly database logger setting

* Typos

* Update changelog

* BlockCache improvements

* typo;

* Ignore missing head

* Fix dipdup initialization order

* Changelog

* Cleanup

* Fix cleaning up old blocks

* Drop Index.hash

* Cleanup

* Cleanup

* Drop `stateless` fields

* Drop BlockCache, simplify block hash checks

* Cleanup

* Check operation status before converting

* Comment

* Changelog

* Cleanup, drop useless self.level check in datasource

* Head constraint

* Revert multiple heads, add docs

* Changelog

* Lint

* Kill me please

* Dont do sql in ws loop

* Head sub, typing

* Codestyle, cache blocks on init, bigmap fixes

* Typo

* Docs

Author: droserasprout

CHANGELOG.md

@@ -5,16 +5,24 @@
 ### Added
 
 * Human-readable `CHANGELOG.md` 🕺
-* `--forbid-reindexing` option added to `dipdup run` command. When this flag is set DipDup will raise `ReindexingRequiredError` instead of truncating database when reindexing is triggered for any reason.
+* `--forbid-reindexing` option added to `dipdup run` command. If this flag is set, DipDup will raise `ReindexingRequiredError` when reindexing is triggered for any reason. A database won't be truncated. To continue indexing with existing database run `UPDATE dipdup_schema SET reindex = NULL;`
+
+### Changed
+
+* Migration to this version requires reindexing.
+* `dipdup_index.head_id` foreign key removed. `dipdup_head` table still contains the latest blocks from Websocket received by each datasource.
 
 ### Fixed
 
-* Removed unnecessary calls to TzKT API during the partial sync.
+* Removed unnecessary calls to TzKT API.
 * Fixed removal of PostgreSQL extensions (`timescaledb`, `pgcrypto`) by function `truncate_database` triggered on reindex.
-* Fixed updating relation between index and head in DB.
 * Fixed creation of missing project package on `init`.
 * Fixed invalid handler callbacks generated on `init`.
 * Fixed detection of existing types in the project.
+* Fixed race condition caused by event emitter concurrency.
+* Capture unknown exceptions with Sentry before wrapping to `DipDupError`.
+* Fixed job scheduler start delay.
+* Fixed processing of reorg messages.
 
 ## 3.0.1 - 2021-09-24
 

poetry.lock

@@ -30,7 +30,6 @@ tortoise-orm = "0.17.5"
 pydantic = "^1.8.1"
 aiosignalrcore = "^0.9.2"
 fcache = "^0.4.7"
-pyee = "^8.1.0"
 APScheduler = "^3.7.0"
 sentry-sdk = "^1.1.0"
 pyhumps = "^3.0.2"

pyproject.toml

@@ -1,6 +1,6 @@
 from dipdup.context import HookContext
 from dipdup.datasources.datasource import Datasource
-from dipdup.exceptions import ReindexingReason
+from dipdup.enums import ReindexingReason
 
 
 async def on_rollback(

src/demo_hic_et_nunc/hooks/on_rollback.py

@@ -1,6 +1,6 @@
 from dipdup.context import HookContext
 from dipdup.datasources.datasource import Datasource
-from dipdup.exceptions import ReindexingReason
+from dipdup.enums import ReindexingReason
 
 
 async def on_rollback(

src/demo_quipuswap/hooks/on_rollback.py

@@ -1,6 +1,6 @@
 from dipdup.context import HookContext
 from dipdup.datasources.datasource import Datasource
-from dipdup.exceptions import ReindexingReason
+from dipdup.enums import ReindexingReason
 
 
 async def on_rollback(

src/demo_registrydao/hooks/on_rollback.py

@@ -1,6 +1,6 @@
 from dipdup.context import HookContext
 from dipdup.datasources.datasource import Datasource
-from dipdup.exceptions import ReindexingReason
+from dipdup.enums import ReindexingReason
 
 
 async def on_rollback(

src/demo_tezos_domains/hooks/on_rollback.py

@@ -1,6 +1,6 @@
 from dipdup.context import HookContext
 from dipdup.datasources.datasource import Datasource
-from dipdup.exceptions import ReindexingReason
+from dipdup.enums import ReindexingReason
 
 
 async def on_rollback(

src/demo_tezos_domains_big_map/hooks/on_rollback.py

@@ -1,6 +1,6 @@
 from dipdup.context import HookContext
 from dipdup.datasources.datasource import Datasource
-from dipdup.exceptions import ReindexingReason
+from dipdup.enums import ReindexingReason
 
 
 async def on_rollback(

src/demo_tzbtc/hooks/on_rollback.py

@@ -1,6 +1,6 @@
 from dipdup.context import HookContext
 from dipdup.datasources.datasource import Datasource
-from dipdup.exceptions import ReindexingReason
+from dipdup.enums import ReindexingReason
 
 
 async def on_rollback(