Performance optimizations, CLI commands, changelog (#174)

* Cache some properties in config

* Faster

* Typo

* Drop expensive check

* Event faster configs, add --hashes flag

* Slightly faster parsing of storage

* typing_extensions are expensive!

* less logs 🌚

* Lint

* Pretty prints in new CLI commands

* cli commands, fix piping, update changelog

* Changelog, small fixes

* changelog

* Lint

* force_single_line >:3

Author: droserasprout

CHANGELOG.md

@@ -1,32 +1,67 @@
 # Changelog
 
-## 4.0.0 - [unreleased]
+Please use [this](https://docs.gitlab.com/ee/development/changelog.html) document as guidelines to keep a changelog.
 
-### Fixed
+## 4.0.0-rc1 - [unreleased]
 
-* tzkt: Realtime connection watchdog is no longer triggered after TzKT outtage.
-* tzkt: Adding many indexes in runtime no longer leads to sending useless subscription requests.
-* tzkt: Fixed `get_originated_contracts` and `get_similar_contracts` methods whose output was limited to `batch_size` tunable.
-* index: Fixed head index callback invocation.
-* coinbase: Candles caching disabled.
+### ⚠ Migration
 
-### Changed
+* Run `dipdup schema approve --hashes` command on every database you want to use with 4.0.0-rc1.
+
+### Added
+
+* cli: Added `dipdup run --skip-hasura` flag to skip updating Hasura metadata.
+* cli: Added `dipdip run --early-realtime` flag to establish a realtime connection before all indexes are synchronized.
+* cli: Added`dipdup run --merge-subscriptions`  flag to subscribe to all operations/big map diffs during realtime indexing. This flag helps to avoid reaching TzKT subscriptions limit (currently 10000 channels). 
+* cli: Added `dipdup status` command to print the current status of indexes from database
+* cli: Added `dipdup config export [--unsafe]` command to print config after resolving all links and variables. Add `--unsafe` option to substitute environment variables.
+* cli: Added `dipdup cache show` command to get information about file caches used by DipDup.
+* cli: Added `dipdup schema approve --hashes` flag to recalculate schema and index config hashes on the next run.
+* config: Added `first_level` and `last_level` optional fields to `TemplateIndexConfig`. These limits are applied after ones from the template itself.
+* config: Added `daemon` boolean field to `JobConfig` to run a single callback indefinitely. Conflicts with `crontab` and `interval` fields.
+* config: Added `advanced` top-level section with following fields:
+
+```yaml
+advanced:
+  early_realtime: False
+  merge_subscriptions: False
+  oneshot: False
+  postpone_jobs: False
+  reindex:
+    manual: exception
+    migration: wipe
+    rollback: ignore
+    config_modified: exception
+    schema_modified: wipe
+  skip_hasura: False
+```
+
+`ReindexingRequiredError` exception raised by default when reindexing is triggered. CLI flags have priority over self-titled `AdvancedConfig` fields.
 
-* cli: `run --oneshot` option is deprecated and will be removed in the next major release. Oneshot mode applied automatically when `first_level` and `last_level` fields are set in index config.  
+### Fixed
+
+* cli: Fixed crashes and output inconsistency when piping DipDup commands.
 * codegen: Fixed missing imports in handlers generated during init.
+* coinbase: Fixed possible data inconsistency caused by caching enabled for method `get_candles`.
+* http: Fixed increasing sleep time between failed request attempts.
+* index: Fixed invocation of head index callback.
+* index: Fixed `CallbackError` raised instead of `ReindexingRequiredError` in some cases.
+* tzkt: Fixed resubscribing when realtime connectivity is lost for a long time.
+* tzkt: Fixed sending useless subscription requests when adding indexes in runtime.
+* tzkt: Fixed `get_originated_contracts` and `get_similar_contracts` methods whose output was limited to `HTTPConfig.batch_size` field.
+* tzkt: Fixed lots of SignalR bugs by replacing `aiosignalrcore` library with `pysignalr`.
 
-### Added
+### Deprecated
 
-* cli: New flag `dipdip run --early-realtime` to establish a real-time connection before all indexes are synchronized.
-* cli: New flag `dipdup run --skip-hasura` to skip updating Hasura metadata.
-* cli: New command `dipdup status` to print the current status of indexes from database
-* cli: New command `dipdup config` to print config after resolving all links and variables
-* config: Added optional fields `first_level` and `last_level` to `TemplateIndexConfig`. These limits are applied after ones from the template itself.
+* cli: `run --oneshot` option is deprecated and will be removed in the next major release. The oneshot mode applies automatically when `last_level` field is set in the index config.
+* cli: `clear-cache` command is deprecated and will be removed in the next major release. Use `cache clear` command instead.
 
-### Improved
+### Performance
 
-* index: Time required to initialize indexes presented in database reduced by ~25%.
-* tzkt: Replaced `aiosignalrcore` library with `pysignalr`.
+* config: Configuration files are loaded 10x times faster.
+* index: Number of operations processed by matcher reduced by 40%-95% depending on number of addresses and entrypoints used.
+* tzkt: Rate limit was increased. Try to set `connection_timeout` to a higher value if requests fail with `ConnectionTimeout` exception.
+* tzkt: Improved performance of response deserialization. 
 
 ## 3.1.3 - 2021-11-15
 
@@ -45,7 +80,7 @@
 
 ### Fixed
 
-* Fixed loss of real-time subscriptions occurred after TzKT API outage.
+* Fixed loss of realtime subscriptions occurred after TzKT API outage.
 * Fixed updating schema hash in `schema approve` command.
 * Fixed possible crash occurred while Hasura is not ready.
 
@@ -86,7 +121,7 @@
 
 * Fixed unexpected reindexing caused by the bug in processing zero- and single-level rollbacks.
 * Removed unnecessary file IO calls that could cause `PermissionError` exception in Docker environments.
-* Fixed possible violation of block-level atomicity during real-time indexing.
+* Fixed possible violation of block-level atomicity during realtime indexing.
 
 ### Changes
 

poetry.lock

@@ -55,6 +55,7 @@ pytest-cov = "^3.0.0"
 testcontainers = "^3.4.1"
 types-pytz = "^2021.1.2"
 types-tabulate = "^0.8.2"
+pprofile = "^2.1.0"
 
 [tool.poetry.extras]
 pytezos = ["pytezos"]
@@ -64,6 +65,7 @@ dipdup = 'dipdup.cli:cli'
 
 [tool.isort]
 line_length = 140
+force_single_line = true
 
 [tool.black]
 line-length = 140

pyproject.toml

@@ -1,6 +1,7 @@
 from enum import IntEnum
 
-from tortoise import Model, fields
+from tortoise import Model
+from tortoise import fields
 
 
 class SwapStatus(IntEnum):

src/demo_hic_et_nunc/models.py

@@ -3,7 +3,8 @@
 
 from __future__ import annotations
 
-from pydantic import BaseModel, Extra
+from pydantic import BaseModel
+from pydantic import Extra
 
 
 class CollectParameter(BaseModel):

src/demo_hic_et_nunc/types/hen_minter/parameter/collect.py

@@ -3,7 +3,8 @@
 
 from __future__ import annotations
 
-from pydantic import BaseModel, Extra
+from pydantic import BaseModel
+from pydantic import Extra
 
 
 class MintOBJKTParameter(BaseModel):