4.0.0-rc1

4.0.0-rc1 - 2021-12-02

⚠ Migration

• Run dipdup schema approve command on every database you want to use with 4.0.0-rc1. Running dipdup migrate is not necessary since spec_version hasn't changed in this release.

Added

• cli: Added run --early-realtime flag to establish a realtime connection before all indexes are synchronized.
• cli: Addedrun --merge-subscriptions flag to subscribe to all operations/big map diffs during realtime indexing. This flag helps to avoid reaching TzKT subscriptions limit (currently 10000 channels).
• cli: Added status command to print the current status of indexes from the database.
• cli: Added config export [--unsafe] command to print config after resolving all links and variables. Add --unsafe option to substitute environment variables.
• cli: Added cache show command to get information about file caches used by DipDup.
• config: Added first_level and last_level optional fields to TemplateIndexConfig. These limits are applied after ones from the template itself.
• config: Added daemon boolean field to JobConfig to run a single callback indefinitely. Conflicts with crontab and interval fields.
• config: Added advanced top-level section with following fields:

advanced:
  early_realtime: False
  merge_subscriptions: False
  oneshot: False
  postpone_jobs: False
  reindex:
    manual: exception
    migration: wipe
    rollback: ignore
    config_modified: exception
    schema_modified: wipe

ReindexingRequiredError exception is raised by default when reindexing is triggered. CLI flags have priority over self-titled AdvancedConfig fields.

Fixed

• cli: Fixed crashes and output inconsistency when piping DipDup commands.
• cli: Fixed schema wipe --immune flag being ignored.
• codegen: Fixed missing imports in handlers generated during init.
• coinbase: Fixed possible data inconsistency caused by caching enabled for method get_candles.
• hasura: Fixed unnecessary reconfiguration in restart.
• http: Fixed increasing sleep time between failed request attempts.
• index: Fixed invocation of head index callback.
• index: Fixed CallbackError raised instead of ReindexingRequiredError in some cases.
• tzkt: Fixed resubscribing when realtime connectivity is lost for a long time.
• tzkt: Fixed sending useless subscription requests when adding indexes in runtime.
• tzkt: Fixed get_originated_contracts and get_similar_contracts methods whose output was limited to HTTPConfig.batch_size field.
• tzkt: Fixed lots of SignalR bugs by replacing aiosignalrcore library with pysignalr.

Changed

• cli: schema wipe command now requires confirmation when invoked in the interactive shell.
• cli: schema approve command now also causes a recalculation of schema and index config hashes.
• index: DipDup will recalculate respective hashes if reindexing is triggered with config_modified: ignore or schema_modified: ignore in advanced config.

Deprecated

• cli: run --oneshot option is deprecated and will be removed in the next major release. The oneshot mode applies automatically when last_level field is set in the index config.
• cli: clear-cache command is deprecated and will be removed in the next major release. Use cache clear command instead.

Performance

• config: Configuration files are loaded 10x times faster.
• index: Number of operations processed by matcher reduced by 40%-95% depending on the number of addresses and entrypoints used.
• tzkt: Rate limit was increased. Try to set connection_timeout to a higher value if requests fail with ConnectionTimeout exception.
• tzkt: Improved performance of response deserialization.