Database-level rollbacks (copy-on-write) (#392)

* Initial VersionedModel

* Fix flakeheaven config

* Whooooah, a working system-wide model!!1!

* TransactionManager and sync_level filtering

* Register models

* Fix TransactionManager registration

* Cleanup

* Cleanup

* Cleanup task

* Per-index diffs

* Ignore immune tables

* Rename DatabaseTransaction

* Remove cleanup task

* Make callbacks and transactions private in context

* Cleanup if at least one index had messages in queue

* Faster model update data

* Use model name instead of table, codegen, rollback (except updating index level

* Refresh rolled back level

* Fix is_model_class, ignore reverse rekations

* Fix HookContext not passed to hook

* Drop single-level rollback

* Logs

* Drop `on_rollback`

* Enforce our model class, faster model preparation

* Log

* Forbid bulk model methods

* Changelog, fix operation batch level check

* Fix log

* Release notes WIP

* Tag GHCR images by ref_name

* Try to fix ref URLs

* Again

* Again...

* Bump actions, use metadata one to escape chars

* Build for branches?

* Lint

* Fix changelog

* Lint, config export --full

* Fix Hasura metadata generation

* Fix `hasura configure` command crash when models have empty `Meta.table`

* Allow customizing `server_name` and `release` tags with corresponding fields in Sentry config

* Refactoring

* Preprocess JSON from modelupdate

* Basic insert test

* Still testing

* Fix updates

* Lint

* More tests

* Fix optionals

* Deferred update

* Refactoring, save original data

* Update docstrings

* bulk_update and bulk_create

* Update with prefetch

* Do not save updates until queryset is awaited

* Drop reverse relations form versioned data

* Fix crash when adding an index with new subscriptions in runtime

* Another backwards relation fix

* AGEN

* More optimal deserializing

* Save diffs when possible

* Skip empty updates, more logs

* Unify index sync_level

* Fix prometheus metric, rename head_level to sync_level (more correct)

* Update changelog

* Update release notes

Author: droserasprout

.github/workflows/build.yml

@@ -2,12 +2,9 @@ name: Build
 on:
   push:
     branches:
-      - 'master'
+      - "**"
     tags:
       - '*.*.*'
-  pull_request:
-    branches:
-      - 'master'
 
 jobs:
   build:
@@ -17,6 +14,7 @@ jobs:
       ACTIONS_ALLOW_UNSECURE_COMMANDS: true
       DOCKER_REGISTRY: ghcr.io
       DOCKER_IMAGE_NAME: ${{ github.repository }}
+
     steps:
       - name: Check out the repo
         uses: actions/checkout@v2
@@ -40,11 +38,14 @@ jobs:
         uses: docker/metadata-action@v3
         with:
           images: ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}
+          tags: |
+            type=ref,event=branch
+            type=ref,event=tag
           flavor: |
             latest=false
 
       - name: Publish image on GHCR
-        uses: docker/build-push-action@v2
+        uses: docker/build-push-action@v3
         with:
           context: .
           file: Dockerfile

CHANGELOG.md

@@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog],
 and this project adheres to [Semantic Versioning].
 
+## [Unreleased]
+
+### Added
+
+- cli: Added `config export --full` flag to resolve templates before printing config.
+- config: Added `advanced.rollback_depth` field, a number of levels to keep in a database for rollback.
+- context: Added `rollback` method to perform database rollback.
+- database: Added an internal `ModelUpdate` model to store the latest database changes.
+
+### Fixed
+
+- prometheus: Fixed updating `dipdup_index_handlers_matched_total` metric.
+
+### Changed
+
+- codegen: `on_index_rollback` hook calls `ctx.rollback` by default.
+- database: Project models must be subclassed from `dipdup.models.Model`
+- database: `bulk_create` and `bulk_update` model methods are no longer supported.
+
+### Removed
+
+- hooks: Removed deprecated `on_rollback` hook.
+- index: Do not try to avoid single-level rollbacks by comparing operation hashes.
+
 ## [5.2.5] - 2022-07-26
 
 ### Fixed

docs/SUMMARY.md

@@ -84,6 +84,7 @@
 
 * [Changelog](CHANGELOG.md)
 * [Release notes](release-notes/README.md)
+  * [6.0.0 (unreleased)](release-notes/6.0.0.md)
   * [5.1.0](release-notes/5.1.0.md)
   * [5.0.0](release-notes/5.0.0.md)
   * [4.2.0](release-notes/4.2.0.md)

docs/advanced/context/reference.md

@@ -1,7 +1,7 @@
 
   <span class="target" id="module-dipdup.context"></span><dl class="py class">
 <dt class="sig sig-object py" id="dipdup.context.DipDupContext">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">DipDupContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">CallbackManager</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.DipDupContext" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">DipDupContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">CallbackManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">transactions</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">TransactionManager</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.DipDupContext" title="Permalink to this definition">¶</a></dt>
 <dd><p>Class to store application context</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -71,13 +71,13 @@
 
 <dl class="py class">
 <dt class="sig sig-object py" id="dipdup.context.HandlerContext">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">HandlerContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">CallbackManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logger</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">FormattedLogger</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">handler_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.HandlerConfig" title="dipdup.config.HandlerConfig"><span class="pre">HandlerConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">datasource</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">TzktDatasource</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.HandlerContext" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">HandlerContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">CallbackManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">transactions</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">TransactionManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logger</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">FormattedLogger</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">handler_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.HandlerConfig" title="dipdup.config.HandlerConfig"><span class="pre">HandlerConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">datasource</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">TzktDatasource</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.HandlerContext" title="Permalink to this definition">¶</a></dt>
 <dd><p>Common handler context.</p>
 </dd></dl>
 
 <dl class="py class">
 <dt class="sig sig-object py" id="dipdup.context.HookContext">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">HookContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">CallbackManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logger</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">FormattedLogger</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">hook_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.HookConfig" title="dipdup.config.HookConfig"><span class="pre">HookConfig</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.HookContext" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">HookContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">CallbackManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">transactions</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">TransactionManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logger</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">FormattedLogger</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">hook_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.HookConfig" title="dipdup.config.HookConfig"><span class="pre">HookConfig</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.HookContext" title="Permalink to this definition">¶</a></dt>
 <dd><p>Hook callback context.</p>
 </dd></dl>
 

docs/cli-reference.md

@@ -76,7 +76,7 @@
 </section>
 <section id="dipdup-config-export">
 <h3>export<a class="headerlink" href="#dipdup-config-export" title="Permalink to this heading">¶</a></h3>
-<p>Print config after resolving all links and templates.</p>
+<p>Print config after resolving all links and, optionally, templates.</p>
 <p>WARNING: Avoid sharing output with 3rd-parties when <cite>–unsafe</cite> flag set - it may contain secrets!</p>
 <div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup config <span class="nb">export</span> <span class="o">[</span>OPTIONS<span class="o">]</span>
 </pre></div>
@@ -88,6 +88,12 @@
 <dd><p>Resolve environment variables or use default values from config.</p>
 </dd></dl>
 
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-config-export-full">
+<span class="sig-name descname"><span class="pre">--full</span></span><span class="sig-prename descclassname"></span><a class="headerlink" href="#cmdoption-dipdup-config-export-full" title="Permalink to this definition">¶</a></dt>
+<dd><p>Resolve index templates.</p>
+</dd></dl>
+
 </section>
 </section>
 <section id="dipdup-hasura">