Unified CLI reference, new pages, proofreading (#386)

* on_rollback -> on_index_rollback

* Poetry usage

* Proofreading

* CLI reference migration WIP

* Lint

* cli fixes, lint

* Merge Improve performance

* Merge caching models

* Rewrite Core concepts

* More about SQL scripts

* Finish cli-reference

* Fix summary

* Typo

* Move CLI reference higher in summary

Author: droserasprout

docs/Makefile

@@ -21,7 +21,7 @@ help:
 
 docs:
 	make html
-	tail -n '+33' _build/html/cli-reference.html | head -n '-63' > cli/reference.md
+	tail -n '+33' _build/html/cli-reference.html | head -n '-63' > cli-reference.md
 	tail -n '+33' _build/html/config-reference.html | head -n '-63' > config/reference.md
 	tail -n '+33' _build/html/context-reference.html | head -n '-63' > advanced/context/reference.md
 	mdbook build  --dest-dir=dist/docs

docs/SUMMARY.md

@@ -21,10 +21,7 @@
   * [Reindexing](advanced/reindexing.md)
   * [Feature flags](advanced/feature-flags.md)
   * [Executing SQL scripts](advanced/sql.md)
-  * [Improving performance](advanced/performance/README.md)
-    * [Working with time series](advanced/performance/time-series.md)
-    * [Tuning datasources](advanced/performance/datasources.md)
-    * [Optimizing database queries](advanced/performance/database.md)
+  * [Improving performance](advanced/performance.md)
   * [Callback context (ctx)](advanced/context/README.md)
     * [Full reference](advanced/context/reference.md)
   * [Internal models](advanced/internal-models.md)
@@ -52,7 +49,6 @@
   * [Processing offchain data](cookbook/offchain-data.md)
   * [Reusing typename for different contracts](cookbook/reusing-typenames.md)
   * [Synchronizing multiple handlers/hooks](cookbook/sync-callbacks.md)
-  * [Caching models](cookbook/caching-models.md)
   * [Multiprocessing](cookbook/multiprocessing.md)
 * [Examples](examples/README.md)
   * [Demo projects](examples/demo-projects.md)
@@ -63,6 +59,7 @@
 
 # Reference Guide
 
+* [Command-line reference](cli-reference.md)
 * [Config file reference](config/README.md)
   * [Full reference](config/reference.md)
   * [advanced](config/advanced.md)
@@ -82,20 +79,6 @@
   * [sentry](config/sentry.md)
   * [spec_version](config/spec_version.md)
   * [templates](config/templates.md)
-* [Command-line reference](cli/README.md)
-  * [Full reference](cli/reference.md)
-  * [cache clear](cli/cache-clear.md)
-  * [cache show](cli/cache-show.md)
-  * [config export](cli/config-export.md)
-  * [hasura configure](cli/hasura-configure.md)
-  * [init](cli/init.md)
-  * [migrate](cli/migrate.md)
-  * [run](cli/run.md)
-  * [schema approve](cli/schema-approve.md)
-  * [schema export](cli/schema-export.md)
-  * [schema init](cli/schema-init.md)
-  * [schema wipe](cli/schema-wipe.md)
-  * [status](cli/status.md)
 
 # What's New
 

docs/advanced/context/README.md

@@ -8,7 +8,7 @@ An instance of the `HandlerContext` class is passed to every handler providing a
 
 ## `.reindex() -> None`
 
-Drops the entire database and starts the indexing process from scratch. `on_rollback` hook calls this helper by default.
+Drops the entire database and starts the indexing process from scratch. `on_index_rollback` hook calls this helper by default.
 
 ## `.add_contract(name, address, typename) -> Coroutine`
 

docs/advanced/context/reference.md

@@ -1,7 +1,7 @@
 
   <span class="target" id="module-dipdup.context"></span><dl class="py class">
 <dt class="sig sig-object py" id="dipdup.context.DipDupContext">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">DipDupContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">dipdup.datasources.datasource.Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="dist/docs/config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">dipdup.config.DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.context.CallbackManager</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.DipDupContext" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">DipDupContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">dipdup.datasources.datasource.Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">dipdup.config.DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.context.CallbackManager</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.DipDupContext" title="Permalink to this definition">¶</a></dt>
 <dd><p>Class to store application context</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -71,13 +71,13 @@
 
 <dl class="py class">
 <dt class="sig sig-object py" id="dipdup.context.HandlerContext">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">HandlerContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">dipdup.datasources.datasource.Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="dist/docs/config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">dipdup.config.DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.context.CallbackManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logger</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.utils.FormattedLogger</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">handler_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="dist/docs/config-reference.html#dipdup.config.HandlerConfig" title="dipdup.config.HandlerConfig"><span class="pre">dipdup.config.HandlerConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">datasource</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.datasources.tzkt.datasource.TzktDatasource</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.HandlerContext" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">HandlerContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">dipdup.datasources.datasource.Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">dipdup.config.DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.context.CallbackManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logger</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.utils.FormattedLogger</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">handler_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.HandlerConfig" title="dipdup.config.HandlerConfig"><span class="pre">dipdup.config.HandlerConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">datasource</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.datasources.tzkt.datasource.TzktDatasource</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.HandlerContext" title="Permalink to this definition">¶</a></dt>
 <dd><p>Common handler context.</p>
 </dd></dl>
 
 <dl class="py class">
 <dt class="sig sig-object py" id="dipdup.context.HookContext">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">HookContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">dipdup.datasources.datasource.Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="dist/docs/config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">dipdup.config.DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.context.CallbackManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logger</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.utils.FormattedLogger</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">hook_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="dist/docs/config-reference.html#dipdup.config.HookConfig" title="dipdup.config.HookConfig"><span class="pre">dipdup.config.HookConfig</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.HookContext" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.context.</span></span><span class="sig-name descname"><span class="pre">HookContext</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">datasources</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">dipdup.datasources.datasource.Datasource</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.DipDupConfig" title="dipdup.config.DipDupConfig"><span class="pre">dipdup.config.DipDupConfig</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">callbacks</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.context.CallbackManager</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logger</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.utils.FormattedLogger</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">hook_config</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="config-reference.html#dipdup.config.HookConfig" title="dipdup.config.HookConfig"><span class="pre">dipdup.config.HookConfig</span></a></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.context.HookContext" title="Permalink to this definition">¶</a></dt>
 <dd><p>Hook callback context.</p>
 </dd></dl>
 

docs/advanced/hooks/default.md

@@ -10,7 +10,7 @@ If your indexer is stateless, you can just drop DB data saved after `to_level` a
 
 ## `on_restart`
 
-This hook executes right before starting indexing. It allows configuring DipDup in runtime based on data from external sources. Datasources are already initialized at the execution time and available at `ctx.datasources`. You can, for example, configure logging here or add contracts and indexes in runtime instead of from static config.
+This hook executes right before starting indexing. It allows configuring DipDup in runtime based on data from external sources. Datasources are already initialized at execution and available at `ctx.datasources`. You can, for example, configure logging here or add contracts and indexes in runtime instead of from static config.
 
 ## `on_reindex`
 

docs/advanced/metadata-interface.md

@@ -16,7 +16,7 @@ Keeping aside the metadata schema, let's focus on which approaches for storing a
 
 The most straightforward approach is to store everything in the contract storage, especially if it's just the basic fields (name, symbol, decimals):
 
-```
+```text
 storage
 └── token_metadata [big_map]
     └── 0
@@ -30,7 +30,7 @@ storage
 But typically, you want to store more like a token thumbnail icon, and it is no longer feasible to keep such large data on-chain (because you pay gas for every byte stored).  
 Then you can put large files somewhere off-chain (e.g., IPFS) and store just links:
 
-```
+```text
 storage
 └── token_metadata [big_map]
     └── 0
@@ -43,7 +43,7 @@ storage
 This approach is still costly, but sometimes (in rare cases), you need to have access to the metadata from the contract (example: [Dogami](https://tzkt.io/KT1NVvPsNDChrLRH5K2cy6Sc9r1uuUwdiZQd/storage/115420)).  
 We can go further and put the entire token info structure to IPFS:
 
-```
+```text
 storage
 └── token_metadata [big_map]
     └── 0