5.2 docs update (#406)

droserasprout · web-flow · commit d8de9c2d6355 · 2022-06-28T10:58:59.000Z
* Enable mdbook-linkcheck, fix lots of links

* Magic link preprocessor!

* Fix links

* Universal depth parsing

* Lint

* Disable linkcheck

* 5.2 features

* Changelog
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -40,9 +40,7 @@ jobs:
           ${{ runner.os }}-venv-
 
     - name: Set up mdBook
-      uses: peaceiris/actions-mdbook@v1
-      with:
-        mdbook-version: '0.4.18'
+      uses: jontze/action-mdbook@v1
 
     - name: Install project
       run: make install
diff --git a/Makefile b/Makefile
@@ -31,7 +31,7 @@ test:           ## Run test suite
 
 docs:           ## Build docs
 	cd docs
-	make -s clean docs lint || true
+	make -s clean docs markdownlint orphans || true
 
 homepage:       ## Build homepage
 	cd docs
diff --git a/docs/Makefile b/docs/Makefile
@@ -37,9 +37,7 @@ serve:
 wc:
 	find . -name \*.md -type f -exec wc {} \;
 
-lint: markdown orphans links
-
-markdown:
+markdownlint:
 	markdownlint -f . --disable=MD013 --disable=MD033  || true
 
 clean:
@@ -49,9 +47,6 @@ orphans:
 	cat SUMMARY.md | grep ".md" > mentioned.tmp
 	sed -i -E 's@^.*\[(.*)\]\((.*)\).*$$@\2@p' mentioned.tmp
 	cat mentioned.tmp | uniq | sort -o mentioned.tmp
-	find . | grep ".md" | grep -v -e "node_modules/" -e " book/" -e "placeholders.md" -e "SUMMARY.md" | cut -c "3-" | sort -o existing.tmp
+	find . | grep ".md" | grep -v -e "node_modules/" -e " book/" -e "placeholders.md" -e "SUMMARY.md" -e "mdbook-summary" | cut -c "3-" | sort -o existing.tmp
 	diff --color mentioned.tmp existing.tmp
 	rm mentioned.tmp existing.tmp
-
-links:
-	python -m linkcheckmd -r . 
diff --git a/docs/README.md b/docs/README.md
@@ -1 +1 @@
-{{#include ../README.md}}
+{{ #include ../README.md }}
diff --git a/docs/advanced/datasources.md b/docs/advanced/datasources.md
@@ -123,4 +123,4 @@ Datasource HTTP connection parameters (ratelimit, backoff, etc.) are applied on
 
 > 🤓 **SEE ALSO**
 >
-> * [12.4. datasources](../config/datasources.md)
+> * {{ #summary config/datasources.md}}
diff --git a/docs/advanced/hooks/README.md b/docs/advanced/hooks/README.md
@@ -1,6 +1,6 @@
 # Hooks
 
-Hooks are user-defined callbacks called either from the [`ctx.fire_hook`](../advanced/handler-context.md#helpers) method or by scheduler ([`jobs`](jobs.md) config section, we'll return to this topic later).
+Hooks are user-defined callbacks called either from the [`ctx.fire_hook`](../context/reference.md) method or by scheduler ([`jobs`](jobs.md) config section, we'll return to this topic later).
 
 Let's assume we want to calculate some statistics on-demand to avoid blocking an indexer with heavy computations. Add the following lines to DipDup config:
 
@@ -46,4 +46,4 @@ By default, hooks execute SQL scripts from the corresponding subdirectory of `sq
 
 > 🤓 **SEE ALSO**
 >
-> * [12.6. hooks](../../config/hooks.md)
+> * {{ #summary config/hooks.md}}
diff --git a/docs/advanced/hooks/default.md b/docs/advanced/hooks/default.md
@@ -22,6 +22,6 @@ This hook fires when every active index reaches a realtime state. Here you can c
 
 > 🤓 **SEE ALSO**
 >
-> * [5.3. Reindexing](../../advanced/reindexing.md)
-> * [5.5. Executing SQL scripts](../sql.md)
-> * [Handler context](../advanced/handler-context.md)
+> * {{ #summary advanced/reindexing.md}}
+> * {{ #summary advanced/sql.md}}
+> * {{ #summary advanced/context/README.md}}
diff --git a/docs/advanced/index-factories.md b/docs/advanced/index-factories.md
@@ -3,12 +3,12 @@
 DipDup allows spawning new indexes from a template in runtime. There are two ways to do that:
 
 * From another index (e.g., handling factory originations)
-* In [`on_configure` hook](../cli/dipdup-run.md#custom-initialization)
+* In `on_configure` hook (see {{ #summary advanced/hooks/default.md}})
 
 > ⚠ **WARNING**
 >
-> DipDup is currently not able to automatically generate types and handlers for template indexes unless there is at least one [static instance](indexes/template.md).
+> DipDup is currently not able to automatically generate types and handlers for template indexes unless there is at least one [static instance](../config/indexes/template.md).
 
-DipDup exposes several context methods that extend the current configuration with new contracts and template instances. See [5.8. Handler context](../advanced/handler-context.md) for details.
+DipDup exposes several context methods that extend the current configuration with new contracts and template instances. See {{ #summary advanced/context/README.md}} for details.
 
-See [12.13. templates](../config/templates.md) for details.
+See {{ #summary config/templates.md}} for details.
diff --git a/docs/advanced/performance.md b/docs/advanced/performance.md
@@ -66,10 +66,10 @@ By default, DipDup retries failed requests infinitely, exponentially increasing
 >
 > * [Tortoise ORM documentation](https://tortoise-orm.readthedocs.io/en/latest/)
 > * [Tortoise ORM examples](https://tortoise-orm.readthedocs.io/en/latest/examples.html)
-> * [8.1. Database engines](../deployment/database-engines.md)
-> * [8.9. Backup and restore](../deployment/backups.md)
+> * {{ #summary deployment/database-engines.md}}
+> * {{ #summary deployment/backups.md}}
 
-See [12.4. datasources](../../config/datasources.md) for details.
+See [12.4. datasources](../config/datasources.md) for details.
 
 ## Use TimescaleDB for time-series
 
diff --git a/docs/book.toml b/docs/book.toml
@@ -4,3 +4,6 @@ language = "en"
 multilingual = false
 src = "."
 title = "DipDup framework"
+
+[preprocessor.summary]
+command = "python mdbook-summary"
diff --git a/docs/config/README.md b/docs/config/README.md
@@ -44,4 +44,4 @@ DipDup allows you to customize the configuration for a specific environment or a
 dipdup -c dipdup.yml -c dipdup.prod.yml run
 ```
 
-Run [`config export`](../cli/config-export.md) command if unsure about final config used by DipDup.
+Run [`config export`](../cli-reference.md#dipdup-config-export) command if unsure about final config used by DipDup.
diff --git a/docs/config/advanced.md b/docs/config/advanced.md
@@ -28,5 +28,5 @@ CLI flags have priority over self-titled `AdvancedConfig` fields.
 
 > 🤓 **SEE ALSO**
 >
-> * [5.3. Reindexing](../advanced/reindexing.md)
-> * [5.4. Feature flags](../advanced/feature-flags.md)
+> * {{ #summary advanced/reindexing.md}}
+> * {{ #summary advanced/feature-flags.md}}
diff --git a/docs/config/contracts.md b/docs/config/contracts.md
@@ -23,6 +23,6 @@ contracts:
 
 A `typename` field is only required when using index templates, but it helps to improve the readability of auto-generated code and avoid repetition.
 
-Contract entry does not contain information about the network, so it's a good idea to include the network name in the alias. This design choice makes possible a generic index parameterization via templates. See [4.5. Templates and variables](templates-and-variables) for details.
+Contract entry does not contain information about the network, so it's a good idea to include the network name in the alias. This design choice makes possible a generic index parameterization via templates. See [4.5. Templates and variables](../getting-started/templates-and-variables.md) for details.
 
-If multiple contracts you index have the same interface but different code, see [8.2. Reusing typename for different contracts](cookbook/reusing-typenames.md).
+If multiple contracts you index have the same interface but different code, see [8.2. Reusing typename for different contracts](../cookbook/reusing-typenames.md).
diff --git a/docs/config/database.md b/docs/config/database.md
@@ -55,7 +55,7 @@ You can also use compose-style environment variable substitutions with default v
 
 ### Immune tables
 
-In some cases, DipDup can't continue indexing with an existing database. See [5.3. Reindexing](../advanced/reindexing.md) for details. One of the solutions to resolve reindexing state is to drop the database and start indexing from scratch. To achieve this, either invoke the [`schema wipe` command](../cli/schema-wipe.md) or set an action to `wipe` in the [`advanced.reindex` config section](../config/advanced.md).
+In some cases, DipDup can't continue indexing with an existing database. See [5.3. Reindexing](../advanced/reindexing.md) for details. One of the solutions to resolve reindexing state is to drop the database and start indexing from scratch. To achieve this, either invoke the [`schema wipe` command](../cli-reference.md#dipdup-schema-wipe.md) or set an action to `wipe` in the [`advanced.reindex` config section](../config/advanced.md).
 
 You might want to keep several tables during schema wipe if data in them is not dependent on index states yet heavy. A typical example is indexing IPFS data — rollbacks do not affect off-chain storage, so you can safely continue after receiving a reorg message.
 
diff --git a/docs/config/datasources.md b/docs/config/datasources.md
@@ -53,4 +53,4 @@ datasources:
 
 > 🤓 **SEE ALSO**
 >
-> * [5.1. datasources](../advanced/datasources.md)
+> * {{ #summary advanced/datasources.md}}
diff --git a/docs/config/hasura.md b/docs/config/hasura.md
@@ -15,5 +15,5 @@ hasura:
 
 > 🤓 **SEE ALSO**
 >
-> * [6.1. Hasura integration](../graphql/hasura.md)
-> * [13.5. hasura configure](../cli/hasura-configure.md)
+> * {{ #summary graphql/hasura.md}}
+> * {{ #summary cli-reference.md#dipdup-hasura-configure}}
diff --git a/docs/config/hooks.md b/docs/config/hooks.md
@@ -14,4 +14,4 @@ hooks:
 
 > 🤓 **SEE ALSO**
 >
-> * [12.6. hooks](../advanced/hooks/README.md)
+> * {{ #summary advanced/hooks/README.md}}
diff --git a/docs/config/indexes/big_map.md b/docs/config/indexes/big_map.md
@@ -21,7 +21,7 @@ indexes:
 
 Each big\_map handler contains three required fields:
 
-* `callback` —  name of the _async_ function with a particular [**signature**](../../cli/init.md#handlers); DipDup will try to load it from the module with the same name `<package_name>.handlers.<callback>`
+* `callback` —  name of the _async_ function with a particular signature; DipDup will try to load it from the module with the same name `<package_name>.handlers.<callback>`
 * `contract` — Big map parent contract (from the [inventory](../contracts.md))
 * `path` — path to the Big map in the contract storage (use dot as a delimiter)
 
diff --git a/docs/config/indexes/operation.md b/docs/config/indexes/operation.md
@@ -44,7 +44,7 @@ Note that in the case of originations, DipDup will query operations where either
 
 Each operation handler contains two required fields:
 
-* `callback` —  name of the _async_ function with a particular [**signature**](../../cli/init.md#handlers); DipDup will try to load it from the module with the same name `<package_name>.handlers.<callback>`
+* `callback` —  name of the _async_ function with a particular signature; DipDup will try to load it from the module with the same name `<package_name>.handlers.<callback>`
 * `pattern` — a non-empty list of items that have to be matched
 
 ```yaml
diff --git a/docs/config/sentry.md b/docs/config/sentry.md
@@ -15,4 +15,4 @@ sentry:
 
 > 🤓 **SEE ALSO**
 >
-> * [8.5. Sentry integration](../deployment/sentry.md)
+> * {{ #summary deployment/sentry.md}}
diff --git a/docs/config/spec_version.md b/docs/config/spec_version.md
@@ -15,4 +15,4 @@ This table shows which specific SDK releases support which DipDup file versions.
 | 1.1 | >=2.0.0, <=2.0.9 |
 | 1.2 | >=3.0.0 |
 
-If you're getting `MigrationRequiredError` after updating the framework, run [`dipdup migrate`](../cli/migrate.md) command to perform project migration.
+If you're getting `MigrationRequiredError` after updating the framework, run [`dipdup migrate`](../cli-reference.md#migrate) command to perform project migration.
diff --git a/docs/deployment/logging.md b/docs/deployment/logging.md
@@ -1,52 +1,18 @@
 # Logging
 
-Currently, you have two options to configure logging:
+To control the number of logs DipDup produces, set the `logging` field in config:
 
-1. Manually in `on_restart` hook
+```yaml
+logging: default|verbose|quiet
+```
+
+If you need more fined tuning, perform it in the `on_restart` hook:
 
 ```python
 import logging
 
 async def on_restart(
     ctx: HookContext,
 ) -> None:
-    logging.getLogger('dipdup').setLevel('DEBUG')
-```
-
-2. With Python logging config
-
-> ⚠ **WARNING**
->
-> This feature will be deprecated soon. Consider configuring logging inside of `on_restart` hook.
-
-```shell
-dipdup -l logging.yml run
-```
-
-Example config:
-
-```yaml
-version: 1
-disable_existing_loggers: false
-formatters:
-  brief:
-    format: "%(levelname)-8s %(name)-20s %(message)s"
-handlers:
-  console:
-    level: INFO
-    formatter: brief
-    class: logging.StreamHandler
-    stream: ext://sys.stdout
-loggers:
-  dipdup:
-    level: INFO
-
-  aiosqlite:
-    level: INFO
-  db_client:
-    level: INFO
-root:
-  level: INFO
-  handlers:
-    - console
+    logging.getLogger('some_logger').setLevel('DEBUG')
 ```
diff --git a/docs/getting-started/defining-models.md b/docs/getting-started/defining-models.md
@@ -29,5 +29,5 @@ There are some limitations introduced to make Hasura GraphQL integration easier.
 >
 > * [Tortoise ORM documentation](https://tortoise-orm.readthedocs.io/en/latest/)
 > * [Tortoise ORM examples](https://tortoise-orm.readthedocs.io/en/latest/examples.html)
-> * [8.1. Database engines](../deployment/database-engines.md)
-> * [8.9. Backup and restore](../deployment/backups.md)
+> * {{ #summary deployment/database-engines.md }}
+> * {{ #summary deployment/backups.md }}
diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md
@@ -18,7 +18,7 @@ We [aim to improve](https://github.com/dipdup-net/dipdup-py/pull/358) cross-plat
 
 > 🤓 **SEE ALSO**
 >
-> * [5.6. Improving performance](advanced/performance/)
+> * {{ #summary advanced/performance.md }}
 > * [What is the Windows Subsystem for Linux?](https://docs.microsoft.com/en-us/windows/wsl/about)
 
 ## Local installation
@@ -48,6 +48,6 @@ pip install dipdup
 
 > 🤓 **SEE ALSO**
 >
-> * [8.2. Building Docker images](../deployment/docker.md)
-> * [8.3. Deploying with docker-compose](../deployment/docker-compose.md)
-> * [8.4. Deploying with Docker Swarm](../deployment/swarm.md)
+> * {{ #summary deployment/docker.md }}
+> * {{ #summary deployment/docker-compose.md }}
+> * {{ #summary deployment/swarm.md }}
diff --git a/docs/getting-started/project-structure.md b/docs/getting-started/project-structure.md
@@ -87,8 +87,8 @@ The same rules apply to handler callbacks. Note that the `callback` field must b
 
 > 🤓 **SEE ALSO**
 >
-> * [4.6. Defining models](defining-models.md)
-> * [4.7. Implementing handlers](implementing-handlers.md)
-> * [5.2. Hooks](../advanced/hooks/README.md)
-> * [5.5. Executing SQL scripts](../advanced/sql.md)
-> * [5.6.1. Hasura integration](../graphql/hasura.md)
+> * {{ #summary getting-started/defining-models.md }}
+> * {{ #summary getting-started/implementing-handlers.md }}
+> * {{ #summary advanced/hooks/README.md }}
+> * {{ #summary advanced/sql.md }}
+> * {{ #summary graphql/hasura.md }}
diff --git a/docs/getting-started/templates-and-variables.md b/docs/getting-started/templates-and-variables.md
@@ -16,7 +16,7 @@ templates:
             entrypoint: call
 ```
 
-Templates have the same syntax as [indexes](indexes/README.md) of all kinds; the only difference is that they additionally support placeholders enabling parameterization:
+Templates have the same syntax as [indexes](../config/indexes/README.md) of all kinds; the only difference is that they additionally support placeholders enabling parameterization:
 
 ```yaml
 field: <placeholder>
@@ -28,5 +28,5 @@ Any index implementing a template must have a value for each existing placeholde
 
 > 🤓 **SEE ALSO**
 >
-> * [5.9. Spawning indexes at runtime](../advanced/index-factories.md)
-> * [12.13. templates](../config/templates.md)
+> * {{ #summary advanced/index-factories.md }}
+> * {{ #summary config/templates.md }}
diff --git a/docs/graphql/hasura.md b/docs/graphql/hasura.md
@@ -1,6 +1,6 @@
 # Hasura integration
 
-This optional section used by DipDup executor to automatically configure Hasura engine to track your tables.
+DipDup uses this optional section to configure the Hasura engine to track your tables automatically.
 
 ```yaml
 hasura:
@@ -20,11 +20,11 @@ The current version of Hasura GraphQL Engine treats `public` and other schemas d
 
 ## Authentication
 
-DipDup sets READ only permissions for all tables and enables non-authorized access to the `/graphql` endpoint.
+DipDup sets read-only permissions for all tables and enables non-authorized access to the `/graphql` endpoint.
 
-## Limit number of rows
+## Limit the number of rows
 
-DipDup creates `user` role allowed to perform queries without authorization. Now you can limit the maximum number of rows such queries return and also disable aggregation queries automatically generated by Hasura:
+DipDup creates `user` role that allows performing queries without authorization. Now you can limit the maximum number of rows such queries return and also disable aggregation queries automatically generated by Hasura:
 
 ```yaml
 hasura:
@@ -42,7 +42,7 @@ hasura:
 
 ## Convert field names to camel case
 
-For those of you from JavaScript world, it may be more familiar to use _camelCase_ for variable names instead of _snake\_case_ Hasura uses by default. DipDup now allows to convert all fields in metadata to this casing:
+For those of you from the JavaScript world, it may be more familiar to use _camelCase_ for variable names instead of _snake\_case_ Hasura uses by default. DipDup now allows to convert all fields in metadata to this casing:
 
 ```yaml
 hasura:
@@ -77,5 +77,5 @@ Keep in mind that "camelcasing" is a separate stage performed after all tables a
 
 > 🤓 **SEE ALSO**
 >
-> * [6.1. Hasura integration](../../advanced/config/datasources.md)
-> * [13.5. hasura configure](../../cli/hasura-configure.md)
+> * {{ #summary config/hasura.md }}
+> * {{ #summary cli-reference.md#hasura-configure }}
diff --git a/docs/mdbook-summary b/docs/mdbook-summary
diff --git a/docs/quickstart.md b/docs/quickstart.md
diff --git a/docs/release-notes/5.0.0.md b/docs/release-notes/5.0.0.md
diff --git a/docs/release-notes/5.1.0.md b/docs/release-notes/5.1.0.md
diff --git a/docs/services/mempool.md b/docs/services/mempool.md
diff --git a/docs/services/metadata.md b/docs/services/metadata.md
diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml

Original file line number	Diff line number	Diff line change
`@@ -1 +1 @@`
`1`		`-{{#include ../README.md}}`
	`1`	`+{{ #include ../README.md }}`
Original file line number	Diff line number	Diff line change
`@@ -123,4 +123,4 @@ Datasource HTTP connection parameters (ratelimit, backoff, etc.) are applied on`
`123`	`123`
`124`	`124`	`> 🤓 SEE ALSO`
`125`	`125`	`>`
`126`		`-> * [12.4. datasources](../config/datasources.md)`
	`126`	`+> * {{ #summary config/datasources.md}}`
Original file line number	Diff line number	Diff line change
`@@ -1,6 +1,6 @@`
`1`	`1`	`# Hooks`
`2`	`2`
`3`		-Hooks are user-defined callbacks called either from the [`ctx.fire_hook`](../advanced/handler-context.md#helpers) method or by scheduler ([`jobs`](jobs.md) config section, we'll return to this topic later).
	`3`	+Hooks are user-defined callbacks called either from the [`ctx.fire_hook`](../context/reference.md) method or by scheduler ([`jobs`](jobs.md) config section, we'll return to this topic later).
`4`	`4`
`5`	`5`	`Let's assume we want to calculate some statistics on-demand to avoid blocking an indexer with heavy computations. Add the following lines to DipDup config:`
`6`	`6`
@@ -46,4 +46,4 @@ By default, hooks execute SQL scripts from the corresponding subdirectory of `sq
`46`	`46`
`47`	`47`	`> 🤓 SEE ALSO`
`48`	`48`	`>`
`49`		`-> * [12.6. hooks](../../config/hooks.md)`
	`49`	`+> * {{ #summary config/hooks.md}}`
Original file line number	Diff line number	Diff line change
`@@ -22,6 +22,6 @@ This hook fires when every active index reaches a realtime state. Here you can c`
`22`	`22`
`23`	`23`	`> 🤓 SEE ALSO`
`24`	`24`	`>`
`25`		`-> * [5.3. Reindexing](../../advanced/reindexing.md)`
`26`		`-> * [5.5. Executing SQL scripts](../sql.md)`
`27`		`-> * [Handler context](../advanced/handler-context.md)`
	`25`	`+> * {{ #summary advanced/reindexing.md}}`
	`26`	`+> * {{ #summary advanced/sql.md}}`
	`27`	`+> * {{ #summary advanced/context/README.md}}`