Document internal env variables; update ctx reference and feature flags page (#597)

Author: droserasprout

docs/Makefile

@@ -22,6 +22,7 @@ build:
 lint: markdownlint orphans
 
 serve:
+	make clean build
 	mdbook serve -o
 
 wc:

docs/advanced/feature-flags.md

@@ -1,22 +1,27 @@
 # Feature flags
 
-Feature flags allow users to modify some system-wide tunables that affect the behavior of the whole framework. These options are either experimental or unsuitable for generic configurations.
-
-A good practice is to use set feature flags in environment-specific config files.
-
-```yaml
-advanced:
-  early_realtime: False
-  merge_subscriptions: False
-  postpone_jobs: False
-  metadata_interface: False
-  skip_version_check: False
-  crash_reporting: False
+Feature flags set in `advanced` config section allow users to modify parameters that affect the behavior of the whole framework. Choosing the right combination of flags for an indexer project can improve performance, reduce RAM consumption, or enable useful features.
+
+| flag                  | description                                                          |
+| --------------------- | -------------------------------------------------------------------- |
+| `crash_reporting`     | Enable sending crash reports to the Baking Bad team                  |
+| `early_realtime`      | Start collecting realtime messages while sync is in progress         |
+| `merge_subscriptions` | Subscribe to all operations/big map diffs during realtime indexing   |
+| `metadata_interface`  | Enable contract and token metadata interfaces                        |
+| `postpone_jobs`       | Do not start the job scheduler until all indexes are synchronized    |
+| `skip_version_check`  | Disable warning about running unstable or out-of-date DipDup version |
+
+## Crash reporting
+
+Enables sending crash reports to the Baking Bad team. This is **disabled by default**. You can inspect crash dumps saved as `/tmp/dipdup/crashdumps/XXXXXXX.json` before enabling this option.
+
+```admonish info title="See Also"
+* {{ #summary troubleshooting.md}}
 ```
 
 ## Early realtime
 
-By default, DipDup enters a sync state twice: before and after establishing a realtime connection. This flag allows starting collecting realtime messages while sync is in progress, right after indexes load.
+By default, DipDup enters a sync state twice: before and after establishing a realtime connection. This flag allows collecting realtime messages while the sync is in progress, right after indexes load.
 
 Let's consider two scenarios:
 
@@ -28,20 +33,34 @@ If you do not have strict RAM constraints, it's recommended to enable this flag.
 
 ## Merge subscriptions
 
-Subscribe to all operations/big map diffs during realtime indexing instead of separate channels. This flag helps to avoid the 10.000 subscription limit of TzKT and speed up processing. The downside is an increased RAM consumption during sync, especially if `early_realtimm` flag is enabled too.
+Subscribe to all operations/big map diffs during realtime indexing instead of separate channels. This flag helps to avoid the 10.000 subscription limit of TzKT and speed up processing. The downside is an increased RAM consumption during sync, especially if `early_realtime` flag is enabled too.
 
-## Postpone jobs
+## Metadata interface
 
-Do not start the job scheduler until all indexes are synchronized. If your jobs perform some calculations that make sense only after indexing is fully finished, this toggle can save you some IOPS.
+Without this flag calling `ctx.update_contract_metadata` and `ctx.update_token_metadata` methods will have no effect. Corresponding internal tables are created on reindexing in any way.
 
-## Metadata interface
+## Postpone jobs
 
-Without this flag calling `ctx.update_contract_metadata` and `ctx.update_token_metadata` will make no effect. Corresponding internal tables are created on reindexing in any way.
+Do not start the job scheduler until all indexes are synchronized. If your jobs perform some calculations that make sense only after the indexer has reached realtime, this toggle can save you some IOPS.
 
 ## Skip version check
 
 Disables warning about running unstable or out-of-date DipDup version.
 
-## Crash reporting
+## Internal environment variables
 
-Enables sending crash reports to the Baking Bad team. This is **disabled by default**. You can inspect crash dumps saved as `/tmp/dipdup/crashdumps/XXXXXXX.json` before enabling this option.
+DipDup uses multiple environment variables internally. They read once on process start and usually do not change during runtime. Some variables modify the framework's behavior, while others are informational.
+
+Please note that they are not currently a part of the public API and can be changed without notice.
+
+| env variable          | module path               | description                                                           |
+| --------------------- | ------------------------- | --------------------------------------------------------------------- |
+| `DIPDUP_CI`           | `dipdup.env.CI`           | Running in GitHub Actions                                             |
+| `DIPDUP_DOCKER`       | `dipdup.env.DOCKER`       | Running in Docker                                                     |
+| `DIPDUP_DOCKER_IMAGE` | `dipdup.env.DOCKER_IMAGE` | Base image used when building Docker image (default, slim or pytezos) |
+| `DIPDUP_NEXT`         | `dipdup.env.NEXT`         | Enable features thar require schema changes                           |
+| `DIPDUP_PACKAGE_PATH` | `dipdup.env.PACKAGE_PATH` | Path to the currently used package                                    |
+| `DIPDUP_REPLAY_PATH`  | `dipdup.env.REPLAY_PATH`  | Path to datasource replay files; used in tests                        |
+| `DIPDUP_TEST`         | `dipdup.env.TEST`         | Running in pytest                                                     |
+
+`DIPDUP_NEXT` flag will give you the picture of what's coming in the next major release, but enabling it on existing schema will trigger a reindexing.

docs/context-reference.rst

@@ -1,2 +1,19 @@
-.. automodule:: dipdup.context
-   :members: DipDupContext, HandlerContext, HookContext
+.. autoclass:: dipdup.context.DipDupContext
+.. autoclass:: dipdup.context.HandlerContext
+.. autoclass:: dipdup.context.HookContext
+
+.. automethod:: dipdup.context.DipDupContext.add_contract
+.. automethod:: dipdup.context.DipDupContext.add_index
+.. automethod:: dipdup.context.DipDupContext.execute_sql
+.. automethod:: dipdup.context.DipDupContext.execute_sql_query
+.. automethod:: dipdup.context.DipDupContext.fire_hook
+.. automethod:: dipdup.context.DipDupContext.get_coinbase_datasource
+.. automethod:: dipdup.context.DipDupContext.get_http_datasource
+.. automethod:: dipdup.context.DipDupContext.get_ipfs_datasource
+.. automethod:: dipdup.context.DipDupContext.get_metadata_datasource
+.. automethod:: dipdup.context.DipDupContext.get_tzkt_datasource
+.. automethod:: dipdup.context.DipDupContext.reindex
+.. automethod:: dipdup.context.DipDupContext.restart
+.. automethod:: dipdup.context.DipDupContext.update_contract_metadata
+.. automethod:: dipdup.context.DipDupContext.update_token_metadata
+.. automethod:: dipdup.context.HookContext.rollback

docs/getting-started/project-structure.md

@@ -96,3 +96,4 @@ The same rules apply to handler callbacks. Note that the `callback` field must b
 * {{ #summary advanced/hooks.md }}
 * {{ #summary advanced/sql.md }}
 * {{ #summary graphql/hasura.md }}
+```

docs/graphql/README.md

@@ -29,13 +29,11 @@ Hasura GraphQL engine subscriptions are **live queries**, i.e., a subscription w
 
 This feature is essential to avoid complex state management (merging query results and subscription feed). In most scenarios, live queries are what you need to sync the latest changes from the backend.
 
-> ⚠ **WARNING**
->
-> If the live query has a significant response size that does not fit into the limits, you need one of the following:
->
-> 1. Paginate with offset (which is not convenient)
-> 2. Use cursor-based pagination (e.g., by an increasing unique id).
-> 3. Narrow down request scope with filtering (e.g., by timestamp or level).
+If the live query has a significant response size that does not fit into the limits, you need one of the following:
+
+1. Paginate with offset (which is not convenient)
+2. Use cursor-based pagination (e.g., by an increasing unique id).
+3. Narrow down request scope with filtering (e.g., by timestamp or level).
 
 Ultimately you can get "subscriptions" on top of live quires by requesting all the items having ID greater than the maximum existing or all the items with a timestamp greater than now.