Tombstones generation and crash reporting (#435)

Author: droserasprout

CHANGELOG.md

@@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning].
 
 ## [Unreleased]
 
+### Added
+
+- config: Added `advanced.crash_reporting` flag to enable reporting crashes to Baking Bad.
+- dipdup: Save Sentry tombstone in `/tmp/dipdup-tombstone_XXXXXXX.json` on a crash.
+
 ### Fixed
 
 - config: Do not perform env variable substitution in commented out lines.

Dockerfile

@@ -10,7 +10,7 @@ RUN <<eot
     pip install --no-cache-dir poetry
     useradd -ms /bin/bash dipdup
     mkdir /home/dipdup/source
-    rm -r /var/lib/apt/lists/* /var/log/* /tmp/*
+    rm -r /var/lib/apt/lists/* /var/log/*
 eot
 
 COPY --chown=dipdup Makefile pyproject.toml poetry.lock README.md /home/dipdup/source/

docs/config/reference.md

@@ -1,7 +1,7 @@
 
   <span class="target" id="module-dipdup.config"></span><dl class="py class">
 <dt class="sig sig-object py" id="dipdup.config.AdvancedConfig">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.</span></span><span class="sig-name descname"><span class="pre">AdvancedConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">reindex:</span> <span class="pre">~typing.Dict[~dipdup.enums.ReindexingReason</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.enums.ReindexingAction]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scheduler:</span> <span class="pre">~typing.Optional[~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Any]]</span> <span class="pre">=</span> <span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">postpone_jobs:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">early_realtime:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">merge_subscriptions:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">metadata_interface:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">skip_version_check:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">rollback_depth:</span> <span class="pre">int</span> <span class="pre">=</span> <span class="pre">2</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.config.AdvancedConfig" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.</span></span><span class="sig-name descname"><span class="pre">AdvancedConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">reindex:</span> <span class="pre">~typing.Dict[~dipdup.enums.ReindexingReason</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.enums.ReindexingAction]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scheduler:</span> <span class="pre">~typing.Optional[~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Any]]</span> <span class="pre">=</span> <span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">postpone_jobs:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">early_realtime:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">merge_subscriptions:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">metadata_interface:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">skip_version_check:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">rollback_depth:</span> <span class="pre">int</span> <span class="pre">=</span> <span class="pre">2</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">crash_reporting:</span> <span class="pre">bool</span> <span class="pre">=</span> <span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.config.AdvancedConfig" title="Permalink to this definition">¶</a></dt>
 <dd><p>Feature flags and other advanced config.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -14,6 +14,7 @@
 <li><p><strong>metadata_interface</strong> – Expose metadata interface for TzKT</p></li>
 <li><p><strong>skip_version_check</strong> – Do not check for new DipDup versions on startup</p></li>
 <li><p><strong>rollback_depth</strong> – A number of levels to keep for rollback</p></li>
+<li><p><strong>crash_reporting</strong> – Enable crash reporting</p></li>
 </ul>
 </dd>
 </dl>
@@ -113,7 +114,7 @@
 
 <dl class="py class">
 <dt class="sig sig-object py" id="dipdup.config.DipDupConfig">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.</span></span><span class="sig-name descname"><span class="pre">DipDupConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">spec_version:</span> <span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">package:</span> <span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">datasources:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Union[~dipdup.config.TzktDatasourceConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.CoinbaseDatasourceConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.MetadataDatasourceConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.IpfsDatasourceConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.HttpDatasourceConfig]]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">database:</span> <span class="pre">~typing.Union[~dipdup.config.SqliteDatabaseConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.PostgresDatabaseConfig]</span> <span class="pre">=</span> <span class="pre">SqliteDatabaseConfig(kind='sqlite'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">path=':memory:')</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">contracts:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.ContractConfig]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">indexes:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Union[~dipdup.config.OperationIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.BigMapIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.HeadIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.TokenTransferIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.IndexTemplateConfig]]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">templates:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Union[~dipdup.config.OperationIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.BigMapIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.HeadIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.TokenTransferIndexConfig]]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">jobs:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.JobConfig]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">hooks:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.HookConfig]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">hasura:</span> <span class="pre">~typing.Optional[~dipdup.config.HasuraConfig]</span> <span class="pre">=</span> <span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sentry:</span> <span class="pre">~typing.Optional[~dipdup.config.SentryConfig]</span> <span class="pre">=</span> <span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">prometheus:</span> <span class="pre">~typing.Optional[~dipdup.config.PrometheusConfig]</span> <span class="pre">=</span> <span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">advanced:</span> <span class="pre">~dipdup.config.AdvancedConfig</span> <span class="pre">=</span> <span class="pre">AdvancedConfig(reindex={}</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scheduler=None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">postpone_jobs=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">early_realtime=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">merge_subscriptions=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">metadata_interface=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">skip_version_check=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">rollback_depth=2)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">custom:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Any]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logging:</span> <span class="pre">~dipdup.enums.LoggingValues</span> <span class="pre">=</span> <span class="pre">LoggingValues.default</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.config.DipDupConfig" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.</span></span><span class="sig-name descname"><span class="pre">DipDupConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">spec_version:</span> <span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">package:</span> <span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">datasources:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Union[~dipdup.config.TzktDatasourceConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.CoinbaseDatasourceConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.MetadataDatasourceConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.IpfsDatasourceConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.HttpDatasourceConfig]]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">database:</span> <span class="pre">~typing.Union[~dipdup.config.SqliteDatabaseConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.PostgresDatabaseConfig]</span> <span class="pre">=</span> <span class="pre">SqliteDatabaseConfig(kind='sqlite'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">path=':memory:')</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">contracts:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.ContractConfig]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">indexes:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Union[~dipdup.config.OperationIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.BigMapIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.HeadIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.TokenTransferIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.IndexTemplateConfig]]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">templates:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Union[~dipdup.config.OperationIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.BigMapIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.HeadIndexConfig</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.TokenTransferIndexConfig]]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">jobs:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.JobConfig]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">hooks:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~dipdup.config.HookConfig]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">hasura:</span> <span class="pre">~typing.Optional[~dipdup.config.HasuraConfig]</span> <span class="pre">=</span> <span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sentry:</span> <span class="pre">~typing.Optional[~dipdup.config.SentryConfig]</span> <span class="pre">=</span> <span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">prometheus:</span> <span class="pre">~typing.Optional[~dipdup.config.PrometheusConfig]</span> <span class="pre">=</span> <span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">advanced:</span> <span class="pre">~dipdup.config.AdvancedConfig</span> <span class="pre">=</span> <span class="pre">AdvancedConfig(reindex={}</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scheduler=None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">postpone_jobs=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">early_realtime=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">merge_subscriptions=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">metadata_interface=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">skip_version_check=False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">rollback_depth=2</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">crash_reporting=False)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">custom:</span> <span class="pre">~typing.Dict[str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">~typing.Any]</span> <span class="pre">=</span> <span class="pre">&lt;factory&gt;</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logging:</span> <span class="pre">~dipdup.enums.LoggingValues</span> <span class="pre">=</span> <span class="pre">LoggingValues.default</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.config.DipDupConfig" title="Permalink to this definition">¶</a></dt>
 <dd><p>Main indexer config</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters<span class="colon">:</span></dt>