operation_unfiltered index release (#514)

Author: droserasprout

CHANGELOG.md

@@ -6,10 +6,15 @@ The format is based on [Keep a Changelog], and this project adheres to [Semantic
 
 ## [Unreleased]
 
+### Added
+
+- index: New index kind `operation_unfiltered`.
+
 ### Fixed
 
 - cli: Configure package logger in addition to `dipdup` one.
 - database: Fixed crash when using F expressions inside versioned transactions.
+- project: Fixed options in DipDup version question.
 - tzkt: Fixed deserializing `EventData` model.
 
 ## [6.2.0] - 2022-10-12

CONTRIBUTING.md

@@ -49,7 +49,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S
 
 - Crash reporting MUST be opt-in (disabled by default) both in config and project templates.
 - Sentry events and crash reports MUST NOT contain any sensitive information (IP addresses, hostnames, etc.)
-- DipDup SHOULD NOT perform network requests to APIs not defined in config as datasources.
+- DipDup SHOULD NOT perform network requests to APIs not defined in config as datasources. Current exceptions: GitHub.
 
 ## Docker images
 

docs/config/reference.md

@@ -470,12 +470,12 @@ http: HTTP client configuration</p>
 
 <dl class="py class">
 <dt class="sig sig-object py" id="dipdup.config.OperationUnfilteredIndexConfig">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.</span></span><span class="sig-name descname"><span class="pre">OperationUnfilteredIndexConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="pre">kind:</span> <span class="pre">~typing.Literal['operation_unfiltered'],</span> <span class="pre">datasource:</span> <span class="pre">~typing.Union[str,</span> <span class="pre">~dipdup.config.TzktDatasourceConfig],</span> <span class="pre">handlers:</span> <span class="pre">~typing.Tuple[~dipdup.config.OperationUnfilteredHandlerConfig,</span> <span class="pre">...],</span> <span class="pre">types:</span> <span class="pre">~typing.Tuple[~dipdup.enums.OperationType,</span> <span class="pre">...]</span> <span class="pre">=</span> <span class="pre">(&lt;OperationType.transaction:</span> <span class="pre">'transaction'&gt;,),</span> <span class="pre">first_level:</span> <span class="pre">int</span> <span class="pre">=</span> <span class="pre">0,</span> <span class="pre">last_level:</span> <span class="pre">int</span> <span class="pre">=</span> <span class="pre">0</span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.config.OperationUnfilteredIndexConfig" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.</span></span><span class="sig-name descname"><span class="pre">OperationUnfilteredIndexConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="pre">kind:</span> <span class="pre">~typing.Literal['operation_unfiltered'],</span> <span class="pre">datasource:</span> <span class="pre">~typing.Union[str,</span> <span class="pre">~dipdup.config.TzktDatasourceConfig],</span> <span class="pre">handlers:</span> <span class="pre">~typing.Tuple[~dipdup.config.OperationUnfilteredHandlerConfig,</span> <span class="pre">...],</span> <span class="pre">types:</span> <span class="pre">~typing.Tuple[~dipdup.enums.OperationType,</span> <span class="pre">...]</span> <span class="pre">=</span> <span class="pre">(&lt;OperationType.origination:</span> <span class="pre">'origination'&gt;,),</span> <span class="pre">first_level:</span> <span class="pre">int</span> <span class="pre">=</span> <span class="pre">0,</span> <span class="pre">last_level:</span> <span class="pre">int</span> <span class="pre">=</span> <span class="pre">0</span></em><span class="sig-paren">)</span><a class="headerlink" href="#dipdup.config.OperationUnfilteredIndexConfig" title="Permalink to this definition">¶</a></dt>
 <dd><p>Operation index config</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters<span class="colon">:</span></dt>
 <dd class="field-odd"><ul class="simple">
-<li><p><strong>kind</strong> – always <cite>operation</cite></p></li>
+<li><p><strong>kind</strong> – always <cite>operation_unfiltered</cite></p></li>
 <li><p><strong>handlers</strong> – List of indexer handlers</p></li>
 <li><p><strong>first_level</strong> – Level to start indexing from</p></li>
 <li><p><strong>last_level</strong> – Level to stop indexing at (DipDup will terminate at this level)</p></li>

docs/indexes/big_map.md

@@ -1,5 +1,7 @@
 # `big_map` index
 
+Big maps are lazy structures allowing to access and update only exact keys. Gas costs for these operations doesn't depend on the size of a big map, but you can't iterate over it's keys onchain.
+
 `big_map` index allows querying only updates of specific big maps. In some cases, it can drastically reduce the amount of data transferred and thus indexing speed compared to fetching all operations.
 
 ```yaml

docs/indexes/operation_unfiltered.md

@@ -0,0 +1 @@
+# `operation_unfiltered` index

src/dipdup/__init__.py

@@ -1,3 +1,7 @@
+"""Root DipDup package.
+
+Contains software versions and other metadata. Also applies some CLI environment hacks on import.
+"""
 import logging
 import sys
 import warnings
@@ -7,6 +11,10 @@
 __version__ = '6.2.0'
 __spec_version__ = '1.2'
 
+version = tuple(int(x) for x in __version__.split('.'))
+major_version = version[0]
+minor_version = f'{version[0]}.{version[1]}'
+
 spec_version_mapping = {
     '0.1': '<=0.4.3',
     '1.0': '>=1.0.0, <=1.1.2',

src/dipdup/config.py

@@ -949,15 +949,15 @@ def address_filter(self) -> Set[str]:
 class OperationUnfilteredIndexConfig(IndexConfig):
     """Operation index config
 
-    :param kind: always `operation`
+    :param kind: always `operation_unfiltered`
     :param handlers: List of indexer handlers
     :param first_level: Level to start indexing from
     :param last_level: Level to stop indexing at (DipDup will terminate at this level)
     """
 
     kind: Literal['operation_unfiltered']
     handlers: Tuple[OperationUnfilteredHandlerConfig, ...]
-    types: Tuple[OperationType, ...] = (OperationType.transaction,)
+    types: Tuple[OperationType, ...] = (OperationType.origination,)
 
     first_level: int = 0
     last_level: int = 0
@@ -1094,6 +1094,14 @@ class HeadIndexConfig(IndexConfig):
     datasource: Union[str, TzktDatasourceConfig]
     handlers: Tuple[HeadHandlerConfig, ...]
 
+    @property
+    def first_level(self) -> int:
+        return 0
+
+    @property
+    def last_level(self) -> int:
+        return 0
+
 
 @dataclass
 class TokenTransferHandlerConfig(HandlerConfig, kind='handler'):

src/dipdup/index.py

@@ -9,13 +9,15 @@
 from typing import DefaultDict
 from typing import Deque
 from typing import Dict
+from typing import Generic
 from typing import Iterable
 from typing import Iterator
 from typing import List
 from typing import Optional
 from typing import Sequence
 from typing import Set
 from typing import Tuple
+from typing import TypeVar
 from typing import Union
 from typing import cast
 
@@ -36,6 +38,7 @@
 from dipdup.config import OperationHandlerTransactionPatternConfig
 from dipdup.config import OperationIndexConfig
 from dipdup.config import OperationType
+from dipdup.config import OperationUnfilteredIndexConfig
 from dipdup.config import ResolvedIndexConfigT
 from dipdup.config import SkipHistory
 from dipdup.config import TokenTransferHandlerConfig
@@ -71,6 +74,8 @@
 
 _logger = logging.getLogger(__name__)
 
+ConfigT = TypeVar('ConfigT', bound=ResolvedIndexConfigT)
+
 
 @dataclass(frozen=True)
 class OperationSubgroup:
@@ -108,14 +113,10 @@ def extract_operation_subgroups(
     for _operation_index, operation in enumerate(operations):
         # NOTE: Filtering out operations that are not part of any index
         if operation.type == 'transaction':
-            if operation.entrypoint not in entrypoints and len(entrypoints) != 0:
+            if entrypoints and operation.entrypoint not in entrypoints:
                 filtered += 1
                 continue
-            if (
-                operation.sender_address not in addresses
-                and operation.target_address not in addresses
-                and len(addresses) != 0
-            ):
+            if addresses and operation.sender_address not in addresses and operation.target_address not in addresses:
                 filtered += 1
                 continue
 
@@ -146,7 +147,7 @@ def extract_operation_subgroups(
         )
 
 
-class Index:
+class Index(Generic[ConfigT]):
     """Base class for index implementations
 
     Provides common interface for managing index state and switching between sync and realtime modes.
@@ -155,7 +156,7 @@ class Index:
     message_type: MessageType
     _queue: Deque
 
-    def __init__(self, ctx: DipDupContext, config: ResolvedIndexConfigT, datasource: TzktDatasource) -> None:
+    def __init__(self, ctx: DipDupContext, config: ConfigT, datasource: TzktDatasource) -> None:
         self._ctx = ctx
         self._config = config
         self._datasource = datasource
@@ -298,9 +299,8 @@ def _extract_level(
         return batch_levels.pop()
 
 
-class OperationIndex(Index):
+class OperationIndex(Index[OperationIndexConfig]):
     message_type = MessageType.operation
-    _config: OperationIndexConfig
 
     def __init__(self, ctx: DipDupContext, config: OperationIndexConfig, datasource: TzktDatasource) -> None:
         super().__init__(ctx, config, datasource)
@@ -611,7 +611,6 @@ async def _get_contract_hashes(self, address: str) -> Tuple[int, int]:
 
 class BigMapIndex(Index):
     message_type = MessageType.big_map
-    _config: BigMapIndexConfig
 
     def __init__(self, ctx: DipDupContext, config: BigMapIndexConfig, datasource: TzktDatasource) -> None:
         super().__init__(ctx, config, datasource)
@@ -838,7 +837,6 @@ def _get_big_map_pairs(self) -> Set[Tuple[str, str]]:
 
 class HeadIndex(Index):
     message_type: MessageType = MessageType.head
-    _config: HeadIndexConfig
 
     def __init__(self, ctx: DipDupContext, config: HeadIndexConfig, datasource: TzktDatasource) -> None:
         super().__init__(ctx, config, datasource)
@@ -887,7 +885,6 @@ def push_head(self, head: HeadBlockData) -> None:
 
 class TokenTransferIndex(Index):
     message_type = MessageType.token_transfer
-    _config: TokenTransferIndexConfig
 
     def __init__(self, ctx: DipDupContext, config: TokenTransferIndexConfig, datasource: TzktDatasource) -> None:
         super().__init__(ctx, config, datasource)
@@ -1026,10 +1023,10 @@ def _get_token_ids(self) -> Set[int]:
 
 class OperationUnfilteredIndex(OperationIndex):
     message_type = MessageType.operation
-    _config: OperationIndexConfig
 
-    def __init__(self, ctx: DipDupContext, config: OperationIndexConfig, datasource: TzktDatasource) -> None:
-        super().__init__(ctx, config, datasource)
+    def __init__(self, ctx: DipDupContext, config: OperationUnfilteredIndexConfig, datasource: TzktDatasource) -> None:
+        # FIXME: Ugly inheritance hack
+        Index.__init__(self, ctx, config, datasource)  # type: ignore
         self._queue: Deque[Tuple[OperationSubgroup, ...]] = deque()
         self._contract_hashes: Dict[str, Tuple[int, int]] = {}
 
@@ -1071,7 +1068,6 @@ async def _match_operation_subgroup(self, operation_subgroup: OperationSubgroup)
 
 class EventIndex(Index):
     message_type = MessageType.event
-    _config: EventIndexConfig
 
     def __init__(self, ctx: DipDupContext, config: EventIndexConfig, datasource: TzktDatasource) -> None:
         super().__init__(ctx, config, datasource)

src/dipdup/project.py

@@ -9,7 +9,8 @@
 from pydantic import Field
 from tabulate import tabulate
 
-from dipdup import __version__
+from dipdup import major_version
+from dipdup import minor_version
 from dipdup.exceptions import ConfigurationError
 from dipdup.utils.codegen import load_template
 from dipdup.utils.codegen import write
@@ -271,12 +272,12 @@ class BaseProject(Project):
             description='Choose DipDup version',
             default=0,
             choices=(
-                '6',
-                '6.1',
+                major_version,
+                minor_version,
             ),
             comments=(
-                'Latest stable release',
-                'Latest release of 6.1 branch',
+                'latest stable',
+                'current stable',
             ),
         ),
         ChoiceQuestion(
@@ -304,12 +305,12 @@ class BaseProject(Project):
             choices=(
                 'hasura/graphql-engine:v2.14.0',
                 'hasura/graphql-engine:v2.14.0',
-                # 'hasura/graphql-engine:v2.15.0-beta.1',
+                'hasura/graphql-engine:v2.15.0-beta.1',
             ),
             comments=(
-                f'tested with DipDup {__version__}',
-                'latest',
-                # 'beta',
+                f'tested with DipDup {minor_version}',
+                'latest stable',
+                'latest beta',
             ),
         ),
         NotifyQuestion(