Custom Indexes #5373
Draft
Custom Indexes #5373
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The "writing to DuckDB" part and a small part of the management plane has been completed so far:
* A custom index can be created by writing an event with type: `$created` to a stream named: `$secondary-indexes-custom`:
Example:
```
{
"name": "myindex",
"filter": "e => e.type == 'my-event-type'",
"key-selector": "e => e.number",
"key-type": "int32"
}
```
* The following event properties are available to the javascript handlers: stream,number,type,id,isJson,data,metadata,rawData,rawMetadata (UNTESTED)
* The following partition key types are supported: string,int16,int32,int64,uint32,uint64,number (UNTESTED)
* A DuckDB table is created for each custom index: `idx_custom__{myindex}`
* A $all subscription is created for each custom index
* If an event received from $all passes the specified filter, the partition key is extracted and the event details are appended to DuckDB (but not flushed yet)
* After processing (but not necessarily indexing) every `options.CommitBatchSize` (50k) events, data is flushed to DuckDB and an explicit checkpoint for this custom index is written to the DuckDB table: `idx_custom_checkpoints`
* At startup, a subscription to `$secondary-indexes-custom` is created and existing custom indexes are loaded (resuming from their latest checkpoint)
* Events that fail processing are currently skipped and an error is logged (consider parking them)
Deploying eventstore with
|
| Latest commit: |
97ee15f
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://c8124893.eventstore.pages.dev |
| Branch Preview URL: | https://custom-indexes.eventstore.pages.dev |
* Support subscribing & reading custom indexes from the client The stream name format is: "$idx-<custom index>" e.g. "$idx-myindex" Optionally, a partition key can also be specified as follows: "$idx-myindex:<partition>" e.g. "$idx-myindex:0" * Support deletion of custom indexes through the management stream
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.