Skip to content

Conversation

@samwillis
Copy link
Contributor

@samwillis samwillis commented Dec 22, 2024

@KyleAMathews, some quick notes on this:

This is a work in progress draft of how we could do an in-memory "store", materialise to them, and allow then to be queryable with D2.

The fruit-processed.ts example has been updated to use it.

Store is a Map like store, but that emits change events.

You can materialise any keyed D2 pipeline to a Store with Store.materialize, this is a static method on the store rather than the S2 pipeline builder so that it is optional and not included if a user doesn't need it. (they may be materialising to anything)

A Store, or multiple, can be queried with the Store.queryAll static method. You pass a list of stores, and a callback, which receives a D2 pipeline builder.

Mutations to a store can be grouped together with store.transaction, the changes are only emitted at the end of the transaction.

A few (initial) things that I want to change:

  • I'm subclassing EventTarget, but this doesn't have nice typing of events.
  • It would be useful to be able to have a transaction that works across stores - probably a static method.
  • a store.query instance method on a store without having to use the queryAll static method
  • Tests of the query side, I have test of the materialise method.

This is really targeted at front end processing, rather than server side processing.

An Electric ShapeStream could easily be materialised to a Store.

@samwillis samwillis marked this pull request as ready for review December 22, 2024 19:43
@samwillis samwillis changed the title Materialize query pipeline to a store Materialize query pipeline to a store and query it Dec 23, 2024
Copy link
Contributor

@KyleAMathews KyleAMathews left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice!

I think now that the basics are falling into place, it'll be easier to evaluate DX, perf w/ some e2e examples & real-world use cases we can implement and ship. I.e. this PR seems fine but it's harder to evaluate w/o the full context of its use.

@samwillis samwillis force-pushed the samwillis/materialize-to-store branch from e827e0e to e5ecbf5 Compare December 27, 2024 10:57
@samwillis samwillis force-pushed the samwillis/materialize-to-store branch from f460ede to 9e6e86d Compare December 27, 2024 11:06
@samwillis samwillis merged commit 432e3ab into main Dec 27, 2024
1 check passed
@samwillis samwillis deleted the samwillis/materialize-to-store branch December 27, 2024 11:09
cursor bot pushed a commit to samwillis/d2ts that referenced this pull request Jul 13, 2025
* Materialize query pipeline to a store

* Refactor materialize to be static method on Store

* Draft of how a store (or multiple) could be queried qith D2

* Fix linting errors

* Change store to not use EventTarget

* Basic tests for the queryAll method

* Add a query method to store

* Store transactionAll static method
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants