Skip to content

Conversation

@samwillis
Copy link
Contributor

@samwillis samwillis commented Dec 27, 2024

Used something like:

// Create D2 graph
const graph = new D2({ initialFrontier: 0 })

// Create D2 input
const input = graph.newInput<any>()

// Configure the pipeline
input
  .pipe(
    map(([key, data]) => data.value),
    filter(value => value > 10),
    // ... any other processing / joining
    output((msg) => doSomething(msg))
  )

// Finalize graph
graph.finalize()

// Create Electric stream (example)
const electricStream = new ShapeStream({
  url: 'http://localhost:3000/v1/shape',
  params: {
    table: 'items',
    replica: 'full',  // <-- IMPORTANT!
  }
})

// Connect Electric stream to D2 input
electricStreamToD2Input(electricStream, input)

Closes #7 #8

Depends on electric-sql/electric#2381 being released

@samwillis samwillis changed the title Draft of an electricStreamToD2Input method Integration with Electric sync via a electricStreamToD2Input method Feb 26, 2025
@samwillis samwillis linked an issue Feb 27, 2025 that may be closed by this pull request
@samwillis samwillis marked this pull request as ready for review March 3, 2025 18:10
@samwillis samwillis merged commit 7648e99 into main Mar 3, 2025
1 check passed
@samwillis samwillis deleted the samwillis/electric branch March 3, 2025 18:50
cursor bot pushed a commit to samwillis/d2ts that referenced this pull request Jul 13, 2025
* Draft of an electricStreamToD2Input method

* Add tests for the electric adapter

* WIP Example with Electric

* Fix tests

* Working electric adapter with MultiShapeStream

* rename checkForUpdatesAfter to checkForUpdatesAfterMs to match new MultiShapeStream

* SQLite version of the buffer operator

* Fix electric tests

* fully working example

* Update electric example to latest draft electric

* Move the electric example to the examples dir

* readme for the example

* add a outputElectricMessages operator that converts messeges in the D2 pipeline to electric change messages

* Remove import

* Update to use published client packages

* additional tests for electric

* changeset

* Fix linting errors

* use correct docker for electric

* Remove empty test script from example
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add D2 to Electric protocol adapter Add Electric to D2 pipeline adapter

3 participants