Integration with Electric sync via a electricStreamToD2Input method

.changeset/brave-berries-bathe.md

@@ -0,0 +1,5 @@
+---
+'@electric-sql/d2ts': patch
+---
+
+ElectricSQL intigration

README.md

@@ -139,9 +139,6 @@ graph.run()
 
 D2TS can be used in conjunction with [ElectricSQL](https://electric-sql.com) to build data pipelines on top of [ShapeStreams](https://electric-sql.com/docs/api/clients/typescript#shapestream) that can be executed incrementally.
 
-> [!NOTE]
-> Electric support has not yet been merged to main, you can follow the progress [in this PR](https://github.com/electric-sql/d2ts/pull/11).
-
 Here's an example of how to use D2TS with ElectricSQL:
 
 ```typescript
@@ -180,6 +177,8 @@ const electricStream = new ShapeStream({
 electricStreamToD2Input(electricStream, input)
 ```
 
+There is a complete example in the [./examples/electric](./examples/electric) directory.
+
 ## Examples
 
 There are a number of examples in the [./examples](./examples) directory, covering:
@@ -189,6 +188,7 @@ There are a number of examples in the [./examples](./examples) directory, coveri
 - [Joins between two streams](./examples/join.ts)
 - [Iterative computations](./examples/iterate.ts)
 - [Modeling "includes" using joins](./examples/includes.ts)
+- [ElectricSQL example](./examples/electric/) (using D2TS with ElectricSQL)
 
 ## API
 

examples/electric/README.md

@@ -0,0 +1,105 @@
+# ElectricSQL Example with D2TS
+
+This example demonstrates how to use [D2TS](https://github.com/electric-sql/d2ts) with [ElectricSQL](https://electric-sql.com) to build a real-time data pipeline that processes changes incrementally. The example implements an issue tracking system with users, issues, and comments.
+
+## Overview
+
+This example showcases:
+
+1. Setting up ElectricSQL with PostgreSQL
+2. Building an incremental data pipeline using D2TS
+3. Performing joins, aggregations, and transformations on real-time data
+4. Processing data changes efficiently with differential dataflow
+
+## Architecture
+
+The example consists of:
+
+- A PostgreSQL database paired with ElectricSQL for syncing out changes in real-time
+- Schema for users, issues, and comments
+- A D2TS pipeline that:
+  - Joins issues with their creators (users)
+  - Counts comments for each issue
+  - Consolidates the data into a unified view
+
+## Data Model
+
+The example implements a simple issue tracking system with:
+
+- **Users**: People who create issues and comments
+- **Issues**: Tasks or bugs with properties like priority and status
+- **Comments**: Text comments on issues
+
+## Prerequisites
+
+- Node.js and pnpm
+- Docker and Docker Compose (for running PostgreSQL and ElectricSQL)
+
+## Setup and Running
+
+1. **Start the backend services**
+
+   ```bash
+   pnpm backend:up
+   ```
+
+   This starts PostgreSQL and ElectricSQL services using Docker Compose.
+
+2. **Set up the database**
+
+   ```bash
+   pnpm db:migrate
+   ```
+
+   This applies the database migrations to create the necessary tables.
+
+3. **Load sample data**
+
+   ```bash
+   pnpm db:load-data
+   ```
+
+   This loads sample users, issues, and comments data into the database.
+
+4. **Run the example**
+
+   ```bash
+   pnpm start
+   ```
+
+   This starts the D2TS pipeline that consumes data from ElectricSQL and processes it incrementally. The pipeline will output the processed data to the console.
+
+5. **Reset everything (optional)**
+
+   ```bash
+   pnpm reset
+   ```
+
+   This command tears down the services, recreates them, applies migrations, and loads fresh data.
+
+## How It Works
+
+The pipeline in `src/index.ts` demonstrates:
+
+1. **Creating a D2TS graph** - The foundation for the data processing pipeline
+2. **Setting up inputs** - Connecting ElectricSQL shape streams to D2TS inputs
+3. **Building transformations**:
+   - Calculating comment counts per issue
+   - Joining issues with their creators
+   - Transforming and consolidating the data
+4. **Consuming the results** - Outputting processed data as it changes
+
+The ElectricSQL integration uses `MultiShapeStream` to consume multiple shapes (one per table) from the same Electric instance and the `electricStreamToD2Input` helper to connect Electric streams to D2TS inputs.
+
+## Key Concepts
+
+- **Differential Dataflow**: D2TS enables incremental computations, only reprocessing what's changed
+- **ElectricSQL ShapeStreams**: Real-time data streams that emit changes to the database
+- **LSN-based Processing**: Changes are processed based on PostgreSQL Log Sequence Numbers (LSNs) for consistency - these are used at the "version" of the data passed to D2TS
+
+## Resources
+
+- [D2TS Documentation](https://github.com/electric-sql/d2ts)
+- [ElectricSQL Documentation](https://electric-sql.com/docs)
+- [Differential Dataflow Paper](https://github.com/frankmcsherry/blog/blob/master/posts/2015-09-29.md)
+

examples/electric/db/generate_data.js

@@ -0,0 +1,62 @@
+import { faker } from '@faker-js/faker'
+import { v4 as uuidv4 } from 'uuid'
+
+export function generateUsers(numUsers) {
+  return Array.from({ length: numUsers }, generateUser)
+}
+
+function generateUser() {
+  return {
+    id: uuidv4(),
+    username: faker.internet.userName(),
+    email: faker.internet.email(),
+    full_name: faker.person.fullName()
+  }
+}
+
+export function generateIssues(numIssues, userIds) {
+  if (!userIds?.length) {
+    throw new Error('User IDs are required to generate issues')
+  }
+
+  return Array.from({ length: numIssues }, () => generateIssue(userIds))
+}
+
+function generateIssue(userIds) {
+  const issueId = uuidv4()
+  const createdAt = faker.date.past()
+  return {
+    id: issueId,
+    title: faker.lorem.sentence({ min: 3, max: 8 }),
+    description: faker.lorem.sentences({ min: 2, max: 6 }, `\n`),
+    priority: faker.helpers.arrayElement([`none`, `low`, `medium`, `high`]),
+    status: faker.helpers.arrayElement([
+      `backlog`,
+      `todo`,
+      `in_progress`,
+      `done`,
+      `canceled`,
+    ]),
+    created: createdAt.toISOString(),
+    modified: faker.date
+      .between({ from: createdAt, to: new Date() })
+      .toISOString(),
+    user_id: faker.helpers.arrayElement(userIds),
+    comments: faker.helpers.multiple(
+      () => generateComment(issueId, createdAt, userIds),
+      { count: faker.number.int({ min: 0, max: 10 }) }
+    ),
+  }
+}
+
+function generateComment(issueId, issueCreatedAt, userIds) {
+  return {
+    id: uuidv4(),
+    body: faker.lorem.text(),
+    user_id: faker.helpers.arrayElement(userIds),
+    issue_id: issueId,
+    created_at: faker.date
+      .between({ from: issueCreatedAt, to: new Date() })
+      .toISOString(),
+  }
+}

examples/electric/db/load_data.js

@@ -0,0 +1,69 @@
+import createPool, { sql } from '@databases/pg'
+import { generateUsers, generateIssues } from './generate_data.js'
+
+const DATABASE_URL = process.env.DATABASE_URL || 'postgresql://postgres:password@localhost:54321/electric'
+const ISSUES_TO_LOAD = process.env.ISSUES_TO_LOAD || 10
+const USERS_TO_LOAD = process.env.USERS_TO_LOAD || 3
+
+console.info(`Connecting to Postgres at ${DATABASE_URL}`)
+const db = createPool(DATABASE_URL)
+
+async function makeInsertQuery(db, table, data) {
+  const columns = Object.keys(data)
+  const columnsNames = columns.join(`, `)
+  const values = columns.map((column) => data[column])
+  return await db.query(sql`
+    INSERT INTO ${sql.ident(table)} (${sql(columnsNames)})
+    VALUES (${sql.join(values.map(sql.value), `, `)})
+  `)
+}
+
+async function importUser(db, user) {
+  return await makeInsertQuery(db, `user`, user)
+}
+
+async function importIssue(db, issue) {
+  const { comments: _, ...rest } = issue
+  return await makeInsertQuery(db, `issue`, rest)
+}
+
+async function importComment(db, comment) {
+  return await makeInsertQuery(db, `comment`, comment)
+}
+
+// Generate and load users first
+const users = generateUsers(USERS_TO_LOAD)
+const userIds = users.map(user => user.id)
+
+console.info(`Loading ${users.length} users...`)
+await db.tx(async (db) => {
+  for (const user of users) {
+    await importUser(db, user)
+  }
+})
+
+// Then generate and load issues with comments
+const issues = generateIssues(ISSUES_TO_LOAD, userIds)
+const issueCount = issues.length
+let commentCount = 0
+const batchSize = 100
+
+for (let i = 0; i < issueCount; i += batchSize) {
+  await db.tx(async (db) => {
+    db.query(sql`SET CONSTRAINTS ALL DEFERRED;`) // disable FK checks
+    for (let j = i; j < i + batchSize && j < issueCount; j++) {
+      process.stdout.write(`Loading issue ${j + 1} of ${issueCount}\r`)
+      const issue = issues[j]
+      await importIssue(db, issue)
+      for (const comment of issue.comments) {
+        commentCount++
+        await importComment(db, comment)
+      }
+    }
+  })
+}
+
+process.stdout.write(`\n`)
+
+await db.dispose()
+console.info(`Loaded ${users.length} users, ${issueCount} issues with ${commentCount} comments.`)

examples/electric/db/migrations/01-create_tables.sql

@@ -0,0 +1,31 @@
+CREATE TABLE IF NOT EXISTS "user" (
+    "id" UUID NOT NULL,
+    "username" TEXT NOT NULL,
+    "email" TEXT NOT NULL,
+    "full_name" TEXT NOT NULL,
+    CONSTRAINT "user_pkey" PRIMARY KEY ("id")
+);
+
+CREATE TABLE IF NOT EXISTS "issue" (
+    "id" UUID NOT NULL,
+    "title" TEXT NOT NULL,    
+    "description" TEXT NOT NULL,
+    "priority" TEXT NOT NULL,
+    "status" TEXT NOT NULL,
+    "modified" TIMESTAMPTZ NOT NULL,
+    "created" TIMESTAMPTZ NOT NULL,
+    "user_id" UUID NOT NULL,
+    CONSTRAINT "issue_pkey" PRIMARY KEY ("id"),
+    FOREIGN KEY (user_id) REFERENCES "user"(id) ON DELETE CASCADE DEFERRABLE
+);
+
+CREATE TABLE  IF NOT EXISTS "comment" (
+    "id" UUID NOT NULL,
+    "body" TEXT NOT NULL,
+    "user_id" UUID NOT NULL,
+    "issue_id" UUID NOT NULL,
+    "created_at" TIMESTAMPTZ NOT NULL,
+    CONSTRAINT "comment_pkey" PRIMARY KEY ("id"),
+    FOREIGN KEY (user_id) REFERENCES "user"(id) ON DELETE CASCADE DEFERRABLE,
+    FOREIGN KEY (issue_id) REFERENCES issue(id) ON DELETE CASCADE DEFERRABLE
+);

examples/electric/docker-compose.yml

@@ -0,0 +1,29 @@
+version: "3.3"
+name: "electric_d2_example"
+
+services:
+  postgres:
+    image: postgres:16-alpine
+    environment:
+      POSTGRES_DB: electric
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: password
+    ports:
+      - 54321:5432
+    tmpfs:
+      - /var/lib/postgresql/data
+      - /tmp
+    command:
+      - -c
+      - listen_addresses=*
+      - -c
+      - wal_level=logical
+
+  backend:
+    image: electricsql/electric:latest
+    environment:
+      DATABASE_URL: postgresql://postgres:password@postgres:5432/electric?sslmode=disable
+    ports:
+      - 3000:3000
+    depends_on:
+      - postgres

examples/electric/package.json

@@ -0,0 +1,34 @@
+{
+  "name": "@electric-sql/d2ts-electric-example",
+  "version": "0.0.0",
+  "private": true,
+  "description": "",
+  "main": "index.js",
+  "scripts": {
+    "backend:down": "docker compose down --volumes",
+    "backend:up": "docker compose up -d",
+    "db:load-data": "node ./db/load_data.js",
+    "db:migrate": "DATABASE_URL=postgresql://postgres:password@localhost:54321/electric pnpm exec pg-migrations apply --directory ./db/migrations",
+    "reset": "pnpm backend:down && pnpm backend:up && pnpm db:migrate && pnpm db:load-data",
+    "start": "tsx src/index.ts",
+    "format": "prettier --write \"src/**/*.{ts,tsx,js,jsx,json}\"",
+    "format:check": "prettier --check \"src/**/*.{ts,tsx,js,jsx,json}\""
+  },
+  "keywords": [],
+  "author": "",
+  "license": "Apache-2.0",
+  "devDependencies": {
+    "@databases/pg": "^5.5.0",
+    "@databases/pg-migrations": "^5.0.3",
+    "@faker-js/faker": "^8.4.1",
+    "prettier": "^3.2.5",
+    "tsx": "^4.7.0",
+    "typescript": "^5.0.0",
+    "uuid": "^9.0.0"
+  },
+  "dependencies": {
+    "@electric-sql/client": ">=1.0.0-beta.4",
+    "@electric-sql/experimental": ">=0.1.2-beta.3",
+    "@electric-sql/d2ts": "workspace:*"
+  }
+}