Multi state api

[TheAndrewJackson]

Context / Motivation

LiveStore’s current state model is effectively “single state backend” (SQLite). Designing a truly generic multi-state API (e.g. SQLite + CRDT + JS-object state) likely requires broad refactors across schema representation, querying, migrations, rollback, syncing, and devtools.

This PR takes a pragmatic step first: a proof-of-concept that runs multiple independent SQLite databases side-by-side in a single store, in order to:

• explore what an eventual multi-backend API could look like,
• pressure-test routing decisions (event → backend, table → backend),
• surface “good vs. bad” API ergonomics early,
• identify where existing internals are too tightly coupled to “one SQLite DB”.

---

What this PR is

A focused exploration / scaffolding PR that introduces a “multi-state” shape internally and threads it end-to-end through:

• schema + state construction (multiple SQLite backends)
• leader thread (materialize/rollback/trim per backend; shared eventlog)
• store runtime (backend-aware query routing + reactive invalidation)
• adapter snapshot plumbing (export/import snapshot per backend)
• tests validating the core behavior

The POC example supported by these changes is:

“One LiveStore, two separate SQLite state DBs (‘a’ and ‘b’), with independent schemas/tables/materializers, and correct routing even when table names overlap.”

---

What this PR is not

This is not the final multi-state architecture, and it intentionally does not attempt to:

• provide a generic interface for non-SQLite backends (CRDTs, JS objects, etc.)
• fully decouple Store APIs from SQLite querying
• finalize public API ergonomics / naming
• ensure broad backwards compatibility / migrations across all historical shapes
• solve “multi-backend transaction semantics” beyond a simple rule (see below)

In other words: this PR is about learning and de-risking design, not production-hardening.

---

High-level API

Defining multiple backends (SQLite-only, for now)

const backendA = State.SQLite.makeBackend({
  id: 'a',
  tables: { items: itemsTableA },
  materializers: State.SQLite.materializers(eventsA, {
    'v1.AItemCreated': ({ id, title }) => itemsTableA.insert({ id, title }),
  }),
})

const backendB = State.SQLite.makeBackend({
  id: 'b',
  tables: { items: itemsTableB },
  materializers: State.SQLite.materializers(eventsB, {
    'v1.BItemCreated': ({ id, title }) => itemsTableB.insert({ id, title }),
  }),
})

const schema = makeSchema({
  state: State.SQLite.makeMultiState({ backends: [backendA, backendB] }),
  events: { ...eventsA, ...eventsB },
})

There is a new web-multi-state-todo example where you can seeing an implementation of this api.

Mental model

• schema.state.backends is now the canonical “all backends” container (Map).
• Each backend is still “SQLite flavored” today (tables, migrations, hash), but the shape is intentionally preparing for a world where there may be other backend kinds later.

A rough conceptual future (not implemented here):

schema.state.backends.get('sqlite') // ...
schema.state.backends.get('crdt')  // ...
schema.state.backends.get('xstate') // ...

Note: the current InternalStateBackend is still SQLite/table/migration-centric, and is expected to evolve.

flowchart TD
    store["Store"] --> schema["Schema"]
    schema --> state["State"]

    state --> events["Events"]
    state --> backends["Backends"]

    events --> a_events["A-events"]
    events --> b_events["B-events"]

    backends --> a_backend["A-backend"]
    backends --> b_backend["B-backend"]

    a_backend --> a_tables["tables"]
    a_backend --> a_materializers["materializers"]

    b_backend --> b_tables["tables"]
    b_backend --> b_materializers["materializers"]

Loading

Routing rules introduced by this POC

• Event routing: eventName → backendId via schema.state.materializersByEventName
• QueryBuilder routing: tableDef → backendId via backend tagging on table defs
• Raw SQL routing: can specify backendId explicitly (used by queryDb raw input)

Reactive invalidation / table keys

To avoid collisions when two backends have the same table name (e.g. both have items), reactive table refs are keyed as:

`${backendId}:${tableName}`

---

Implementation Walkthrough (what changed)

1) Schema/state representation is now “multi-backend”

Key additions include:

• schema.state.backends: Map<StateBackendId, InternalStateBackend>
• schema.state.defaultBackendId
• schema.state.materializersByEventName: Map<eventName, { backendId, materializer }>
• helper: resolveBackendIdForEventName(schema, eventName)

This is the backbone that lets the rest of the system route correctly.

Relevant files

• packages/@livestore/common/src/schema/schema.ts
• packages/@livestore/common/src/schema/state/sqlite/mod.ts

---

2) Backend-scoped system tables + backend tagging for QueryBuilder routing

To make QueryBuilder routing deterministic, tables are tagged with a backend id:

• setTableBackendId(tableDef, backendId)
• getTableBackendId(tableDef) (fail-fast if unassigned)

System tables are also now instantiated per backend (so metadata stays backend-local and tagging stays correct).

Relevant files

• packages/@livestore/common/src/schema/state/sqlite/table-def.ts
• packages/@livestore/common/src/schema/state/sqlite/system-tables/state-tables.ts
• packages/@livestore/common/src/schema/state/sqlite/system-tables/mod.ts

---

3) Leader thread sync/materialization supports multiple state DBs

The leader thread now:

• maintains dbStates: Map<backendId, db>
• materializes events in the correct backend db
• rolls back per backend during rebase
• trims changesets across all backend dbs
• computes materializer hashes using the correct backend db (important when schemas diverge)

Relevant files

• packages/@livestore/common/src/leader-thread/make-leader-thread-layer.ts
• packages/@livestore/common/src/leader-thread/LeaderSyncProcessor.ts

---

4) Store runtime becomes backend-aware

Changes include:

• per-backend SqliteDbWrapper map (sqliteDbWrappers)
• query execution routes based on QueryBuilder table backend id
• commit batches are rejected if they span multiple backends
  • rationale: keep “single commit = single SQLite transaction” in this POC
• reactive invalidation uses backendId:tableName keys

Relevant files

• packages/@livestore/livestore/src/store/store.ts
• packages/@livestore/livestore/src/store/table-key.ts
• packages/@livestore/livestore/src/live-queries/db-query.ts

---

5) Snapshot export/import is now per-backend

Adapters/workers now export:

• snapshotsByBackend: Array<[backendId, snapshotBytes]>

And boot paths validate snapshot completeness where needed.

Relevant files

• packages/@livestore/adapter-node/src/worker-schema.ts
• packages/@livestore/adapter-node/src/make-leader-worker.ts
• packages/@livestore/adapter-node/src/leader-thread-shared.ts
• packages/@livestore/adapter-web/src/web-worker/leader-worker/make-leader-worker.ts
• packages/@livestore/adapter-web/src/web-worker/client-session/snapshot-completeness.ts

---

Known limitations / open questions (intentional)

• Event names must be globally unique (enforced by current materializer map approach).
• Commit transactions are single-backend only (mixed-backend batches throw).
• State backend interface is still SQLite-centric (tables, migrations, hash).
• API ergonomics are not final (this is “internal-shape-first” exploration).

---

Reviewer Guide (what feedback is most useful)

• Does this feel like the correct approach or should a different direction be taken entirely?
• Does schema.state.backends + materializersByEventName feel like the right “central routing primitive”?
• Are the routing rules (event-name-based + table-def-tag-based) coherent and likely to scale?
• Are there obvious footguns in the current API surface (makeBackend, makeMultiState, backendId tagging)?
• Any “you’ll regret this later” architectural constraints worth addressing before iterating?