New Data Layer

[maxnowack]

As already discussed some time ago on discord, Here are my ideas and plans for the improved data handling for SignalDB. Keep in mind that this is a draft and not final. I'm open for feedback and suggestions. Feel free to add your comments and to challenge the design.

Currently all data is stored in memory as well as in the storage of the persistence adapters. This is not optimal, as the data is stored twice and the memory usage is quite high.
Another issue with this is, that all operation take place in the main thread as the memory is accessed from here.

My plan is to move to add a separate layer that handles the data storage, querying and write operations. This layer can be moved to a separate thread or web worker to not block the main thread. This data layer will "talk" to the connected collection in the mean threads via events and also update the data in the collection.

The data in the collection itself is just a cached representation of the data in the data layer. The collection "registeres" a query, once it's used in a reactive context. The query result then will be empty in the first place, but will trigger the data layer to fetch the data from the storage and update the collection with the result. The data will be then cached in the collection and the query will rerun and present the result. There will be also a way to get the current loading state of a query, so that the UI can show a loading indicator. If the reactive context is terminated, the query will be “unregistered” and the cache will be cleared (maybe with a configurable grace period).

The collection class will be trimmed down to a minimum. The most logic will be moved to the data layer. The collection only acts as a proxy to the data layer. It doesn’t store any data and also does not contain any complex logic. The API of the collection stays the same as current. But calling methods will then communicate with the data layer. So the collection act more as a reactivity proxy for the data layer.

The memoryAdapter of the collection will be ditched as it's functionality will be obsolete or merged into the data layer.

The data layer will be collection agnostic and can act as the backend for many collection instances.

Usage

The API how collections are instanciated will slightly change. It's also thinkable to develop the whole thing backwards compatible, but I'm unsure if it's worth it.
The data layer will be created with a factory function that takes the storage adapter as an argument. The storage adapter will be created with a factory function that takes the storage name as an argument. This way the storage adapter can be used for multiple collection instances.
The data layer will be passed to the collection constructor. The collection will then be connected to the data layer and will be able to communicate with it.

import { Collection, createDataLayer } from '@signaldb/core';
import createIndexedDBAdapter from '@signaldb/indexeddb'
import maverickjsReactivityAdapter from '@signaldb/maverickjs'

const dataLayer = createDataLayer({
  storage: name => createIndexedDBAdapter(name),
});

const collection = new Collection('posts', dataLayer, {
  reactivity: maverickjsReactivityAdapter,
});

Implementation

Data Layer

interface DataLayerOptions {
  // factory function to create a persistence adapter for a collection
  storage: (name: string) => PersistenceAdapter,
}

interface DataLayer {
  createCollectionInterface(name: string): CollectionInterface,
}

interface CollectionInterfaceEvents {
  // events that will be emitted when the loading state of a query changes.
  'query-state-change': { selector: Selector, options: QueryOptions, state: 'active' | 'complete' | 'error' },
}

interface CollectionInterface extends EventEmitter<CollectionInterfaceEvents> {
  // CRUD operations will be proxied from the collection to the collection interface of the data layer. The CRUD logic itself will be done inside of the data layer.
  insert(data: any): void,
  updateOne(selector: Selector, modifier: Modifier): void,
  updateMany(selector: Selector, modifier: Modifier): void,
  removeOne(selector: Selector): void,
  removeMany(selector: Selector): void,

  // methods for registering and unregistering queries that will be called from the collection during find/findOne
  registerQuery(selector: Selector, options: QueryOptions): void,
  unregisterQuery(selector: Selector, options: QueryOptions): void,
  getQueryState(selector: Selector, options: QueryOptions): 'active' | 'complete' | 'error',
  getQueryResult(selector: Selector, options: QueryOptions): null | any[],
}

export default function createDataLayer(options: DataLayerOptions): DataLayer

The main benefits of this design are a reduced memory footprint as well as that are more operations not blocking the UI thread anymore (if a web worker is used)

Open questions:

• Indices are currently initialized with the collection. With the new approach, the data layer should be responsible for the indices. How should the API look like create and manage indices?

Connected issues/discussions: #397, #998

[DankTechnologies]

This layer can be moved to a separate thread or web worker to not block the main thread

Dumb question, but I'm trying to square this limitation with the web worker compat notes for Svelte which imply collections work on the main thread and in web workers. What are the current limitations today in 1.x w.r.t. querying and modifying the collections off the main thread?

The collection "registeres" a query, once it's used in a reactive context.
If the reactive context is terminated, the query will be “unregistered” and the cache will be cleared (maybe with a configurable grace period)

Is the idea that the reactive context would be set up on a per component basis? With Svelte, I've moved to a model with the reactive contexts consolidated in one place that initializes at startup, using $effect.root and exporting functions that return the reactive variables. Components then import and use $derived to further query the reactive variables.

db.svelte.ts

let feeds = $state.raw<Feed[]>([]);
let episodes = $state.raw<Episode[]>([]);

$effect.root(() => {
	$effect(() => {
		const feedsCursor = db.feeds.find();
		feeds = feedsCursor.fetch();

		return () => {
			feedsCursor.cleanup();
		};
	});

	$effect(() => {
		const episodesCursor = db.episodes.find();
		episodes = episodesCursor.fetch();

		return () => {
			episodesCursor.cleanup();
		};
	});

function getFeeds() {
	return feeds;
}

function getEpisodes() {
	return episodes;
}

export { getFeeds, getEpisodes };

component

	import { getEpisodes, getFeeds } from '$lib/stores/db.svelte';

	let episodes = $derived(
		getEpisodes()
			.filter((episode) => episode.feedId === feedId)
			.sort((a, b) => b.publishedAt.getTime() - a.publishedAt.getTime())
			.slice(0, limit)
	);

	let feed = $derived(getFeeds().find((feed) => feed.id === feedId));

So far so good, even with ~12K rows in the episodes collection. Having this global state management is nice w.r.t. keeping components simple, but curious for your thoughts on this approach in general and in context of the data layer idea.

Thanks!

[maxnowack]

Dumb question, but I'm trying to square this limitation with the web worker compat notes for Svelte which imply collections work on the main thread and in web workers.

The idea is to keep the Collection with all reactivity and stuff in the main thread, while outsourcing the data handling to a web worker. Reactivity will stay as it is with no additional limitations compared to today.

What are the current limitations today in 1.x w.r.t. querying and modifying the collections off the main thread?

If you run SignalDB in a worker context today, it creates entirely new collection instances that communicate with the same storage backend. This behavior will remain the same with the redesign.
In the current version, all data must be kept in memory, so using a worker effectively doubles the memory usage. In the redesigned approach, collections won't store any data themselves. Instead, the data layer will handle caching, specifically for reactive contexts.
Since there is no reactive context outside the main thread (or at least there shouldn't be), the data layer in the worker can directly query the storage without needing to cache anything.

Is the idea that the reactive context would be set up on a per component basis? With Svelte, I've moved to a model with the reactive contexts consolidated in one place that initializes at startup, using $effect.root and exporting functions that return the reactive variables. Components then import and use $derived to further query the reactive variables.

It depends. The idea is to cache data only while a reactive context is active, and to remove the cached data once that context is terminated. In component terms, you could think of it as caching on mount and clearing the cache on unmount.
If you move your queries into a global reactive context, the queried data would remain cached indefinitely. This might be a deliberate and valid design choice – especially if your application consistently needs access to that data. However, for all other data, keeping it cached globally could introduce unnecessary overhead, particularly in terms of memory usage. In such cases, it is better to use dedicated reactive contexts at the component level. This is also my recommendation for the current version of SignalDB.

[jswhisperer]

Another silly question but very high level, couldn't the library mostly run in a web-worker with just the storage persistence layer elsewhere; especially the query engine?

Yes the data would kinda be duplicated but just this step would probably increase the performance 1000 fold (random guess)

I think it's a very cool goal to eventually just load the appropriate collections, but I wonder will it ever be possible to know for certain all the needed collections, as some may be dynamically requested or injected?

[maxnowack]

That's basically my plan with the new data layer 🙂

[jswhisperer]

Totally thought that's what you were saying! 🎉
Just was verifying it was a a iterative solution hopefully as it would be easier for you and the first step of workers the bigger win by far.
Keep on rocking!

[Mihailoff]

How are you going to translate mingo queries into storage adapter/data layer queries? I think this is a major issue, mingo is memory-only, so loading all the data first is the only solution. I guess the idea is to "offload" / remove from cache data that is not needed, and load back when needed? What I think could be an interim solution is to provide a "materialized view" query before you create a Collection to limit the number of items to operate with.

// instead of loading all upfront
const posts = new Collection({
  persistence: createIndexedDBAdapter('data')
})
const recentPosts = posts.find({ date: { $gte: startDate, $lte: endDate } }).fetch()

// provide a materialized view
const recentPosts = new Collection({
  persistence: createIndexedDBAdapter('data', {
    getAllItems: async (store) => {
      // Override getAllItems() with a narrow subset of records
      return await store.index('date').getAll(IDBKeyRange.bound(startDate, endDate))
    }
  })
})

[maxnowack]

This is definitely an issue. One idea to address it is to create a separate file (or equivalent storage unit, depending on the underlying storage system) for each index. Then, only the files containing matching indexes would be loaded into memory when executing a query.

In the new architecture, the data layer is responsible for query execution and memory management, rather than the collection itself. The collection's role is limited to reading from the cache and delegating query execution and data loading to the data layer. This separation of concerns allows us to optimize query execution and memory usage in the future, unlike the current setup where everything is tightly coupled within the collection.

Moreover, this design makes it possible to build custom data layers that doesn't use mingo for query execution at all, giving us more flexibility.

However, in the current unoptimized implementation – especially when using a collection with no additional indexes beyond the id index, running on the main thread (not in a Web Worker) – the collection loads all data into memory, runs the query using mingo, discards the irrelevant data, and then stores the result in the cache. This causes memory spikes during query execution, instead of maintaining a steady memory usage profile over time.

[Mihailoff]

Thank you, I think I understand it now. Pretty much how databases work: 1) keep indexes in memory 2) use cache and read from disk to populate data.

Maybe, there is a way to keep mingo then. Will try to describe the logic.

Query execution planner, from the list of fields used in query:

• no indexed fields --> load everything in cache memory (pretty much as now)
• has indexed field in query --> apply partial mingo query on index and get the list of record IDs (primary unique key) --> read records from cache, fetch from disk what is missing
  • if additional fields in query --> apply mingo query to the loaded dataset from previous stages

Basically, you need a query execution planner and a data layer to fulfill it. Let me give a few examples.

---

// "a" is indexed, "b" is not
const query = { a: { $gt: 5 }, b: { $eq: 'abc' } }

• Apply { a: { $gt: 5 } } query on "a" field index. Get the list of records IDs ids from index.
• Load a set of documents by respective ids. Some may be in cache, some to be fetched from disk and placed in cache.
• Apply full query on the interim set of documents { a: { $gt: 5 }, b: { $eq: 'abc' } }.
• Return resulting subset.

---

// "a" and "b" are indexed (separate indexes)
const query = { $or: [ { a: { $gt: 5 } }, { b: { $eq: 'abc' } } }

• Apply { a: { $gt: 5 } } query on "a" field index.
• Apply { b: { $eq: 'abc' } } query on "b" field index.
• Union two sets of ids

---

// "a" indexed, "b" not
const query = { $or: [ { a: { $gt: 5 } }, { b: { $eq: 'abc' } } }

• Full scan ... as with any database, if you want fast and efficient queries, you need to cover it with an index..

[thesmartwon]

I love this proposal!

It's also thinkable to develop the whole thing backwards compatible, but I'm unsure if it's worth it.

It's not.

Indices are currently initialized with the collection. With the new approach, the data layer should be responsible for the indices. How should the API look like create and manage indices?

Just put it on the CollectionInterface and have the data layer deal with it, no?

interface CollectionInterface {
  async createIndex(field: string): void;
  async deleteIndex(field: string): void; // to free up space
  async getIndexes(): string[]; // to know what to delete
}