-
Notifications
You must be signed in to change notification settings - Fork 51
Description
Storage: IndexedDB structure for events & tasks
Overall goals
- Allow the user to create events and view them on their calendar without requiring them to go through the OAuth flow to get their google credentials.
- Let unauthenticated users create/view events and tasks without going through OAuth.
- Persist these in IndexedDB in a way that:
- Mirrors our existing event and task types as closely as possible.
- Minimizes “local-only” special cases.
- Is easy to migrate to provider-backed events/tasks once the user authenticates.
- Once authenticated and migrated, do not keep a long‑lived event mirror in IndexedDB; that would complicate 2‑way sync.
Implementation
1. Database & object stores
Database name: compass-local
Initial version: 1
On upgradeneeded, we’ll create two object stores:
events– for pre-auth calendar events.tasks– for local tasks (used both pre-auth and post-auth unless/until we add a backend for tasks).
const DB_NAME = "compass-local";
const DB_VERSION = 1;Upgrade handler sketch:
function openCompassLocalDB(): Promise<IDBDatabase> {
return new Promise((resolve, reject) => {
const request = indexedDB.open(DB_NAME, DB_VERSION);
request.onupgradeneeded = (event) => {
const db = request.result;
const oldVersion = event.oldVersion;
// Version 0 → 1: create initial stores
if (oldVersion < 1) {
const events = db.createObjectStore("events", { keyPath: "_id" });
events.createIndex("by_startDate", "startDate", { unique: false });
events.createIndex("by_endDate", "endDate", { unique: false });
events.createIndex("by_isSomeday", "isSomeday", { unique: false });
events.createIndex("by_createdAt", "createdAt", { unique: false });
const tasks = db.createObjectStore("tasks", { keyPath: "id" });
tasks.createIndex("by_dateKey", "dateKey", { unique: false });
tasks.createIndex("by_completed", "completed", { unique: false });
}
// Future versions: add conditional blocks here (see “Upgrades” section below).
};
request.onsuccess = () => resolve(request.result);
request.onerror = () => reject(request.error);
});
}2. Events store: reuse existing schema, override ID only
We will not introduce a new LocalEvent type. Instead:
- Use the existing event/grid-event types (e.g.
Schema_Event/Schema_GridEvent) as the base. - For local events, we:
- Generate a client-side
_id(e.g. a UUID withlocal_evt_prefix). - Optionally tag them with a
source/originflag if the existing schema already has such a field (e.g."local"vs"provider").
- Generate a client-side
- Persist the same shape used by the UI/sagas into IndexedDB, so mapping in and out is trivial.
Store
- Name:
events - Key path:
_id(string; same property we already use to identify events) _idfor local events must be unique and never collide with provider IDs; recommended:local_evt_${uuid}.
Record shape
Conceptually:
// Pseudocode – this is how we should *think* about it, not a new exported type:
type IndexedDBEvent = Schema_GridEvent & {
// For local-only events, _id is a client-generated id.
// For provider events, we do NOT store them here long term.
_id: string;
// Only if we already have some notion of source/origin in the existing schema.
// If this doesn't exist, we SHOULD NOT add it solely for IndexedDB.
source?: "local" | "provider";
};Here’s an updated, concrete plan you can paste into #1407. I’ll structure it as an “Implementation Plan” section that incorporates all your feedback.
Because the automation couldn’t modify the issue, you’ll need to update the issue text manually.
Implementation Plan (Updated)
1. Recommended Technology: Dexie.js for IndexedDB
Instead of using raw IndexedDB, use Dexie.js as the abstraction layer:
- Add Dexie as a dependency in
packages/web:yarn add dexie(anddexie-react-hooksif we wantuseLiveQuerylater).
- Create a small database module, e.g.
packages/web/src/common/db/compass.db.ts:
import Dexie, { Table } from "dexie";
import { Schema_GridEvent } from "../../common/types/event.types";
import { Task } from "../../common/types/task.types";
export type ClientEvent = Schema_GridEvent & {
/**
* For local-only events, `_id` is a client-generated string.
* For now, we only persist local events here, not provider-backed ones.
*/
_id: string;
/**
* Whether this event has been successfully synced to the backend.
* This will matter for future offline mode when authenticated.
*/
synced?: boolean;
};
export class CompassDatabase extends Dexie {
events!: Table<ClientEvent, string>;
tasks!: Table<Task & { id: string; dateKey: string }, string>;
constructor() {
super("CompassLocalDB");
// Version 1: initial schema
this.version(1).stores({
// Primary key and indexes
// _id is the PK; we also index on startDate/endDate/isSomeday for range queries
events: "_id, startDate, endDate, isSomeday, synced",
// id is PK; dateKey is used heavily in Day view; status/order allow
// richer querying and sorting later.
tasks: "id, dateKey, status, order",
});
// Future versions: add .version(2)... etc with explicit migrations.
}
}
export const compassDB = new CompassDatabase();Key points:
- We reuse
Schema_GridEventandTasktypes and just augment them with what’s necessary (_id+syncedfor events;id+dateKeyif not already present onTask). - Dexie gives us:
-
Strong typing.
-
Clean schema versioning (
this.version(n)). -
Easy queries, e.g.:
const events = await compassDB.events .where("startDate") .between(startISO, endISO, true, true) .toArray();
-
2. Architectural Pattern: Event/Task Service Layer
To avoid coupling UI/sagas directly to Axios or Dexie:
-
Define service interfaces:
// packages/web/src/common/services/events.service.ts import { Schema_GridEvent } from "../types/event.types"; export interface EventService { getEvents(params: { startDate: string; endDate: string; someday?: boolean; }): Promise<Schema_GridEvent[]>; createEvent(event: Schema_GridEvent): Promise<Schema_GridEvent>; updateEvent( id: string, event: Partial<Schema_GridEvent>, ): Promise<Schema_GridEvent>; deleteEvent(id: string): Promise<void>; }
// packages/web/src/common/services/tasks.service.ts import { Task } from "../types/task.types"; export interface TaskService { getTasksByDate(dateKey: string): Promise<Task[]>; createTask(task: Task & { dateKey: string }): Promise<Task>; updateTask(id: string, update: Partial<Task>): Promise<Task>; deleteTask(id: string): Promise<void>; }
-
Implement Local vs Remote services:
LocalEventService(Dexie-backed; used when user is unauthenticated).RemoteEventService(existingEventApi/Axios; used when authenticated).LocalTaskService(Dexie; used always for now).
Example LocalEventService:
// packages/web/src/common/services/local-event.service.ts import { compassDB, ClientEvent } from "../db/compass.db"; import { EventService } from "./events.service"; import { Schema_GridEvent } from "../types/event.types"; import { v4 as uuid } from "uuid"; export class LocalEventService implements EventService { async getEvents({ startDate, endDate, someday, }: { startDate: string; endDate: string; someday?: boolean; }): Promise<Schema_GridEvent[]> { if (someday) { return compassDB.events .where("isSomeday") .equals(true) .toArray(); } // basic range query by startDate; can be refined if we want true overlap return compassDB.events .where("startDate") .between(startDate, endDate, true, true) .toArray(); } async createEvent(event: Schema_GridEvent): Promise<Schema_GridEvent> { const _id = event._id ?? `local_evt_${uuid()}`; const stored: ClientEvent = { ...event, _id, synced: false, }; await compassDB.events.put(stored); return stored; } async updateEvent( id: string, patch: Partial<Schema_GridEvent>, ): Promise<Schema_GridEvent> { const existing = await compassDB.events.get(id); if (!existing) throw new Error("Local event not found"); const updated: ClientEvent = { ...existing, ...patch, _id: id, synced: false, }; await compassDB.events.put(updated); return updated; } async deleteEvent(id: string): Promise<void> { await compassDB.events.delete(id); } }
-
Service selection (factory/hook):
- Add a thin selector that uses auth state to choose the implementation:
// packages/web/src/common/services/useEventService.ts import { useMemo } from "react"; import { useAuthState } from "../auth/useAuthState"; // or equivalent import { EventService } from "./events.service"; import { LocalEventService } from "./local-event.service"; import { RemoteEventService } from "./remote-event.service"; export function useEventService(): EventService { const { isAuthenticated } = useAuthState(); return useMemo( () => isAuthenticated ? new RemoteEventService() : new LocalEventService(), [isAuthenticated], ); }
- Sagas / ducks should be refactored to call the
EventServiceinterface instead ofEventApidirectly. This keeps them storage-agnostic.
3. Missing Migration: localStorage → IndexedDB for Tasks
Currently, tasks are stored under localStorage keys like compass.today.tasks.* (see storage.util.ts). When we move tasks to IndexedDB, we must migrate the legacy data:
Plan:
-
On app startup (e.g. in a top-level
Appeffect or a smallinitLocalStorageMigrationmodule), run:- Scan
localStoragekeys forcompass.today.tasks.prefix. - For each key:
- Parse the stored JSON array of tasks.
- Derive the
dateKeyfrom the key suffix (or re-usegetDateKeylogic if the stored key includes a date). - Insert those tasks into the Dexie
tasksstore.
- After successful migration of a key, remove that key from
localStorage.
- Scan
-
Make the migration idempotent:
- Before migrating, check whether there are already tasks in Dexie for that
dateKey; if yes, either:- Skip, or
- Merge, with localStorage tasks winning or losing by a defined rule.
- Before migrating, check whether there are already tasks in Dexie for that
-
Once we’re confident in the migration, remove any remaining code paths that read/write tasks from
localStorage.
4. Migration Strategy: Local Events → Remote Events
We need a robust, repeatable migration triggered after a successful OAuth login.
Trigger:
- Hook into the same place we already handle “login success,” e.g.:
- Supertokens callback.
App.tsxuseEffectthat runs after auth state changes from unauthenticated → authenticated.
Process:
-
Fetch all unsynced events from Dexie:
const unsynced = await compassDB.events .where("synced") .equals(false) .toArray();
-
Send them to the backend:
- Preferred: add a bulk create endpoint (e.g.
POST /events/bulk) that takes an array of event payloads. This reduces network overhead and latency. - Fallback: loop and call
EventApi.createfor each event.
- Preferred: add a bulk create endpoint (e.g.
-
Handle partial failures:
- For each event:
- On success:
- Either delete it from Dexie (
compassDB.events.delete(localId)), or: - Mark it as
synced: trueand store the remote_idmapping if we decide we need that temporarily.
- Either delete it from Dexie (
- On failure:
- Leave the record in Dexie with
synced: false. - Optionally store an
errorfield if we add that to the shared schema later.
- Leave the record in Dexie with
- On success:
- For each event:
-
Cleanup strategy:
- If all events were successfully migrated:
await compassDB.events.where("synced").equals(true).delete();or justclear()the store (since we don’t use it for remote events).
- If some fail:
- Only delete the successfully synced ones.
- Optionally surface a banner/toast like “Some events could not be synced; we’ll keep them locally and try again.”
- If all events were successfully migrated:
This approach ensures we never delete data that the user hasn’t successfully migrated.
5. Schema Design Details
Events
- Base:
Schema_GridEvent(and/orSchema_Event), with_idas the primary key. - Additional fields we rely on in Dexie:
synced: boolean– indicates whether the event has been successfully written to the backend.
- Dexie index fields:
_id(primary).startDate,endDate,isSomeday,synced.
Tasks
- Base:
Tasktype fromcommon/types/task.types. - Required fields for Dexie:
id: string– primary key. IfTaskdoesn’t already include this, we should add it at the type level.dateKey: string– matchesgetDateKey(currentDate)used byuseTaskState.
- Recommended additional field:
- A real
Dateor ISO string field (if not already present) to enable future range queries (e.g. “all tasks in this week”).
- A real
- Indexed fields:
id(primary).dateKey.status(if tasks have states like “open/completed/in-progress”).order(for custom ordering on the day view).
6. Testing Strategy
Unit / Integration
-
Dexie database:
- Confirm
CompassDatabasecreateseventsandtaskstables with the expected schema. - Validate query behavior:
- Events by
startDate,endDate,isSomeday. - Tasks by
dateKey.
- Events by
- Confirm
-
Service layer:
LocalEventService:- Creating events assigns a
local_evt_*_idif absent. getEventsrespects the provided date range andsomedayflag.
- Creating events assigns a
RemoteEventService:- Simply a thin wrapper around
EventApi(mock Axios).
- Simply a thin wrapper around
-
Migration:
localStorage→ Dexie:- Given a fake
localStoragedataset, ensure entries are moved into Dexie and keys removed.
- Given a fake
- Dexie → remote:
- With a mocked backend:
- All-success case: all events removed from Dexie.
- Partial-failure case: successfully synced events removed, failing ones remain.
- With a mocked backend:
E2E (Playwright)
Add a test roughly like:
- Start as an unauthenticated user.
- Navigate to Compass Day/Week view.
- Create a new event (verify it appears in the UI).
- Simulate login:
- Mock the auth flow (e.g., using a fake provider or test env).
- Ensure the app’s auth state transitions to authenticated.
- After login:
- Verify that:
- The newly logged-in view includes the event (now sourced from the backend).
- No duplicate event appears.
- Optionally assert (via a Playwright test helper or log inspection) that the event was posted to the backend during auth transition.
- Verify that:
If you’d like, next step can be drafting a concrete checklist for #1407’s acceptance criteria (including “uses Dexie”, “implements EventService abstraction”, and “migrates localStorage tasks”), or we can start proposing actual file patch content (e.g., compass.db.ts, local-event.service.ts, useEventService.ts).
Constraints:
- Do not introduce a separate
LocalEventtype. - Do not add IndexedDB-only fields unless they’re already part of our shared model. If we need metadata like
createdAt/updatedAtand they’re not currently present onSchema_Event, add them in a way that makes sense across the app, not just for IndexedDB.
Indexes
On the events store:
by_startDate– keyPath:"startDate"– non-unique (for day/week range queries).by_endDate– keyPath:"endDate"– non-unique (for overlap checks).by_isSomeday– keyPath:"isSomeday"– non-unique (for Someday view).by_createdAt– keyPath:"createdAt"– non-unique (for cleanup/migration).
These keys should already exist on Schema_Event/Schema_GridEvent, or be added there if missing.
Usage pattern (unauthenticated)
-
When creating an event:
- Build a
Schema_GridEvent(or equivalent) object as we do today. - If
_idis missing, generate it:_id = "local_evt_" + uuid(). - Optionally set
source: "local"if that field exists. - Write this object into
eventsvia IndexedDB. - Do not call the backend.
- Build a
-
When querying:
- For day/week:
- Use
by_startDate/by_endDateindexes to fetch events overlapping the desired date range. - Return them directly as
Schema_GridEventarray (they’re already in the right shape).
- Use
- For Someday:
- Use
by_isSomedayindex whereisSomeday === true.
- Use
- For day/week:
Usage pattern (authenticated)
- Normal operations:
- Do not use IndexedDB as a cache for provider events.
- Use the existing backend APIs and sagas only.
- Only special case: the one-time migration of local events after the first successful OAuth (see Migration section).
3. Tasks store
Tasks are managed in the Day view via useTaskState and Task/UndoOperation types in common/types/task.types. We’ll store tasks in IndexedDB in a shape aligned with Task + the dateKey we already use.
Store
- Name:
tasks - Key path:
id(string; client-generated UUID)
Record shape
This should map directly to the existing Task type plus dateKey (which we already compute via getDateKey):
// Pseudocode – align with existing `Task` type:
type IndexedDBTask = Task & {
id: string; // primary key; must already be part of Task or added globally
dateKey: string; // matches getDateKey(currentDate), e.g. "2026-01-03"
// any other Task fields stay as-is
};If Task already includes id, title, completed, etc., reuse them as-is. Only add dateKey if it’s not already present and makes sense more broadly.
Indexes
On the tasks store:
by_dateKey– keyPath:"dateKey"– non-unique (fetch tasks per day).by_completed– keyPath:"completed"– non-unique (if we ever filter completed vs active).
Usage pattern
- On the Day view, we already compute
dateKey = getDateKey(currentDate). - When saving:
- Use
putwith{ ...task, dateKey }.
- Use
- When loading:
- Query
by_dateKeyfor the currentdateKeyand cast directly toTask[].
- Query
Tasks are local-only for now, so this pattern applies both before and after auth. If/when we add backend tasks, we can mirror the event migration behavior.
4. Auth-aware read/write behavior
Unauthenticated
-
Events
- All create/update/delete operations:
- Only touch the
eventsstore in IndexedDB. - Never call
EventApi.*.
- Only touch the
- Calendar views (day/week/someday):
- Read from
eventsusing the indexes above. - Treat results exactly like normal events in the UI (they are the same type, only
_idand possiblysourcediffer).
- Read from
- All create/update/delete operations:
-
Tasks
- Read/write exclusively via
tasksstore in IndexedDB. - Use
dateKeyfor per-day grouping.
- Read/write exclusively via
Authenticated
-
Events
- Primary source of truth is the backend.
- Use existing sagas and
EventApicalls. - The only interaction with IndexedDB is:
- Detect if there are any leftover local events created pre-auth.
- Migrate them forward (see below), then clear them.
-
Tasks
- If tasks remain local-only: continue using IndexedDB even when authenticated.
- If we later add task APIs:
- Follow the same pattern as events: one-time migration, then use backend only.
5. Migration & cleanup after OAuth
Once the user successfully completes the OAuth flow for Google Calendar, we should migrate any local events the user has created into “real” provider events.
Algorithm
- Open
compass-localDB and get theeventsstore. - Read all events (or iterate with a cursor).
- For each event:
- Convert the stored
Schema_GridEventinto the payload forEventApi.create(or equivalent). - Call the backend to create the event.
- Convert the stored
- If creation succeeds:
- Option A (simpler): immediately delete the local event from IndexedDB (by
_id). - Option B (if we need to retain audit info): add a global “migration log” elsewhere (not recommended unless there is a clear product requirement).
- Option A (simpler): immediately delete the local event from IndexedDB (by
- After all events have either:
- Succeeded and been deleted, or
- Failed and been recorded for retry,
then: - If all succeeded: we can
clear()the entireeventsstore. - If some failed: leave only the failed ones and consider a retry UI or background retry job.
Important constraints
- Do not keep successfully migrated events in IndexedDB after auth. That would give us two sources of truth and complicate sync.
- After migration, all event CRUD should go through the backend only.
Tasks:
- For now, tasks are local-only, so no migration is needed.
- If/when we add remote tasks:
- Reuse the same pattern with the
tasksstore and mapTask→ backend task payload.
- Reuse the same pattern with the
6. Upgrade and versioning strategy
We need to plan for schema changes without breaking existing users or silently losing data.
Principles
- Only do backwards-compatible changes in-place (adding new optional fields, adding new indexes).
- For destructive changes (changing primary keys, removing fields), either:
- Implement an in-place migration that copies data to a new store; or
- Bump the version and explicitly
deleteObjectStore+ recreate it, but only if it’s acceptable to drop old local data.
Pattern for future upgrades
Example: adding a new boolean field isPinned to events and a new index for it, in DB version 2:
// When bumping DB_VERSION to 2
request.onupgradeneeded = (event) => {
const db = request.result;
const oldVersion = event.oldVersion;
if (oldVersion < 1) {
// initial creation (as above)
}
if (oldVersion < 2) {
const events = event.target.transaction!.db
.transaction("events", "readwrite")
.objectStore("events");
// Add new index if it doesn't exist
if (!events.indexNames.contains("by_isPinned")) {
events.createIndex("by_isPinned", "isPinned", { unique: false });
}
// Backfill is optional; if we need to normalize old records:
const cursorReq = events.openCursor();
cursorReq.onsuccess = () => {
const cursor = cursorReq.result;
if (!cursor) return;
const value = cursor.value;
if (value.isPinned === undefined) {
value.isPinned = false;
cursor.update(value);
}
cursor.continue();
};
}
// Additional version blocks go here (oldVersion < 3, < 4, etc.)
};Guidelines:
- Each version bump has a small, self-contained block guarded by
if (oldVersion < N). - Always check
objectStoreNames/indexNamesbefore creating to avoid runtime errors when upgrading from intermediate versions. - Avoid schema changes that require changing keyPaths (
_idfor events,idfor tasks). If we ever need that, treat it as a destructive migration: read all old data, write to a new store, then delete the old store insideonupgradeneeded.
7. Testing expectations (to validate this spec)
-
Unit / integration
- Opening the DB at version 1 creates:
eventswith keyPath_idand the listed indexes.taskswith keyPathidand the listed indexes.
- When unauthenticated:
- Creating an event does not hit
EventApiand writes toeventsonly. - Day/week/someday views use IndexedDB data and render as normal.
- Tasks in the Day view are persisted by
dateKeyand survive reloads.
- Creating an event does not hit
- When authenticated:
- A migration flow is triggered after OAuth (or on next app start), which:
- Reads pre-auth events from IndexedDB.
- Calls the real create event API.
- Deletes or clears local events on success.
- Future event operations go exclusively through the backend.
- A migration flow is triggered after OAuth (or on next app start), which:
- Opening the DB at version 1 creates:
-
E2E
- Unauthenticated user can:
- Open Compass, create events and tasks, refresh, and still see them.
- After authenticating:
- Previously local events reappear as real provider events via backend.
- IndexedDB
eventsstore is empty (or holds only failed-to-migrate events, if we support retries).
- Unauthenticated user can:
- Reuse existing event/task types; only
_idand (optionally) a source flag distinguish local events. - Keep IndexedDB schemas versioned and upgrade-friendly.
- Migrate once after OAuth, then let the backend be the only source of truth for events.
Use Case
This'll help new users onboard with less friction
It'll help contributors get a feel for the app without having to set up a GCP project
It'll help LLMs write and debug e2e tests
It'll help us support offline mode
Additional Context
Implementation Guidance
Storage
Act as an expert architect and design an elegant way to save our event data in indexeddb, so that the keys and values make sense
auth flow
If the user hasn't authenticated, do not attempt to make GET requests to the backend. Insteaad, simply read from the indexeddb
Once the user goes through the oauth flow, then convert their events from indexeddb into real events
- Do not keep them in indexeddb after we have successfully authenticated. That will make it unnecessarily difficult to maintain 2-way sync with the calendar provider and our indexeddb storage
testing
Write e2e tests to confirm this works. The user shouldn't need to go through the oauth flow anymore, so they should just go to the app and then start adding tasks and events and see them on the page
Metadata
Metadata
Assignees
Labels
Type
Projects
Status