Skip to content

Save events to indexeddb if user hasn't authenticated their gcal #1407

@tyler-dane

Description

@tyler-dane

Storage: IndexedDB structure for events & tasks

Overall goals

  • Allow the user to create events and view them on their calendar without requiring them to go through the OAuth flow to get their google credentials.
  • Let unauthenticated users create/view events and tasks without going through OAuth.
  • Persist these in IndexedDB in a way that:
    • Mirrors our existing event and task types as closely as possible.
    • Minimizes “local-only” special cases.
    • Is easy to migrate to provider-backed events/tasks once the user authenticates.
  • Once authenticated and migrated, do not keep a long‑lived event mirror in IndexedDB; that would complicate 2‑way sync.

Implementation

1. Database & object stores

Database name: compass-local
Initial version: 1

On upgradeneeded, we’ll create two object stores:

  • events – for pre-auth calendar events.
  • tasks – for local tasks (used both pre-auth and post-auth unless/until we add a backend for tasks).
const DB_NAME = "compass-local";
const DB_VERSION = 1;

Upgrade handler sketch:

function openCompassLocalDB(): Promise<IDBDatabase> {
  return new Promise((resolve, reject) => {
    const request = indexedDB.open(DB_NAME, DB_VERSION);

    request.onupgradeneeded = (event) => {
      const db = request.result;
      const oldVersion = event.oldVersion;

      // Version 0 → 1: create initial stores
      if (oldVersion < 1) {
        const events = db.createObjectStore("events", { keyPath: "_id" });
        events.createIndex("by_startDate", "startDate", { unique: false });
        events.createIndex("by_endDate", "endDate", { unique: false });
        events.createIndex("by_isSomeday", "isSomeday", { unique: false });
        events.createIndex("by_createdAt", "createdAt", { unique: false });

        const tasks = db.createObjectStore("tasks", { keyPath: "id" });
        tasks.createIndex("by_dateKey", "dateKey", { unique: false });
        tasks.createIndex("by_completed", "completed", { unique: false });
      }

      // Future versions: add conditional blocks here (see “Upgrades” section below).
    };

    request.onsuccess = () => resolve(request.result);
    request.onerror = () => reject(request.error);
  });
}

2. Events store: reuse existing schema, override ID only

We will not introduce a new LocalEvent type. Instead:

  • Use the existing event/grid-event types (e.g. Schema_Event / Schema_GridEvent) as the base.
  • For local events, we:
    • Generate a client-side _id (e.g. a UUID with local_evt_ prefix).
    • Optionally tag them with a source/origin flag if the existing schema already has such a field (e.g. "local" vs "provider").
  • Persist the same shape used by the UI/sagas into IndexedDB, so mapping in and out is trivial.

Store

  • Name: events
  • Key path: _id (string; same property we already use to identify events)
  • _id for local events must be unique and never collide with provider IDs; recommended: local_evt_${uuid}.

Record shape

Conceptually:

// Pseudocode – this is how we should *think* about it, not a new exported type:
type IndexedDBEvent = Schema_GridEvent & {
  // For local-only events, _id is a client-generated id.
  // For provider events, we do NOT store them here long term.
  _id: string;

  // Only if we already have some notion of source/origin in the existing schema.
  // If this doesn't exist, we SHOULD NOT add it solely for IndexedDB.
  source?: "local" | "provider";
};

Here’s an updated, concrete plan you can paste into #1407. I’ll structure it as an “Implementation Plan” section that incorporates all your feedback.

Because the automation couldn’t modify the issue, you’ll need to update the issue text manually.


Implementation Plan (Updated)

1. Recommended Technology: Dexie.js for IndexedDB

Instead of using raw IndexedDB, use Dexie.js as the abstraction layer:

  • Add Dexie as a dependency in packages/web:
    • yarn add dexie (and dexie-react-hooks if we want useLiveQuery later).
  • Create a small database module, e.g. packages/web/src/common/db/compass.db.ts:
import Dexie, { Table } from "dexie";
import { Schema_GridEvent } from "../../common/types/event.types";
import { Task } from "../../common/types/task.types";

export type ClientEvent = Schema_GridEvent & {
  /**
   * For local-only events, `_id` is a client-generated string.
   * For now, we only persist local events here, not provider-backed ones.
   */
  _id: string;
  /**
   * Whether this event has been successfully synced to the backend.
   * This will matter for future offline mode when authenticated.
   */
  synced?: boolean;
};

export class CompassDatabase extends Dexie {
  events!: Table<ClientEvent, string>;
  tasks!: Table<Task & { id: string; dateKey: string }, string>;

  constructor() {
    super("CompassLocalDB");

    // Version 1: initial schema
    this.version(1).stores({
      // Primary key and indexes
      // _id is the PK; we also index on startDate/endDate/isSomeday for range queries
      events: "_id, startDate, endDate, isSomeday, synced",

      // id is PK; dateKey is used heavily in Day view; status/order allow
      // richer querying and sorting later.
      tasks: "id, dateKey, status, order",
    });

    // Future versions: add .version(2)... etc with explicit migrations.
  }
}

export const compassDB = new CompassDatabase();

Key points:

  • We reuse Schema_GridEvent and Task types and just augment them with what’s necessary (_id + synced for events; id + dateKey if not already present on Task).
  • Dexie gives us:
    • Strong typing.

    • Clean schema versioning (this.version(n)).

    • Easy queries, e.g.:

      const events = await compassDB.events
        .where("startDate")
        .between(startISO, endISO, true, true)
        .toArray();

2. Architectural Pattern: Event/Task Service Layer

To avoid coupling UI/sagas directly to Axios or Dexie:

  1. Define service interfaces:

    // packages/web/src/common/services/events.service.ts
    import { Schema_GridEvent } from "../types/event.types";
    
    export interface EventService {
      getEvents(params: {
        startDate: string;
        endDate: string;
        someday?: boolean;
      }): Promise<Schema_GridEvent[]>;
    
      createEvent(event: Schema_GridEvent): Promise<Schema_GridEvent>;
    
      updateEvent(
        id: string,
        event: Partial<Schema_GridEvent>,
      ): Promise<Schema_GridEvent>;
    
      deleteEvent(id: string): Promise<void>;
    }
    // packages/web/src/common/services/tasks.service.ts
    import { Task } from "../types/task.types";
    
    export interface TaskService {
      getTasksByDate(dateKey: string): Promise<Task[]>;
      createTask(task: Task & { dateKey: string }): Promise<Task>;
      updateTask(id: string, update: Partial<Task>): Promise<Task>;
      deleteTask(id: string): Promise<void>;
    }
  2. Implement Local vs Remote services:

    • LocalEventService (Dexie-backed; used when user is unauthenticated).
    • RemoteEventService (existing EventApi/Axios; used when authenticated).
    • LocalTaskService (Dexie; used always for now).

    Example LocalEventService:

    // packages/web/src/common/services/local-event.service.ts
    import { compassDB, ClientEvent } from "../db/compass.db";
    import { EventService } from "./events.service";
    import { Schema_GridEvent } from "../types/event.types";
    import { v4 as uuid } from "uuid";
    
    export class LocalEventService implements EventService {
      async getEvents({
        startDate,
        endDate,
        someday,
      }: {
        startDate: string;
        endDate: string;
        someday?: boolean;
      }): Promise<Schema_GridEvent[]> {
        if (someday) {
          return compassDB.events
            .where("isSomeday")
            .equals(true)
            .toArray();
        }
    
        // basic range query by startDate; can be refined if we want true overlap
        return compassDB.events
          .where("startDate")
          .between(startDate, endDate, true, true)
          .toArray();
      }
    
      async createEvent(event: Schema_GridEvent): Promise<Schema_GridEvent> {
        const _id = event._id ?? `local_evt_${uuid()}`;
        const stored: ClientEvent = {
          ...event,
          _id,
          synced: false,
        };
        await compassDB.events.put(stored);
        return stored;
      }
    
      async updateEvent(
        id: string,
        patch: Partial<Schema_GridEvent>,
      ): Promise<Schema_GridEvent> {
        const existing = await compassDB.events.get(id);
        if (!existing) throw new Error("Local event not found");
        const updated: ClientEvent = {
          ...existing,
          ...patch,
          _id: id,
          synced: false,
        };
        await compassDB.events.put(updated);
        return updated;
      }
    
      async deleteEvent(id: string): Promise<void> {
        await compassDB.events.delete(id);
      }
    }
  3. Service selection (factory/hook):

    • Add a thin selector that uses auth state to choose the implementation:
    // packages/web/src/common/services/useEventService.ts
    import { useMemo } from "react";
    import { useAuthState } from "../auth/useAuthState"; // or equivalent
    import { EventService } from "./events.service";
    import { LocalEventService } from "./local-event.service";
    import { RemoteEventService } from "./remote-event.service";
    
    export function useEventService(): EventService {
      const { isAuthenticated } = useAuthState();
    
      return useMemo(
        () =>
          isAuthenticated
            ? new RemoteEventService()
            : new LocalEventService(),
        [isAuthenticated],
      );
    }
    • Sagas / ducks should be refactored to call the EventService interface instead of EventApi directly. This keeps them storage-agnostic.

3. Missing Migration: localStorage → IndexedDB for Tasks

Currently, tasks are stored under localStorage keys like compass.today.tasks.* (see storage.util.ts). When we move tasks to IndexedDB, we must migrate the legacy data:

Plan:

  1. On app startup (e.g. in a top-level App effect or a small initLocalStorageMigration module), run:

    • Scan localStorage keys for compass.today.tasks. prefix.
    • For each key:
      • Parse the stored JSON array of tasks.
      • Derive the dateKey from the key suffix (or re-use getDateKey logic if the stored key includes a date).
      • Insert those tasks into the Dexie tasks store.
    • After successful migration of a key, remove that key from localStorage.
  2. Make the migration idempotent:

    • Before migrating, check whether there are already tasks in Dexie for that dateKey; if yes, either:
      • Skip, or
      • Merge, with localStorage tasks winning or losing by a defined rule.
  3. Once we’re confident in the migration, remove any remaining code paths that read/write tasks from localStorage.


4. Migration Strategy: Local Events → Remote Events

We need a robust, repeatable migration triggered after a successful OAuth login.

Trigger:

  • Hook into the same place we already handle “login success,” e.g.:
    • Supertokens callback.
    • App.tsx useEffect that runs after auth state changes from unauthenticated → authenticated.

Process:

  1. Fetch all unsynced events from Dexie:

    const unsynced = await compassDB.events
      .where("synced")
      .equals(false)
      .toArray();
  2. Send them to the backend:

    • Preferred: add a bulk create endpoint (e.g. POST /events/bulk) that takes an array of event payloads. This reduces network overhead and latency.
    • Fallback: loop and call EventApi.create for each event.
  3. Handle partial failures:

    • For each event:
      • On success:
        • Either delete it from Dexie (compassDB.events.delete(localId)), or:
        • Mark it as synced: true and store the remote _id mapping if we decide we need that temporarily.
      • On failure:
        • Leave the record in Dexie with synced: false.
        • Optionally store an error field if we add that to the shared schema later.
  4. Cleanup strategy:

    • If all events were successfully migrated:
      • await compassDB.events.where("synced").equals(true).delete(); or just clear() the store (since we don’t use it for remote events).
    • If some fail:
      • Only delete the successfully synced ones.
      • Optionally surface a banner/toast like “Some events could not be synced; we’ll keep them locally and try again.”

This approach ensures we never delete data that the user hasn’t successfully migrated.


5. Schema Design Details

Events

  • Base: Schema_GridEvent (and/or Schema_Event), with _id as the primary key.
  • Additional fields we rely on in Dexie:
    • synced: boolean – indicates whether the event has been successfully written to the backend.
  • Dexie index fields:
    • _id (primary).
    • startDate, endDate, isSomeday, synced.

Tasks

  • Base: Task type from common/types/task.types.
  • Required fields for Dexie:
    • id: string – primary key. If Task doesn’t already include this, we should add it at the type level.
    • dateKey: string – matches getDateKey(currentDate) used by useTaskState.
  • Recommended additional field:
    • A real Date or ISO string field (if not already present) to enable future range queries (e.g. “all tasks in this week”).
  • Indexed fields:
    • id (primary).
    • dateKey.
    • status (if tasks have states like “open/completed/in-progress”).
    • order (for custom ordering on the day view).

6. Testing Strategy

Unit / Integration

  • Dexie database:

    • Confirm CompassDatabase creates events and tasks tables with the expected schema.
    • Validate query behavior:
      • Events by startDate, endDate, isSomeday.
      • Tasks by dateKey.
  • Service layer:

    • LocalEventService:
      • Creating events assigns a local_evt_* _id if absent.
      • getEvents respects the provided date range and someday flag.
    • RemoteEventService:
      • Simply a thin wrapper around EventApi (mock Axios).
  • Migration:

    • localStorage → Dexie:
      • Given a fake localStorage dataset, ensure entries are moved into Dexie and keys removed.
    • Dexie → remote:
      • With a mocked backend:
        • All-success case: all events removed from Dexie.
        • Partial-failure case: successfully synced events removed, failing ones remain.

E2E (Playwright)

Add a test roughly like:

  1. Start as an unauthenticated user.
  2. Navigate to Compass Day/Week view.
  3. Create a new event (verify it appears in the UI).
  4. Simulate login:
    • Mock the auth flow (e.g., using a fake provider or test env).
    • Ensure the app’s auth state transitions to authenticated.
  5. After login:
    • Verify that:
      • The newly logged-in view includes the event (now sourced from the backend).
      • No duplicate event appears.
    • Optionally assert (via a Playwright test helper or log inspection) that the event was posted to the backend during auth transition.

If you’d like, next step can be drafting a concrete checklist for #1407’s acceptance criteria (including “uses Dexie”, “implements EventService abstraction”, and “migrates localStorage tasks”), or we can start proposing actual file patch content (e.g., compass.db.ts, local-event.service.ts, useEventService.ts).

Constraints:

  • Do not introduce a separate LocalEvent type.
  • Do not add IndexedDB-only fields unless they’re already part of our shared model. If we need metadata like createdAt/updatedAt and they’re not currently present on Schema_Event, add them in a way that makes sense across the app, not just for IndexedDB.

Indexes

On the events store:

  • by_startDate – keyPath: "startDate" – non-unique (for day/week range queries).
  • by_endDate – keyPath: "endDate" – non-unique (for overlap checks).
  • by_isSomeday – keyPath: "isSomeday" – non-unique (for Someday view).
  • by_createdAt – keyPath: "createdAt" – non-unique (for cleanup/migration).

These keys should already exist on Schema_Event/Schema_GridEvent, or be added there if missing.

Usage pattern (unauthenticated)

  • When creating an event:

    1. Build a Schema_GridEvent (or equivalent) object as we do today.
    2. If _id is missing, generate it: _id = "local_evt_" + uuid().
    3. Optionally set source: "local" if that field exists.
    4. Write this object into events via IndexedDB.
    5. Do not call the backend.
  • When querying:

    • For day/week:
      • Use by_startDate / by_endDate indexes to fetch events overlapping the desired date range.
      • Return them directly as Schema_GridEvent array (they’re already in the right shape).
    • For Someday:
      • Use by_isSomeday index where isSomeday === true.

Usage pattern (authenticated)

  • Normal operations:
    • Do not use IndexedDB as a cache for provider events.
    • Use the existing backend APIs and sagas only.
  • Only special case: the one-time migration of local events after the first successful OAuth (see Migration section).

3. Tasks store

Tasks are managed in the Day view via useTaskState and Task/UndoOperation types in common/types/task.types. We’ll store tasks in IndexedDB in a shape aligned with Task + the dateKey we already use.

Store

  • Name: tasks
  • Key path: id (string; client-generated UUID)

Record shape

This should map directly to the existing Task type plus dateKey (which we already compute via getDateKey):

// Pseudocode – align with existing `Task` type:
type IndexedDBTask = Task & {
  id: string;         // primary key; must already be part of Task or added globally
  dateKey: string;    // matches getDateKey(currentDate), e.g. "2026-01-03"
  // any other Task fields stay as-is
};

If Task already includes id, title, completed, etc., reuse them as-is. Only add dateKey if it’s not already present and makes sense more broadly.

Indexes

On the tasks store:

  • by_dateKey – keyPath: "dateKey" – non-unique (fetch tasks per day).
  • by_completed – keyPath: "completed" – non-unique (if we ever filter completed vs active).

Usage pattern

  • On the Day view, we already compute dateKey = getDateKey(currentDate).
  • When saving:
    • Use put with { ...task, dateKey }.
  • When loading:
    • Query by_dateKey for the current dateKey and cast directly to Task[].

Tasks are local-only for now, so this pattern applies both before and after auth. If/when we add backend tasks, we can mirror the event migration behavior.


4. Auth-aware read/write behavior

Unauthenticated

  • Events

    • All create/update/delete operations:
      • Only touch the events store in IndexedDB.
      • Never call EventApi.*.
    • Calendar views (day/week/someday):
      • Read from events using the indexes above.
      • Treat results exactly like normal events in the UI (they are the same type, only _id and possibly source differ).
  • Tasks

    • Read/write exclusively via tasks store in IndexedDB.
    • Use dateKey for per-day grouping.

Authenticated

  • Events

    • Primary source of truth is the backend.
    • Use existing sagas and EventApi calls.
    • The only interaction with IndexedDB is:
      • Detect if there are any leftover local events created pre-auth.
      • Migrate them forward (see below), then clear them.
  • Tasks

    • If tasks remain local-only: continue using IndexedDB even when authenticated.
    • If we later add task APIs:
      • Follow the same pattern as events: one-time migration, then use backend only.

5. Migration & cleanup after OAuth

Once the user successfully completes the OAuth flow for Google Calendar, we should migrate any local events the user has created into “real” provider events.

Algorithm

  1. Open compass-local DB and get the events store.
  2. Read all events (or iterate with a cursor).
  3. For each event:
    • Convert the stored Schema_GridEvent into the payload for EventApi.create (or equivalent).
    • Call the backend to create the event.
  4. If creation succeeds:
    • Option A (simpler): immediately delete the local event from IndexedDB (by _id).
    • Option B (if we need to retain audit info): add a global “migration log” elsewhere (not recommended unless there is a clear product requirement).
  5. After all events have either:
    • Succeeded and been deleted, or
    • Failed and been recorded for retry,
      then:
    • If all succeeded: we can clear() the entire events store.
    • If some failed: leave only the failed ones and consider a retry UI or background retry job.

Important constraints

  • Do not keep successfully migrated events in IndexedDB after auth. That would give us two sources of truth and complicate sync.
  • After migration, all event CRUD should go through the backend only.

Tasks:

  • For now, tasks are local-only, so no migration is needed.
  • If/when we add remote tasks:
    • Reuse the same pattern with the tasks store and map Task → backend task payload.

6. Upgrade and versioning strategy

We need to plan for schema changes without breaking existing users or silently losing data.

Principles

  • Only do backwards-compatible changes in-place (adding new optional fields, adding new indexes).
  • For destructive changes (changing primary keys, removing fields), either:
    • Implement an in-place migration that copies data to a new store; or
    • Bump the version and explicitly deleteObjectStore + recreate it, but only if it’s acceptable to drop old local data.

Pattern for future upgrades

Example: adding a new boolean field isPinned to events and a new index for it, in DB version 2:

// When bumping DB_VERSION to 2
request.onupgradeneeded = (event) => {
  const db = request.result;
  const oldVersion = event.oldVersion;

  if (oldVersion < 1) {
    // initial creation (as above)
  }

  if (oldVersion < 2) {
    const events = event.target.transaction!.db
      .transaction("events", "readwrite")
      .objectStore("events");

    // Add new index if it doesn't exist
    if (!events.indexNames.contains("by_isPinned")) {
      events.createIndex("by_isPinned", "isPinned", { unique: false });
    }

    // Backfill is optional; if we need to normalize old records:
    const cursorReq = events.openCursor();
    cursorReq.onsuccess = () => {
      const cursor = cursorReq.result;
      if (!cursor) return;
      const value = cursor.value;
      if (value.isPinned === undefined) {
        value.isPinned = false;
        cursor.update(value);
      }
      cursor.continue();
    };
  }

  // Additional version blocks go here (oldVersion < 3, < 4, etc.)
};

Guidelines:

  • Each version bump has a small, self-contained block guarded by if (oldVersion < N).
  • Always check objectStoreNames / indexNames before creating to avoid runtime errors when upgrading from intermediate versions.
  • Avoid schema changes that require changing keyPaths (_id for events, id for tasks). If we ever need that, treat it as a destructive migration: read all old data, write to a new store, then delete the old store inside onupgradeneeded.

7. Testing expectations (to validate this spec)

  • Unit / integration

    • Opening the DB at version 1 creates:
      • events with keyPath _id and the listed indexes.
      • tasks with keyPath id and the listed indexes.
    • When unauthenticated:
      • Creating an event does not hit EventApi and writes to events only.
      • Day/week/someday views use IndexedDB data and render as normal.
      • Tasks in the Day view are persisted by dateKey and survive reloads.
    • When authenticated:
      • A migration flow is triggered after OAuth (or on next app start), which:
        • Reads pre-auth events from IndexedDB.
        • Calls the real create event API.
        • Deletes or clears local events on success.
      • Future event operations go exclusively through the backend.
  • E2E

    • Unauthenticated user can:
      • Open Compass, create events and tasks, refresh, and still see them.
    • After authenticating:
      • Previously local events reappear as real provider events via backend.
      • IndexedDB events store is empty (or holds only failed-to-migrate events, if we support retries).

  • Reuse existing event/task types; only _id and (optionally) a source flag distinguish local events.
  • Keep IndexedDB schemas versioned and upgrade-friendly.
  • Migrate once after OAuth, then let the backend be the only source of truth for events.

Use Case

This'll help new users onboard with less friction

It'll help contributors get a feel for the app without having to set up a GCP project

It'll help LLMs write and debug e2e tests

It'll help us support offline mode

Additional Context

Implementation Guidance

Storage

Act as an expert architect and design an elegant way to save our event data in indexeddb, so that the keys and values make sense

auth flow

If the user hasn't authenticated, do not attempt to make GET requests to the backend. Insteaad, simply read from the indexeddb

Once the user goes through the oauth flow, then convert their events from indexeddb into real events

  • Do not keep them in indexeddb after we have successfully authenticated. That will make it unnecessarily difficult to maintain 2-way sync with the calendar provider and our indexeddb storage

testing

Write e2e tests to confirm this works. The user shouldn't need to go through the oauth flow anymore, so they should just go to the app and then start adding tasks and events and see them on the page

Metadata

Metadata

Labels

webFrontend/web related issue

Projects

Status

In progress

Relationships

None yet

Development

No branches or pull requests

Issue actions