Skip to content

feat: transactions and rls rules#2426

Merged
geclos merged 13 commits intolatitude-v2from
spike/invert_platform_dependency
Mar 12, 2026
Merged

feat: transactions and rls rules#2426
geclos merged 13 commits intolatitude-v2from
spike/invert_platform_dependency

Conversation

@geclos
Copy link
Collaborator

@geclos geclos commented Mar 12, 2026

Summary

Implements Row-Level Security (RLS) policies and SQL transaction management across the entire stack, from database schema to API boundaries.

SQL Transactions

All sql operations are handled by the SqlClient service. SqlClient abstracts the underlying sql platform as well as the RLS enforcement.

Platforms provide SqlService implementations and app boundaries inject them in the use cases that need them.

  1. Domain Layer (@domain/shared): SqlClient interface—platform-agnostic contract
  2. Platform Layer (@platform/db-postgres): SqlClientLive implementation with RLS
  3. App Layer (apps/*): Provides SqlClientLive layer with organization context

Usage in Code

Repositories use sqlClient.query() for single operations:

findById: (id) =>
  sqlClient
    .query((db) => db.select().from(projects).where(eq(projects.id, id)))
    .pipe(Effect.flatMap(...))

.query checks if in transaction, if not it opens one and enforces RLS, if yes it's a pass-through proxy.

Use cases wrap multi-step operations in sqlClient.transaction():

export const completeAuthIntentUseCase = (input) =>
  Effect.gen(function* () {
    const sqlClient = yield* SqlClient
    yield* sqlClient.transaction(handleIntentByType(intent, input.session))
  })

.transaction checks if in transaction, if not it opens one with RLS enforced and updates the context so that any underlying effects pull from this transaction rather than the global sql client. If in transaction, it's a pass-through proxy.

Boundary routes provide the SqlClient layer with organization context:

// apps/api
const project = await Effect.runPromise(
  createProjectUseCase(input).pipe(
    Effect.provide(ProjectRepositoryLive),
    Effect.provide(SqlClientLive(c.var.postgresClient, c.var.organization.id)),
  )
)

// apps/web
const project = await Effect.runPromise(
  createProjectUseCase({...}).pipe(
    Effect.provide(ProjectRepositoryLive),
    Effect.provide(SqlClientLive(client, organizationId)),
  )
)

Boundaries provide the sql client implementation with proper context. Most of the times this means a postgres client + a tenant id (the ord id). For non-rls ops simply pass the admin postgres client. If you pass the regular postgres client without org id, rls will fail on any rls-protected underlying operation.

Key Behaviors

  • Every transaction automatically sets app.current_organization_id session variable
  • RLS policies filter all queries at the database level
  • No nested transactions (pass-through proxy)
  • Domain errors propagate through Effect error channel
  • Any failure triggers automatic rollback

Files Changed

  • packages/domain/shared/src/sql-client.ts — Domain SqlClient interface
  • packages/platform/db-postgres/src/sql-client.ts — RLS-enabled transaction implementation
  • packages/platform/db-postgres/src/schemaHelpers.ts — RLS policy helper
  • packages/platform/db-postgres/drizzle/* — Migration files for RLS function
  • apps/*/routes/*.ts — Updated routes to provide SqlClient layer
  • apps/web/src/domains/*/*.functions.ts — Server functions with RLS context
  • packages/**/repositories/*.ts — Repositories using sqlClient.query()
  • packages/**/use-cases/*.ts — Use cases using sqlClient.transaction()

@geclos geclos force-pushed the spike/invert_platform_dependency branch 2 times, most recently from e73ede8 to 0878464 Compare March 12, 2026 13:06
@geclos geclos force-pushed the spike/invert_platform_dependency branch 4 times, most recently from 82a508e to 08e8681 Compare March 12, 2026 14:06
@geclos
Copy link
Collaborator Author

geclos commented Mar 12, 2026

@cursoragent fix my comments

@cursor
Copy link

cursor bot commented Mar 12, 2026

All changes are pushed. Here's a summary of what was fixed for each PR comment:

  • apps/api/src/middleware/auth.ts: Restored validateApiKey to return an Effect instead of a Promise, eliminating the unnecessary Effect-to-Promise-to-Effect round-trip. authenticateWithApiKey now yields directly from it.

  • apps/api/src/routes/api-keys.test.ts (insert): Replaced raw SQL INSERT with Drizzle ORM database.db.insert(postgresSchema.apiKeys).values({...}).

  • apps/api/src/routes/api-keys.test.ts (select): Replaced raw SQL SELECT with Drizzle ORM database.db.select(...).from(...).where(...) using eq from drizzle-orm.

  • apps/api/src/test-utils/create-test-app.ts: Replaced all four raw SQL inserts (user, organization, member, api_keys) in createTenantSetup with Drizzle ORM inserts using postgresSchema.

  • apps/api/src/test-utils/in-memory-postgres.ts: Removed the unused withRls method and its OrganizationId import.

Open in Web Open in Cursor 

Implements Row-Level Security (RLS) policies and SQL transaction management.

Row-Level Security (RLS):
- Database function get_current_organization_id() reads app.current_organization_id from session
- RLS policies on tables with organization_id filter rows automatically
- Schema uses organizationRLSPolicy() helper to enable RLS per table

SQL Transactions:
- Domain Layer (@domain/shared): SqlClient interface for database operations
- Platform Layer (@platform/db-postgres): SqlClientLive with automatic RLS context
- App Layer (apps/*): Boundaries provide SqlClientLive with organization context

Usage patterns:
- Repositories use sqlClient.query() for single operations
- Use cases use sqlClient.transaction() for multi-step operations
- Routes provide SqlClientLive(client, organizationId) for RLS enforcement

Key behaviors:
- Every transaction sets app.current_organization_id session variable
- Nested transactions share connection (pass-through proxy)
- Domain errors propagate; failures trigger automatic rollback
@geclos geclos force-pushed the spike/invert_platform_dependency branch from 08e8681 to fee22ce Compare March 12, 2026 14:11
- Return Effect from validateApiKey instead of Promise (restores original pattern)
- Replace raw SQL queries with Drizzle ORM in api-keys.test.ts
- Replace raw SQL queries with Drizzle ORM in create-test-app.ts
- Remove unused withRls method from in-memory-postgres.ts

Co-authored-by: Gerard <gerard@latitude.so>
@geclos geclos marked this pull request as ready for review March 12, 2026 14:59
@geclos
Copy link
Collaborator Author

geclos commented Mar 12, 2026

@cursoragent review

@cursor
Copy link

cursor bot commented Mar 12, 2026

Now I have a complete picture. Let me compile the review.

Here is my review of the PR:


Overall

Strong architectural direction. The SqlClient abstraction cleanly separates transaction management and RLS enforcement from domain logic, and the ports-and-adapters wiring is consistent. The bulk of the PR is solid. Below are the findings worth addressing, ordered by severity.


Critical

1. subscription-repository.ts:97-100 -- exists() always returns true

sqlClient.query(db => db.select(...).limit(1)) returns an array. The .pipe(Effect.map(result => result !== undefined)) checks whether the array itself is undefined -- it never is, so this always returns true.

// Current (broken)
.pipe(Effect.map((result) => result !== undefined))

// Fix
.pipe(Effect.map((results) => results.length > 0))

2. auth.functions.ts:159-162 -- exchangeCliSession creates API key in wrong org

SqlClientLive(adminClient) defaults organizationId to "system". The generateApiKeyUseCase reads organizationId from SqlClient, so the API key is persisted with organization_id = "system" instead of the user's active organization. The Redis entry stores the correct activeOrganizationId, creating a mismatch.

// Current
Effect.provide(SqlClientLive(adminClient))

// Fix -- pass the user's org
Effect.provide(SqlClientLive(adminClient, OrganizationId(activeOrganizationId)))

3. complete-auth-intent.test.ts:314-335 -- Test swallows all errors

The "skips organization creation for signup when existingAccountAtRequest is true" test catches all exceptions and does not assert the expected outcome. Any failure (including mock misconfiguration) is silently swallowed. It should assert on observable state, e.g. that the auth intent was consumed and no organization was created.

4. projects.ts:280-293 / softDelete -- Cross-tenant DELETE returns 204 instead of 404

softDelete runs UPDATE ... SET deleted_at = ... WHERE id = $1. With RLS, the row is invisible to a different tenant, so 0 rows are updated and the effect succeeds silently. The route unconditionally returns 204. This is misleading -- the API spec declares a 404 response but it is never triggered for cross-tenant deletes.

Fix options: have softDelete check affected rows and fail with NotFoundError when 0, or call findById before soft-deleting.


Warnings

5. Multi-step use cases missing transaction()

Several use cases perform check-then-act patterns (read, validate, write) without wrapping in sqlClient.transaction(). Each sqlClient.query() opens its own transaction, so there is no atomicity between steps:

  • create-project.ts: existsByName + existsBySlug + save -- race allows duplicate slug
  • update-project.ts: findById + existsByName + save -- TOCTOU race on name uniqueness
  • revoke-api-key.ts: findById + save -- read and write not atomic
  • change-plan.ts: findActive + findBySubscription + revokeBySubscription + saveMany + save -- partial failure leaves inconsistent state

Recommendation: wrap multi-step DB flows in sqlClient.transaction().

6. members.functions.ts:42 -- findPendingInvitesByOrganizationId is not org-scoped

The auth_intent table has no RLS and the method does not filter by organization. listMembers can return invites belonging to other organizations. Either add org filtering in the repository query, or filter at the handler level.

7. projects.test.ts:22-26 -- Raw SQL still used for project seed data

createProjectRecord uses database.client.query with raw SQL. The review feedback on api-keys.test.ts was applied, but the same pattern remains in projects.test.ts. Should use Drizzle database.db.insert(postgresSchema.projects).values(...) for consistency, and the verification SELECT at line 97 should also use Drizzle.

8. sql-client.ts:29 -- Default OrganizationId("system") can hide mistakes

When no org ID is passed, SqlClientLive silently defaults to "system". If a developer forgets to pass the org, queries run against "system" rather than failing fast. Consider making organizationId required for tenant-scoped use, or using a clearly distinct sentinel that causes RLS to deny access.


Nits

9. auth-intent-repository.ts -- Redundant mapError(toRepositoryError(...))

sqlClient.query() already wraps errors in RepositoryError. The extra .pipe(Effect.mapError(...)) double-wraps without benefit. Can be removed for cleaner error chains.

10. membership-repository.ts:90 -- Unsafe as Effect.Effect<...> cast

The type assertion hides the real error type. Prefer letting TypeScript infer the types or narrowing explicitly.

11. Pattern inconsistency: Effect.provide vs Layer.merge + Layer.provide

API routes use Effect.provide(RepoLive).pipe(Effect.provide(SqlClientLive(...))), while web datasets.functions.ts uses Layer.merge(...).pipe(Layer.provide(SqlClientLive(...))). Both work but diverge in style. Standardizing would improve readability.

12. in-memory-postgres.ts:51-59 -- createRlsMiddleware does not verify role exists

If latitude_app is missing (e.g. migration order change), SET ROLE fails but PGlite may not surface it clearly, causing tests to silently run as owner and bypass RLS.


Summary

Severity Count Key items
Critical 4 exists() bug, wrong org on CLI key, test swallowing errors, silent cross-tenant delete
Warning 4 Missing transactions, unscoped invites query, raw SQL in test, default org hiding bugs
Nit 4 Redundant error wrapping, unsafe cast, style inconsistency, role check
Open in Web Open in Cursor 

cursoragent and others added 4 commits March 12, 2026 15:52
Critical fixes:
- Fix subscription-repository exists() always returning true (checked array not element)
- Fix exchangeCliSession creating API key with wrong org (pass activeOrganizationId)
- Fix complete-auth-intent test swallowing errors (assert expected behavior)
- Fix softDelete returning 204 for cross-tenant delete (now returns NotFoundError)
- Fix findPendingInvitesByOrganizationId to filter by org via JSONB query
- Preserve domain errors through transaction rollback (DomainErrorWrapper)

Cleanup:
- Replace raw SQL with Drizzle ORM in projects.test.ts
- Remove redundant mapError in auth-intent-repository
- Fix unsafe type cast in membership-repository (use Effect.map)
- Standardize datasets.functions.ts to Effect.provide style
- Fix seeds/run.ts passing undefined to SqlClientLive
- Add role existence check to createRlsMiddleware in tests

Co-authored-by: Gerard <gerard@latitude.so>
Rewrites SqlClientLive to use closure-scoped transaction tracking with
a promise-bridge pattern. The inner effect now runs in the parent fiber
(via Effect.exit) instead of Effect.runPromiseExit, so all provided
services (repositories, cache invalidators, etc.) remain available
inside the transaction scope.

Key changes:
- SqlClientLive tracks activeTx via closure variable shared by query()
  and transaction() methods on the same instance
- transaction() uses promise bridge: Drizzle callback signals tx ready,
  awaits effect completion, then commits or rolls back
- query() checks activeTx at call time, reusing the transaction
  connection when one is active
- Repositories captured at boundary still participate in transactions
  because query() reads activeTx dynamically, not at capture time

Multi-step use cases now properly wrapped in sqlClient.transaction():
- createProjectUseCase: existsByName + existsBySlug + save
- updateProjectUseCase: findById + existsByName + save
- revokeApiKeyUseCase: findById + save + cache invalidation
- changePlan: findActive + revokeBySubscription + saveMany + save

Co-authored-by: Gerard <gerard@latitude.so>
Introduces ChSqlClient as the ClickHouse counterpart to SqlClient:
- ChSqlClient domain interface in @domain/shared (pass-through
  transaction, direct query execution)
- ChSqlClientLive implementation in @platform/db-clickhouse
- DatasetRowRepositoryLive migrated from closure-captured client to
  ChSqlClient service pattern (Layer.effect + yield* ChSqlClient)

Adds withPostgres and withClickHouse helpers that bundle repository
layers with their database client in a single call:
- Uses Layer.provideMerge so the SqlClient/ChSqlClient service is
  available both to the repo layers AND to the outer effect (needed
  for use-case-level sqlClient.transaction() calls)
- Repos sharing the same helper call share the same client instance

Boundary callers updated from:
  Effect.provide(RepoLive),
  Effect.provide(SqlClientLive(client, orgId)),
To:
  Effect.provide(withPostgres(client, orgId, RepoLive)),

Co-authored-by: Gerard <gerard@latitude.so>
@geclos
Copy link
Collaborator Author

geclos commented Mar 12, 2026

@cursoragent lockfile is stale. Pnpm install and commit the updated lock file

@cursor
Copy link

cursor bot commented Mar 12, 2026

The lockfile is already in sync — pnpm install --frozen-lockfile succeeds. The CI failure was against an older commit (859f8b67) that predates the lockfile fix. The current HEAD (62319728d) already has the correct lockfile. CI should pass on the next run against the latest push.

Open in Web Open in Cursor 

geclos and others added 5 commits March 12, 2026 19:22
The implementation skips set_config for the system/admin org context
(added in 9256be6), but the test still expected 1 executed statement.
Update the assertion to expect 0 statements.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…x detection

- Add `createInMemoryPostgres` / `closeInMemoryPostgres` to testkit using
  PGlite so tests can run without a real Postgres instance
- Export from testkit index; add `@electric-sql/pglite` and `drizzle-orm`
  as deps to testkit
- Add `vitest.config.ts` to `apps/web` wiring up the vitest preset
- Harden `SqlClientLive`: track `txOpening` flag so concurrent `transaction()`
  calls on the same instance fail fast via `Effect.die` instead of silently
  corrupting connections; add tests for both the error and the sequential-OK path

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…viders

Previously withPostgres and withClickHouse returned a Layer, requiring
callers to wrap every call site with Effect.provide(). They also accepted
(client, orgId, ...layers) which was hard to type correctly with variadic
layer arguments.

New signature: (layer, client, orgId?) => pipe-compatible function via
Effect.provide internally. Call sites can now use effect.pipe(withPostgres(...))
directly without the outer Effect.provide wrapper.

- Rename withClickHouse to the same shape as withPostgres
- Migrate all call sites in apps/web and apps/api to the new signature
- Consolidate auth functions to withPostgres instead of multiple
  Effect.provide(RepositoryLive) + Effect.provide(SqlClientLive) chains
- Switch exchangeCliSession from adminClient to scoped postgresClient
  since API key creation is org-scoped

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@geclos geclos force-pushed the spike/invert_platform_dependency branch from 67e53d2 to 066db53 Compare March 12, 2026 22:13
@geclos geclos merged commit 39ac33b into latitude-v2 Mar 12, 2026
6 checks passed
@github-project-automation github-project-automation bot moved this from Inbox to Done in Latitude LLM Roadmap Mar 12, 2026
@geclos geclos deleted the spike/invert_platform_dependency branch March 12, 2026 22:16
andresgutgon pushed a commit that referenced this pull request Mar 13, 2026
* feat: transactions and rls rules

Implements Row-Level Security (RLS) policies and SQL transaction management.

Row-Level Security (RLS):
- Database function get_current_organization_id() reads app.current_organization_id from session
- RLS policies on tables with organization_id filter rows automatically
- Schema uses organizationRLSPolicy() helper to enable RLS per table

SQL Transactions:
- Domain Layer (@domain/shared): SqlClient interface for database operations
- Platform Layer (@platform/db-postgres): SqlClientLive with automatic RLS context
- App Layer (apps/*): Boundaries provide SqlClientLive with organization context

Usage patterns:
- Repositories use sqlClient.query() for single operations
- Use cases use sqlClient.transaction() for multi-step operations
- Routes provide SqlClientLive(client, organizationId) for RLS enforcement

Key behaviors:
- Every transaction sets app.current_organization_id session variable
- Nested transactions share connection (pass-through proxy)
- Domain errors propagate; failures trigger automatic rollback

* fix: address PR review comments

- Return Effect from validateApiKey instead of Promise (restores original pattern)
- Replace raw SQL queries with Drizzle ORM in api-keys.test.ts
- Replace raw SQL queries with Drizzle ORM in create-test-app.ts
- Remove unused withRls method from in-memory-postgres.ts

Co-authored-by: Gerard <gerard@latitude.so>

* fix: address review findings across the stack

Critical fixes:
- Fix subscription-repository exists() always returning true (checked array not element)
- Fix exchangeCliSession creating API key with wrong org (pass activeOrganizationId)
- Fix complete-auth-intent test swallowing errors (assert expected behavior)
- Fix softDelete returning 204 for cross-tenant delete (now returns NotFoundError)
- Fix findPendingInvitesByOrganizationId to filter by org via JSONB query
- Preserve domain errors through transaction rollback (DomainErrorWrapper)

Cleanup:
- Replace raw SQL with Drizzle ORM in projects.test.ts
- Remove redundant mapError in auth-intent-repository
- Fix unsafe type cast in membership-repository (use Effect.map)
- Standardize datasets.functions.ts to Effect.provide style
- Fix seeds/run.ts passing undefined to SqlClientLive
- Add role existence check to createRlsMiddleware in tests

Co-authored-by: Gerard <gerard@latitude.so>

* fix: enable atomic transactions in multi-step use cases

Rewrites SqlClientLive to use closure-scoped transaction tracking with
a promise-bridge pattern. The inner effect now runs in the parent fiber
(via Effect.exit) instead of Effect.runPromiseExit, so all provided
services (repositories, cache invalidators, etc.) remain available
inside the transaction scope.

Key changes:
- SqlClientLive tracks activeTx via closure variable shared by query()
  and transaction() methods on the same instance
- transaction() uses promise bridge: Drizzle callback signals tx ready,
  awaits effect completion, then commits or rolls back
- query() checks activeTx at call time, reusing the transaction
  connection when one is active
- Repositories captured at boundary still participate in transactions
  because query() reads activeTx dynamically, not at capture time

Multi-step use cases now properly wrapped in sqlClient.transaction():
- createProjectUseCase: existsByName + existsBySlug + save
- updateProjectUseCase: findById + existsByName + save
- revokeApiKeyUseCase: findById + save + cache invalidation
- changePlan: findActive + revokeBySubscription + saveMany + save

Co-authored-by: Gerard <gerard@latitude.so>

* feat: add ChSqlClient and withPostgres/withClickHouse helpers

Introduces ChSqlClient as the ClickHouse counterpart to SqlClient:
- ChSqlClient domain interface in @domain/shared (pass-through
  transaction, direct query execution)
- ChSqlClientLive implementation in @platform/db-clickhouse
- DatasetRowRepositoryLive migrated from closure-captured client to
  ChSqlClient service pattern (Layer.effect + yield* ChSqlClient)

Adds withPostgres and withClickHouse helpers that bundle repository
layers with their database client in a single call:
- Uses Layer.provideMerge so the SqlClient/ChSqlClient service is
  available both to the repo layers AND to the outer effect (needed
  for use-case-level sqlClient.transaction() calls)
- Repos sharing the same helper call share the same client instance

Boundary callers updated from:
  Effect.provide(RepoLive),
  Effect.provide(SqlClientLive(client, orgId)),
To:
  Effect.provide(withPostgres(client, orgId, RepoLive)),

Co-authored-by: Gerard <gerard@latitude.so>

* add tests for the postgres sql client

* update lockfile

* fix signup/login

* added comment, do not set rls if admin context

* test(db-postgres): fix sql-client test for system org RLS skip

The implementation skips set_config for the system/admin org context
(added in 9256be6), but the test still expected 1 executed statement.
Update the assertion to expect 0 statements.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(testkit): add PGlite in-memory Postgres adapter and concurrent tx detection

- Add `createInMemoryPostgres` / `closeInMemoryPostgres` to testkit using
  PGlite so tests can run without a real Postgres instance
- Export from testkit index; add `@electric-sql/pglite` and `drizzle-orm`
  as deps to testkit
- Add `vitest.config.ts` to `apps/web` wiring up the vitest preset
- Harden `SqlClientLive`: track `txOpening` flag so concurrent `transaction()`
  calls on the same instance fail fast via `Effect.die` instead of silently
  corrupting connections; add tests for both the error and the sequential-OK path

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor: make withPostgres/withClickHouse return pipe-compatible providers

Previously withPostgres and withClickHouse returned a Layer, requiring
callers to wrap every call site with Effect.provide(). They also accepted
(client, orgId, ...layers) which was hard to type correctly with variadic
layer arguments.

New signature: (layer, client, orgId?) => pipe-compatible function via
Effect.provide internally. Call sites can now use effect.pipe(withPostgres(...))
directly without the outer Effect.provide wrapper.

- Rename withClickHouse to the same shape as withPostgres
- Migrate all call sites in apps/web and apps/api to the new signature
- Consolidate auth functions to withPostgres instead of multiple
  Effect.provide(RepositoryLive) + Effect.provide(SqlClientLive) chains
- Switch exchangeCliSession from adminClient to scoped postgresClient
  since API key creation is org-scoped

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat: migrate ch remaining repos to new pattern

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants