-
Notifications
You must be signed in to change notification settings - Fork 156
Compile query in DB w/ the columnMapper #1141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Investigation of Discord-reported issues with snake_case to camelCase column mapping when using Electric collections with on-demand sync. Key findings: - columnMapper transformations not applied during SQL compilation - compileSQL uses TypeScript property names (camelCase) directly - Electric's structured expression support (whereExpr) not utilized - Results in PostgreSQL errors expecting snake_case column names Includes reproduction steps, root cause analysis, and proposed solutions.
…lation When using columnMapper (e.g., snakeCamelMapper()) with on-demand sync mode, camelCase column names in queries were not being converted back to snake_case before sending to PostgreSQL. Root cause: The SQL compiler quoted identifiers directly using TypeScript property names without applying the columnMapper's encode function. Changes: - Add encodeColumnName option to compileSQL() to transform column names - Pass columnMapper.encode from shapeOptions through createLoadSubsetDedupe - Apply encoding in quoteIdentifier() before quoting - Add comprehensive tests for column name encoding This fixes the issue where queries like eq(d.programTemplateId, value) would generate "programTemplateId" = $1 instead of "program_template_id" = $1. Fixes: Discord-reported issue with snakeCamelMapper in on-demand mode
… directly to useLiveQuery
Add console warning when a collection with syncMode "on-demand" is passed
directly to useLiveQuery. In on-demand mode, data is only loaded when
queries with predicates request it. Passing the collection directly doesn't
provide any predicates, so no data loads.
The warning guides users to either:
1. Use a query builder function: useLiveQuery((q) => q.from({c: collection}))
2. Switch to syncMode "eager" for automatic sync
This helps prevent confusion when users expect data to appear but nothing
loads due to the on-demand sync behavior.
|
More templates
@tanstack/angular-db
@tanstack/db
@tanstack/db-ivm
@tanstack/electric-db-collection
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit: |
|
Size Change: 0 B Total Size: 90.5 kB ℹ️ View Unchanged
|
|
Size Change: 0 B Total Size: 3.47 kB ℹ️ View Unchanged
|
|
Hey hey! Long time follower first time commenter. I reported this bug 1/15 after noticing it the day prior. I've installed this PR locally and can confirm the fix works perfectly. Thanks for all y'all are doing. Keep up the good work |
Summary
Fixes SQL compilation for on-demand sync to properly encode column names using the configured
columnMapper. Previously, queries with camelCase property names would generate incorrect SQL whensnakeCamelMapperwas configured. Also adds developer warnings when on-demand collections are passed directly touseLiveQuerywithout a query builder.Root Cause
When using
snakeCamelMapper(which converts between camelCase JS properties and snake_case DB columns), the SQL compiler was not applying the encoding transformation. For example:The
columnMapper.encodefunction existed and was used during data sync, but was never passed to the SQL compilation step.Approach
SQL Compiler Enhancement:
encodeColumnNameparameter tocompileSQL()compileBasicExpression,compileOrderBy,compileFunction)quoteIdentifier()before quoting column namesIntegration:
createLoadSubsetDedupe()now receivesshapeOptions.columnMapper?.encodecompileSQL()calls pass the compile options when encoding is availableDeveloper Experience:
useLiveQueryhooks when an on-demand collection is passed directly (which loads no data since there's no predicate)Key Invariants
columnMappertransformation must be applied consistently: during sync (already worked) AND during query compilation (this fix)columnMapperis configured, column names pass through unchangedNon-goals
columnMapperworks for data sync (already correct)Trade-offs
Warning vs Error for on-demand misuse: Chose a warning because some developers might intentionally pass an on-demand collection and rely on external mechanisms to load data. An error would be breaking; a warning educates without blocking.
Verification
pnpm test --filter @tanstack/electric-db-collectionThe new tests in
sql-compiler.test.tsverify:Files Changed
packages/electric-db-collection/src/sql-compiler.tsColumnEncodertype andCompileSQLOptionsinterface; threadedencodeColumnNamethrough all compilation functionspackages/electric-db-collection/src/electric.tsshapeOptions.columnMapper?.encodetocreateLoadSubsetDedupe()and through tocompileSQL()packages/electric-db-collection/tests/sql-compiler.test.tspackages/react-db/src/useLiveQuery.tspackages/vue-db/src/useLiveQuery.tspackages/svelte-db/src/useLiveQuery.svelte.ts