-
Notifications
You must be signed in to change notification settings - Fork 9
[Tutorial] Compound indexing and query optimization #100
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
140 changes: 140 additions & 0 deletions
140
tutorials/client/data/compound-indexing-query-performance.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,140 @@ | ||
--- | ||
title: Best Practices for Compound Indexing and Query Optimization in PowerSync | ||
description: In this tutorial we will show you the Best Practices for Compound Indexing and Query Optimization using the [PowerSync Web SDK](https://github.com/powersync-ja/powersync-js). | ||
sidebarTitle: Compound Indexing & Query Performance | ||
--- | ||
|
||
## Introduction | ||
|
||
This tutorial outlines findings and recommendations based on extensive testing of compound indexes, query execution, and table performance using the [PowerSync Web SDK](https://github.com/powersync-ja/powersync-js). | ||
These best practices are designed to assist developers in optimizing their schemas and queries when working with large datasets. | ||
|
||
## Key Findings | ||
|
||
### 1. Compound Index Behavior | ||
|
||
- Compound indexes significantly improve query performance when filters align with the indexed column order. | ||
- Queries that skip leading columns in a compound index result in full table scans, negating performance benefits. | ||
- Performance is consistent with SQLite's behavior as PowerSync uses SQLite under the hood. | ||
|
||
### 2. Performance Benchmarks | ||
|
||
<Note> | ||
The following results were obtained using a table with 1999 columns and 50k entries | ||
</Note> | ||
|
||
|
||
#### Queries with Leading Column Filters | ||
|
||
```sql | ||
EXPLAIN QUERY PLAN SELECT * FROM dummy_table WHERE col1 = 'val1'; | ||
``` | ||
|
||
- **Execution Time:** 256 ms | ||
- **Query Plan:** SEARCH dummy_table USING COMPOUND INDEX | ||
|
||
#### Queries with Multiple Indexed Columns | ||
|
||
```sql | ||
EXPLAIN QUERY PLAN SELECT * FROM dummy_table WHERE col1 = 'val1' AND col2 = 'val2'; | ||
``` | ||
|
||
- **Execution Time:** 206 | ||
- **Query Plan:** SEARCH dummy_table USING COMPOUND INDEX | ||
|
||
#### Queries Skipping Leading Columns | ||
|
||
```sql | ||
EXPLAIN QUERY PLAN SELECT * FROM dummy_table WHERE col2 = 'val2' AND col3 = 'val3'; | ||
``` | ||
|
||
- **Execution Time:** 556 ms | ||
- **Query Plan:** SCAN dummy_table | ||
#### Performance Summary | ||
|
||
| Query Type | Execution Time | Query Plan | | ||
|-----------------------------------|----------------|-------------------------------------| | ||
| Leading Column Filters | 253 ms | SEARCH dummy_table USING COMPOUND INDEX | | ||
| Multiple Indexed Columns | 206 ms | SEARCH dummy_table USING COMPOUND INDEX | | ||
| Skipping Leading Columns | 556 ms | SCAN dummy_table | | ||
### 3. Column Limitations | ||
|
||
- PowerSync supports a maximum of 1,999 columns in a compound index, aligning with SQLite's limitations. | ||
- Queries including non-indexed columns slightly increase execution time but remain performant under typical workloads. | ||
|
||
## Best Practices | ||
|
||
**Faster query execution** can be achieved by following these best practices. | ||
|
||
### Schema Definition | ||
|
||
Define compound indexes to match your most frequent query patterns: | ||
|
||
```javascript | ||
import { column, Table } from '@powersync/web'; | ||
|
||
const todos = new Table( | ||
{ | ||
list_id: column.text, // Foreign key to lists | ||
created_at: column.text, // Creation timestamp | ||
description: column.text, // Task description | ||
}, | ||
{ | ||
indexes: { | ||
list_created: ['list_id', 'created_at'], | ||
}, | ||
} | ||
); | ||
``` | ||
|
||
### Query Optimization | ||
|
||
#### Aligned Filters | ||
|
||
Ensure that queries align with the column order of compound indexes: | ||
|
||
```javascript | ||
const results = await powerSync.get( | ||
`SELECT * FROM todos WHERE list_id = ? AND created_at = ?`, | ||
['list1', '2025-01-26'] | ||
); | ||
``` | ||
|
||
#### Skipping Leading Columns | ||
|
||
For queries skipping leading columns, define additional indexes: | ||
|
||
```javascript | ||
await powerSync.execute( | ||
`CREATE INDEX idx_description ON todos (description);` | ||
); | ||
``` | ||
|
||
### Advanced Scenarios | ||
|
||
#### Testing Column Order | ||
|
||
The order of indexed columns affects query performance: | ||
|
||
```javascript | ||
// Create compound index with a different order | ||
await powerSync.execute(`CREATE INDEX idx_order ON todos (description, list_id);`); | ||
|
||
// Query execution profiling | ||
console.time('Query with reordered index'); | ||
await powerSync.get( | ||
`SELECT * FROM todos WHERE description = ? AND list_id = ?`, | ||
['Task 1', 'list1'] | ||
); | ||
console.timeEnd('Query with reordered index'); | ||
``` | ||
|
||
## Key Takeaways | ||
|
||
- Define compound indexes aligned with your most frequent query patterns. | ||
- Avoid skipping leading columns in compound indexes; create additional indexes if necessary. | ||
|
||
## Conclusion | ||
|
||
By following these best practices and leveraging PowerSync's schema capabilities, developers can achieve significant performance gains and ensure scalability for their applications. | ||
The findings above can be integrated directly into PowerSync's documentation to assist other developers in navigating these challenges effectively. | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd suggest to link to the SDK reference in docs since it's the higher-level starting point (and also includes link to source code): https://docs.powersync.com/client-sdk-references/javascript-web