Skip to content

Sparse indexes #4404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
atovpeko merged 14 commits into from
Sep 12, 2025
Merged

Sparse indexes #4404

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
179db7b
draft about space indexes
atovpeko Sep 8, 2025
b5e018b
Merge branch 'latest' into 348-docs-rfcadd-configurable-sparse-indexe…
billy-the-fish Sep 10, 2025
b0d0c16
update
atovpeko Sep 10, 2025
ab49034
Update api/hypercore/hypertable_columnstore_settings.md
atovpeko Sep 10, 2025
21edead
Update api/hypercore/chunk_columnstore_settings.md
atovpeko Sep 10, 2025
59a75ec
update
atovpeko Sep 10, 2025
22cf070
revert some stuff
atovpeko Sep 10, 2025
35064aa
updates on review
atovpeko Sep 11, 2025
db83006
Merge branch 'latest' into 348-docs-rfcadd-configurable-sparse-indexe…
atovpeko Sep 11, 2025
da6c117
update on review
atovpeko Sep 11, 2025
d15f6cd
Merge branch 'latest' into 348-docs-rfcadd-configurable-sparse-indexe…
atovpeko Sep 11, 2025
9456c53
Update api/hypercore/chunk_columnstore_settings.md
atovpeko Sep 11, 2025
beeac5a
update on review
atovpeko Sep 11, 2025
2464c1b
update on review
atovpeko Sep 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _partials/_hypercore-conversion-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ Is converted and compressed into arrays in a row in the columnstore:
|-|-|-|-|-|
|[12:00:01, 12:00:01, 12:00:02, 12:00:02, 12:00:03, 12:00:03]|[A, B, A, B, A, B]|[SSD, HDD, SSD, HDD, SSD, HDD]|[70.11, 69.70, 70.12, 69.69, 70.14, 69.70]|[13.4, 20.5, 13.2, 23.4, 13.0, 25.2]|

Because a single row takes up less disk space, you can reduce your chunk size by more than 90%, and can also
Because a single row takes up less disk space, you can reduce your chunk size by up to 98%, and can also
speed up your queries. This saves on storage costs, and keeps your queries operating at lightning speed.
2 changes: 1 addition & 1 deletion _partials/_hypercore-intro.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ $HYPERCORE_CAP’s hybrid approach combines the benefits of row-oriented and col
saving significant storage space.

- **Faster queries on compressed data in $COLUMNSTORE**: in the $COLUMNSTORE conversion, hypertable
chunks are compressed by more than 90%, and organized for efficient, large-scale queries. Combined with [chunk skipping][chunk-skipping], this helps you save on storage costs and keeps your queries operating at lightning speed.
chunks are compressed by up to 98%, and organized for efficient, large-scale queries. Combined with [chunk skipping][chunk-skipping], this helps you save on storage costs and keeps your queries operating at lightning speed.

- **Fast modification of compressed data in $COLUMNSTORE**: just use SQL to add or modify data in the $COLUMNSTORE.
$TIMESCALE_DB is optimized for super fast INSERT and UPSERT performance.
Expand Down
28 changes: 15 additions & 13 deletions api/hypercore/alter_table.md
View file
Open in desktop

Large diffs are not rendered by default.

13 changes: 8 additions & 5 deletions api/hypercore/chunk_columnstore_settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ To retrieve information about settings:
SELECT * FROM timescaledb_information.chunk_columnstore_settings
```
Returns:

```sql
hypertable | chunk | segmentby | orderby
------------+-------+-----------+---------
Expand All @@ -41,6 +42,7 @@ To retrieve information about settings:
WHERE hypertable::TEXT LIKE 'metrics';
```
Returns:

```sql
hypertable | chunk | segmentby | orderby
------------+-------+-----------+---------
Expand All @@ -49,10 +51,11 @@ To retrieve information about settings:

## Returns

| Name | Type | Default | Required | Description |
| Name | Type | Description |
|--|--|--|--|--|
|`hypertable`|`REGCLASS`|-|✖| The name of a hypertable in the $COLUMNSTORE |
|`chunk`|`REGCLASS`|-|✖| The name of a chunk in `hypertable` |
|`segmentby`|`TEXT`|-|✖| A list of columns used to segment `hypertable` |
|`orderby`|`TEXT`|-|✖| A list of columns used to order data in `hypertable`. Along with ordering and NULL ordering information. |
|`hypertable`|`REGCLASS`| The name of the hypertable in the $COLUMNSTORE. |
|`chunk`|`REGCLASS`| The name of the chunk in the `hypertable`. |
|`segmentby`|`TEXT`| The list of columns used to segment the `hypertable`. |
|`orderby`|`TEXT`| The list of columns used to order the data in the `hypertable`, along with ordering and NULL ordering information. |
|`index`| `TEXT` | The sparse index details. |

14 changes: 7 additions & 7 deletions api/hypercore/hypertable_columnstore_settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,13 @@ To retrieve information about settings:

## Returns

|Name|Type| Description |
|-|-|---------------------------------------------------------------------------------------------------------------------|
|`hypertable`|`REGCLASS`| A hypertable which has the [$COLUMNSTORE enabled][compression_alter-table]. |
|`segmentby`|`TEXT`| The list of columns used to segment data |
|`orderby`|`TEXT`| List of columns used to order the data, along with ordering and NULL ordering information |
|`compress_interval_length`|`TEXT`| Interval used for [rolling up chunks during compression][rollup-compression] |

|Name|Type| Description |
|-|-|-------------------------------------------------------------------------------------------|
|`hypertable`|`REGCLASS`| A hypertable which has the [$COLUMNSTORE enabled][compression_alter-table].|
|`segmentby`|`TEXT`| The list of columns used to segment data. |
|`orderby`|`TEXT`| List of columns used to order the data, along with ordering and NULL ordering information. |
|`compress_interval_length`|`TEXT`| Interval used for [rolling up chunks during compression][rollup-compression]. |
|`index`| `TEXT` | The sparse index details. |


[rollup-compression]: /use-timescale/:currentVersion:/compression/manual-compression/#roll-up-uncompressed-chunks-when-compressing
Expand Down
26 changes: 14 additions & 12 deletions api/hypertable/create_table.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ a $HYPERTABLE is partitioned on the time dimension. To add secondary dimensions
As the data cools and becomes more suited for analytics, [add a columnstore policy][add_columnstore_policy] so your data
is automatically converted to the $COLUMNSTORE after a specific time interval. This columnar format enables fast
scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space.
In the $COLUMNSTORE conversion, $HYPERTABLE chunks are compressed by more than 90%, and organized for efficient,
In the $COLUMNSTORE conversion, $HYPERTABLE chunks are compressed by up to 98%, and organized for efficient,
large-scale queries. This columnar format enables fast scanning and aggregation, optimizing performance for analytical
workloads. You can also manually [convert chunks][convert_to_columnstore] in a $HYPERTABLE to the $COLUMNSTORE.

Expand Down Expand Up @@ -61,6 +61,7 @@ arguments specific to $TIMESCALE_DB.
```

1. Enable $HYPERCORE by adding a columnstore policy:

```sql
CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '1d');
```
Expand Down Expand Up @@ -120,20 +121,21 @@ WITH (
tsdb.associated_table_prefix = '<prefix>'
tsdb.orderby = '<column_name> [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]',
tsdb.segmentby = '<column_name> [, ...]',
tsdb.index = '<index>(<column_name>), index(<column_name>)'
)
```

| Name | Type | Default | Required | Description |
|--------------------------------|------------------|----------|-------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `tsdb.hypertable` |BOOLEAN| `true` | ✖ | Create a new [hypertable][hypertable-docs] for time-series data rather than a standard $PG relational table. |
| `tsdb.partition_column` |TEXT| `true` | ✖ | Set the time column to automatically partition your time-series data by. |
| `tsdb.chunk_interval` |TEXT| `7 days` | ✖ | Change this to better suit your needs. For example, if you set `chunk_interval` to 1 day, each chunk stores data from the same day. Data from different days is stored in different chunks. |
| `tsdb.create_default_indexes` | BOOLEAN | `true` | ✖ | Set to `false` to not automatically create indexes. <br/> The default indexes are: <ul><li>On all hypertables, a descending index on `partition_column`</li><li>On hypertables with space partitions, an index on the space parameter and `partition_column`</li></ul> |
| `tsdb.associated_schema` |REGCLASS| `_timescaledb_internal` | ✖ | Set the schema name for internal hypertable tables. |
| `tsdb.associated_table_prefix` |TEXT|`_hyper`| ✖ | Set the prefix for the names of internal hypertable chunks. |
| `tsdb.orderby` |TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the $COLUMNSTORE. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. |
| `tsdb.segmentby` |TEXT| No segmentation by column. | ✖| Set the list of columns used to segment data in the $COLUMNSTORE for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. |

| Name | Type | Default | Required | Description |
|--------------------------------|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `tsdb.hypertable` |BOOLEAN| `true` | ✖ | Create a new [hypertable][hypertable-docs] for time-series data rather than a standard $PG relational table. |
| `tsdb.partition_column` |TEXT| `true` | ✖ | Set the time column to automatically partition your time-series data by. |
| `tsdb.chunk_interval` |TEXT| `7 days` | ✖ | Change this to better suit your needs. For example, if you set `chunk_interval` to 1 day, each chunk stores data from the same day. Data from different days is stored in different chunks. |
| `tsdb.create_default_indexes` | BOOLEAN | `true` | ✖ | Set to `false` to not automatically create indexes. <br/> The default indexes are: <ul><li>On all hypertables, a descending index on `partition_column`</li><li>On hypertables with space partitions, an index on the space parameter and `partition_column`</li></ul> |
| `tsdb.associated_schema` |REGCLASS| `_timescaledb_internal` | ✖ | Set the schema name for internal hypertable tables. |
| `tsdb.associated_table_prefix` |TEXT| `_hyper` | ✖ | Set the prefix for the names of internal hypertable chunks. |
| `tsdb.orderby` |TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the $COLUMNSTORE. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. Setting `tsdb.orderby` automatically creates an implicit min/max sparse index on the `orderby` column. |
| `tsdb.segmentby` |TEXT| No segmentation by column. | ✖| Set the list of columns used to segment data in the $COLUMNSTORE for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. |
|`tsdb.index`| TEXT | $TIMESCALE_DB evaluates the columns you already have indexed and checks which data types are a good fit for sparse indexing, then creates a sparse index as an optimization. | ✖ | Configure the sparse indexes for compressed chunks. Requires setting `tsdb.orderby`. Supported index types include: <li> `bloom(<column_name>)`: a probabilistic index, effective for `=` filters. Cannot be applied to `tsdb.orderby` columns.</li> <li> `minmax(<column_name>)`: stores min/max values for each compressed chunk. Setting `tsdb.orderby` automatically creates an implicit min/max sparse index on the `orderby` column. </li> Define multiple indexes using a comma-separated list. You can set only one index per column. Set to an empty string to avoid using sparse indexes and explicitly disable the default behavior. |

## Returns

Expand Down
2 changes: 1 addition & 1 deletion getting-started/try-key-features-timescale-products.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ challenges in real-time analytics.

When $TIMESCALE_DB converts $CHUNKs from the $ROWSTORE to the $COLUMNSTORE, multiple records are grouped into a single row.
The columns of this row hold an array-like structure that stores all the data. Because a single row takes up less disk
space, you can reduce your $CHUNK size by more than 90%, and can also speed up your queries. This helps you save on storage costs,
space, you can reduce your $CHUNK size by up to 98%, and can also speed up your queries. This helps you save on storage costs,
and keeps your queries operating at lightning speed.

$HYPERCORE is enabled by default when you call [CREATE TABLE][hypertable-create-table]. Best practice is to compress
Expand Down
2 changes: 1 addition & 1 deletion use-timescale/hypercore/real-time-analytics-in-hypercore.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Optimize your data for real-time analytics
excerpt: Reduce your chunk size by more than 90% and speed up your queries by automatically converting data between the rowstore and columnstore
excerpt: Reduce your chunk size by up to 98% and speed up your queries by automatically converting data between the rowstore and columnstore
products: [cloud, self_hosted]
keywords: [hyperscore, hypertable, compression, row-columnar storage, hypercore]
---
Expand Down
Loading