diff --git a/_partials/_hypercore-conversion-overview.md b/_partials/_hypercore-conversion-overview.md index 7cc2bf5ebc..307ccabbbb 100644 --- a/_partials/_hypercore-conversion-overview.md +++ b/_partials/_hypercore-conversion-overview.md @@ -17,5 +17,5 @@ Is converted and compressed into arrays in a row in the columnstore: |-|-|-|-|-| |[12:00:01, 12:00:01, 12:00:02, 12:00:02, 12:00:03, 12:00:03]|[A, B, A, B, A, B]|[SSD, HDD, SSD, HDD, SSD, HDD]|[70.11, 69.70, 70.12, 69.69, 70.14, 69.70]|[13.4, 20.5, 13.2, 23.4, 13.0, 25.2]| -Because a single row takes up less disk space, you can reduce your chunk size by more than 90%, and can also +Because a single row takes up less disk space, you can reduce your chunk size by up to 98%, and can also speed up your queries. This saves on storage costs, and keeps your queries operating at lightning speed. diff --git a/_partials/_hypercore-intro.md b/_partials/_hypercore-intro.md index b8cacb30b6..be757542ad 100644 --- a/_partials/_hypercore-intro.md +++ b/_partials/_hypercore-intro.md @@ -21,7 +21,7 @@ $HYPERCORE_CAP’s hybrid approach combines the benefits of row-oriented and col saving significant storage space. - **Faster queries on compressed data in $COLUMNSTORE**: in the $COLUMNSTORE conversion, hypertable - chunks are compressed by more than 90%, and organized for efficient, large-scale queries. Combined with [chunk skipping][chunk-skipping], this helps you save on storage costs and keeps your queries operating at lightning speed. + chunks are compressed by up to 98%, and organized for efficient, large-scale queries. Combined with [chunk skipping][chunk-skipping], this helps you save on storage costs and keeps your queries operating at lightning speed. - **Fast modification of compressed data in $COLUMNSTORE**: just use SQL to add or modify data in the $COLUMNSTORE. $TIMESCALE_DB is optimized for super fast INSERT and UPSERT performance. diff --git a/api/hypercore/alter_table.md b/api/hypercore/alter_table.md index 554c661273..d7e9e2fe36 100644 --- a/api/hypercore/alter_table.md +++ b/api/hypercore/alter_table.md @@ -17,6 +17,8 @@ import EarlyAccess from "versionContent/_partials/_early_access_2_18_0.mdx"; Enable the $COLUMNSTORE or change the $COLUMNSTORE settings for a $HYPERTABLE. The settings are applied on a per-chunk basis. You do not need to convert the entire $HYPERTABLE back to the $ROWSTORE before changing the settings. The new settings apply only to the chunks that have not yet been converted to $COLUMNSTORE, the existing chunks in the $COLUMNSTORE do not change. This means that chunks with different $COLUMNSTORE settings can co-exist in the same $HYPERTABLE. +$TIMESCALE_DB calculates default $COLUMNSTORE settings for each chunk when it is created. These settings apply to each chunk, and not the entire hypertable. To explicitly disable the defaults, set a setting to an empty string. To remove the current configuration and re-enable the defaults, call `ALTER TABLE RESET ();`. + After you have enabled the $COLUMNSTORE, either: - [add_columnstore_policy][add_columnstore_policy]: create a [job][job] that automatically moves chunks in a hypertable to the $COLUMNSTORE at a specific time interval. @@ -60,8 +62,9 @@ The syntax is: ``` sql ALTER TABLE SET (timescaledb.enable_columnstore, - timescaledb.orderby = ' [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]', - timescaledb.segmentby = ' [, ...]', + timescaledb.compress_orderby = ' [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]', + timescaledb.compress_segmentby = ' [, ...]', + timescaledb.compress_index = '(), ()' timescaledb.compress_chunk_time_interval='interval', SET ACCESS METHOD { new_access_method | DEFAULT }, ALTER SET NOT NULL, @@ -69,17 +72,18 @@ ALTER TABLE SET (timescaledb.enable_columnstore, ); ``` -| Name | Type | Default | Required | Description | -|--------------------------------------------|--|------------------------------------------------------|--|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `table_name` |TEXT| - | ✖ | The hypertable to enable columstore for. | -| `timescaledb.enable_columnstore` |BOOLEAN| `true` | ✖ | Enable $COLUMNSTORE. | -| `timescaledb.orderby` |TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the $COLUMNSTORE. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. | -| `timescaledb.segmentby` |TEXT| No segementation by column. | ✖| Set the list of columns used to segment data in the $COLUMNSTORE for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | -| `column_name` |TEXT| - | ✖ | The name of the column to `orderby` or `segmentby`. | -| `timescaledb.compress_chunk_time_interval` |TEXT| - | ✖ | EXPERIMENTAL: reduce the total number of chunks in the $COLUMNSTORE for `table`. If you set `compress_chunk_time_interval`, chunks added to the $COLUMNSTORE are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call [convert_to_rowstore][convert_to_rowstore], merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. | -| `interval` |TEXT| - | ✖ | Set to a multiple of the [chunk_time_interval][chunk_time_interval] for `table`. | -| `ALTER` |TEXT| | ✖| Set a specific column in the columnstore to be NOT NULL. | -| `ADD CONSTRAINT` |TEXT| | ✖| Add UNIQUE constraints to data in the columnstore. | +| Name | Type | Default | Required | Description | +|-------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------------| +| `table_name` | TEXT | - | ✖ | The hypertable to enable columstore for. | +| `timescaledb.enable_columnstore` | BOOLEAN | `true` | ✖ | Set to `false` to disable $COLUMNSTORE. | +| `timescaledb.compress_orderby` | TEXT | Descending order on the time column in `table_name`. | ✖ | The order in which items are used in the $COLUMNSTORE. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. Setting `timescaledb.compress_orderby` automatically creates an implicit min/max sparse index on the `orderby` column. | +| `timescaledb.compress_segmentby` | TEXT | $TIMESCALE_DB looks at [`pg_stats`](https://www.postgresql.org/docs/current/view-pg-stats.html) and determines an appropriate column based on the data cardinality and distribution. If `pg_stats` is not available, $TIMESCALE_DB looks for an appropriate column from the existing indexes. | ✖ | Set the list of columns used to segment data in the $COLUMNSTORE for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | +| `column_name` | TEXT | - | ✖ | The name of the column to `orderby` or `segmentby`. | +|`timescaledb.compress_index`| TEXT | $TIMESCALE_DB evaluates the columns you already have indexed, checks which data types are a good fit for sparse indexing, then creates a sparse index as an optimization. | ✖ | Configure the sparse indexes for compressed chunks. Requires setting `timescaledb.compress_orderby`. Supported index types include:
  • `bloom()`: a probabilistic index, effective for `=` filters. Cannot be applied to `timescaledb.compress_orderby` columns.
  • `minmax()`: stores min/max values for each compressed chunk. Setting `timescaledb.compress_orderby` automatically creates an implicit min/max sparse index on the `orderby` column.
    • Define multiple indexes using a comma-separated list. You can set only one index per column. Set to an empty string to avoid using sparse indexes and explicitly disable the default behavior. To remove the current sparse index configuration and re-enable default sparse index selection, call `ALTER TABLE your_table_name RESET (timescaledb.compress_index);`. | +| `timescaledb.compress_chunk_time_interval` | TEXT | - | ✖ | EXPERIMENTAL: reduce the total number of chunks in the $COLUMNSTORE for `table`. If you set `compress_chunk_time_interval`, chunks added to the $COLUMNSTORE are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call [convert_to_rowstore][convert_to_rowstore], merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. | +| `interval` | TEXT | - | ✖ | Set to a multiple of the [chunk_time_interval][chunk_time_interval] for `table`. | +| `ALTER` | TEXT | | ✖ | Set a specific column in the columnstore to be `NOT NULL`. | +| `ADD CONSTRAINT` | TEXT | | ✖ | Add `UNIQUE` constraints to data in the columnstore. | [chunk_time_interval]: /api/:currentVersion:/hypertable/set_chunk_time_interval/ [add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ diff --git a/api/hypercore/chunk_columnstore_settings.md b/api/hypercore/chunk_columnstore_settings.md index 14d8300bc8..37a2dab380 100644 --- a/api/hypercore/chunk_columnstore_settings.md +++ b/api/hypercore/chunk_columnstore_settings.md @@ -27,6 +27,7 @@ To retrieve information about settings: SELECT * FROM timescaledb_information.chunk_columnstore_settings ``` Returns: + ```sql hypertable | chunk | segmentby | orderby ------------+-------+-----------+--------- @@ -41,6 +42,7 @@ To retrieve information about settings: WHERE hypertable::TEXT LIKE 'metrics'; ``` Returns: + ```sql hypertable | chunk | segmentby | orderby ------------+-------+-----------+--------- @@ -49,10 +51,11 @@ To retrieve information about settings: ## Returns -| Name | Type | Default | Required | Description | +| Name | Type | Description | |--|--|--|--|--| -|`hypertable`|`REGCLASS`|-|✖| The name of a hypertable in the $COLUMNSTORE | -|`chunk`|`REGCLASS`|-|✖| The name of a chunk in `hypertable` | -|`segmentby`|`TEXT`|-|✖| A list of columns used to segment `hypertable` | -|`orderby`|`TEXT`|-|✖| A list of columns used to order data in `hypertable`. Along with ordering and NULL ordering information. | +|`hypertable`|`REGCLASS`| The name of the hypertable in the $COLUMNSTORE. | +|`chunk`|`REGCLASS`| The name of the chunk in the `hypertable`. | +|`segmentby`|`TEXT`| The list of columns used to segment the `hypertable`. | +|`orderby`|`TEXT`| The list of columns used to order the data in the `hypertable`, along with the ordering and `NULL` ordering information. | +|`index`| `TEXT` | The sparse index details. | diff --git a/api/hypercore/hypertable_columnstore_settings.md b/api/hypercore/hypertable_columnstore_settings.md index 3dfefa21f8..136c509e79 100644 --- a/api/hypercore/hypertable_columnstore_settings.md +++ b/api/hypercore/hypertable_columnstore_settings.md @@ -49,13 +49,13 @@ To retrieve information about settings: ## Returns -|Name|Type| Description | -|-|-|---------------------------------------------------------------------------------------------------------------------| -|`hypertable`|`REGCLASS`| A hypertable which has the [$COLUMNSTORE enabled][compression_alter-table]. | -|`segmentby`|`TEXT`| The list of columns used to segment data | -|`orderby`|`TEXT`| List of columns used to order the data, along with ordering and NULL ordering information | -|`compress_interval_length`|`TEXT`| Interval used for [rolling up chunks during compression][rollup-compression] | - +|Name|Type| Description | +|-|-|-------------------------------------------------------------------------------------------| +|`hypertable`|`REGCLASS`| A hypertable which has the [$COLUMNSTORE enabled][compression_alter-table].| +|`segmentby`|`TEXT`| The list of columns used to segment data. | +|`orderby`|`TEXT`| List of columns used to order the data, along with ordering and NULL ordering information. | +|`compress_interval_length`|`TEXT`| Interval used for [rolling up chunks during compression][rollup-compression]. | +|`index`| `TEXT` | The sparse index details. | [rollup-compression]: /use-timescale/:currentVersion:/compression/manual-compression/#roll-up-uncompressed-chunks-when-compressing diff --git a/api/hypertable/create_table.md b/api/hypertable/create_table.md index b046cdca35..8060dd3404 100644 --- a/api/hypertable/create_table.md +++ b/api/hypertable/create_table.md @@ -27,7 +27,7 @@ a $HYPERTABLE is partitioned on the time dimension. To add secondary dimensions As the data cools and becomes more suited for analytics, [add a columnstore policy][add_columnstore_policy] so your data is automatically converted to the $COLUMNSTORE after a specific time interval. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. -In the $COLUMNSTORE conversion, $HYPERTABLE chunks are compressed by more than 90%, and organized for efficient, +In the $COLUMNSTORE conversion, $HYPERTABLE chunks are compressed by up to 98%, and organized for efficient, large-scale queries. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads. You can also manually [convert chunks][convert_to_columnstore] in a $HYPERTABLE to the $COLUMNSTORE. @@ -35,6 +35,8 @@ $HYPERTABLE_CAP to $HYPERTABLE foreign keys are not allowed, all other combinati The [$COLUMNSTORE][hypercore] settings are applied on a per-chunk basis. You can change the settings by calling [ALTER TABLE][alter_table_hypercore] without first converting the entire $HYPERTABLE back to the [$ROWSTORE][hypercore]. The new settings apply only to the chunks that have not yet been converted to $COLUMNSTORE, the existing chunks in the $COLUMNSTORE do not change. Similarly, if you [remove an existing columnstore policy][remove_columnstore_policy] and then [add a new one][add_columnstore_policy], the new policy applies only to the unconverted chunks. This means that chunks with different $COLUMNSTORE settings can co-exist in the same $HYPERTABLE. +$TIMESCALE_DB calculates default $COLUMNSTORE settings for each chunk when it is created. These settings apply to each chunk, and not the entire hypertable. To explicitly disable the defaults, set a setting to an empty string. + `CREATE TABLE` extends the standard $PG [CREATE TABLE][pg-create-table]. This page explains the features and arguments specific to $TIMESCALE_DB. @@ -61,6 +63,7 @@ arguments specific to $TIMESCALE_DB. ``` 1. Enable $HYPERCORE by adding a columnstore policy: + ```sql CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '1d'); ``` @@ -120,19 +123,22 @@ WITH ( tsdb.associated_table_prefix = '' tsdb.orderby = ' [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]', tsdb.segmentby = ' [, ...]', + tsdb.index = '(), index()' ) ``` -| Name | Type | Default | Required | Description | -|--------------------------------|------------------|----------|-------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `tsdb.hypertable` |BOOLEAN| `true` | ✖ | Create a new [hypertable][hypertable-docs] for time-series data rather than a standard $PG relational table. | -| `tsdb.partition_column` |TEXT| `true` | ✖ | Set the time column to automatically partition your time-series data by. | -| `tsdb.chunk_interval` |TEXT| `7 days` | ✖ | Change this to better suit your needs. For example, if you set `chunk_interval` to 1 day, each chunk stores data from the same day. Data from different days is stored in different chunks. | -| `tsdb.create_default_indexes` | BOOLEAN | `true` | ✖ | Set to `false` to not automatically create indexes.
    The default indexes are:
    • On all hypertables, a descending index on `partition_column`
    • On hypertables with space partitions, an index on the space parameter and `partition_column`
    | -| `tsdb.associated_schema` |REGCLASS| `_timescaledb_internal` | ✖ | Set the schema name for internal hypertable tables. | -| `tsdb.associated_table_prefix` |TEXT|`_hyper`| ✖ | Set the prefix for the names of internal hypertable chunks. | -| `tsdb.orderby` |TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the $COLUMNSTORE. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. | -| `tsdb.segmentby` |TEXT| No segmentation by column. | ✖| Set the list of columns used to segment data in the $COLUMNSTORE for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | +| Name | Type | Default | Required | Description | +|--------------------------------|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `tsdb.hypertable` |BOOLEAN| `true` | ✖ | Create a new [hypertable][hypertable-docs] for time-series data rather than a standard $PG relational table. | +| `tsdb.partition_column` |TEXT| `true` | ✖ | Set the time column to automatically partition your time-series data by. | +| `tsdb.chunk_interval` |TEXT| `7 days` | ✖ | Change this to better suit your needs. For example, if you set `chunk_interval` to 1 day, each chunk stores data from the same day. Data from different days is stored in different chunks. | +| `tsdb.create_default_indexes` | BOOLEAN | `true` | ✖ | Set to `false` to not automatically create indexes.
    The default indexes are:
    • On all hypertables, a descending index on `partition_column`
    • On hypertables with space partitions, an index on the space parameter and `partition_column`
    | +| `tsdb.associated_schema` |REGCLASS| `_timescaledb_internal` | ✖ | Set the schema name for internal hypertable tables. | +| `tsdb.associated_table_prefix` |TEXT| `_hyper` | ✖ | Set the prefix for the names of internal hypertable chunks. | +| `tsdb.orderby` |TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the $COLUMNSTORE. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. Setting `tsdb.orderby` automatically creates an implicit min/max sparse index on the `orderby` column. | +| `tsdb.segmentby` |TEXT| $TIMESCALE_DB looks at [`pg_stats`](https://www.postgresql.org/docs/current/view-pg-stats.html) and determines an appropriate column based on the data cardinality and distribution. If `pg_stats` is not available, $TIMESCALE_DB looks for an appropriate column from the existing indexes. | ✖| Set the list of columns used to segment data in the $COLUMNSTORE for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | +|`tsdb.index`| TEXT | $TIMESCALE_DB evaluates the columns you already have indexed, checks which data types are a good fit for sparse indexing, then creates a sparse index as an optimization. | ✖ | Configure the sparse indexes for compressed chunks. Requires setting `tsdb.orderby`. Supported index types include:
  • `bloom()`: a probabilistic index, effective for `=` filters. Cannot be applied to `tsdb.orderby` columns.
  • `minmax()`: stores min/max values for each compressed chunk. Setting `tsdb.orderby` automatically creates an implicit min/max sparse index on the `orderby` column.
    • Define multiple indexes using a comma-separated list. You can set only one index per column. Set to an empty string to avoid using sparse indexes and explicitly disable the default behavior. | + ## Returns diff --git a/getting-started/try-key-features-timescale-products.md b/getting-started/try-key-features-timescale-products.md index b7faf406c8..6b9830e9df 100644 --- a/getting-started/try-key-features-timescale-products.md +++ b/getting-started/try-key-features-timescale-products.md @@ -180,7 +180,7 @@ challenges in real-time analytics. When $TIMESCALE_DB converts $CHUNKs from the $ROWSTORE to the $COLUMNSTORE, multiple records are grouped into a single row. The columns of this row hold an array-like structure that stores all the data. Because a single row takes up less disk -space, you can reduce your $CHUNK size by more than 90%, and can also speed up your queries. This helps you save on storage costs, +space, you can reduce your $CHUNK size by up to 98%, and can also speed up your queries. This helps you save on storage costs, and keeps your queries operating at lightning speed. $HYPERCORE is enabled by default when you call [CREATE TABLE][hypertable-create-table]. Best practice is to compress diff --git a/use-timescale/hypercore/real-time-analytics-in-hypercore.md b/use-timescale/hypercore/real-time-analytics-in-hypercore.md index ea22a0da2d..a657eeb3ba 100644 --- a/use-timescale/hypercore/real-time-analytics-in-hypercore.md +++ b/use-timescale/hypercore/real-time-analytics-in-hypercore.md @@ -1,6 +1,6 @@ --- title: Optimize your data for real-time analytics -excerpt: Reduce your chunk size by more than 90% and speed up your queries by automatically converting data between the rowstore and columnstore +excerpt: Reduce your chunk size by up to 98% and speed up your queries by automatically converting data between the rowstore and columnstore products: [cloud, self_hosted] keywords: [hyperscore, hypertable, compression, row-columnar storage, hypercore] --- diff --git a/use-timescale/schema-management/indexing.md b/use-timescale/schema-management/indexing.md index b118913b35..d1c7fd0a0a 100644 --- a/use-timescale/schema-management/indexing.md +++ b/use-timescale/schema-management/indexing.md @@ -39,6 +39,8 @@ You can also create an additional index on another column and time. For example: CREATE INDEX ON conditions (location, time DESC); ``` +$TIMESCALE_DB also creates sparse indexes per compressed chunk for optimization. You can manually set up those indexes when you call [`CREATE TABLE`][hypertable-create-table] or [`ALTER_TABLE`][alter-table]. + For more information about the order to use when declaring indexes, see the [about indexing][about-index] section. @@ -99,3 +101,4 @@ to perform indexing transactions on an individual chunk. [about-index]: /use-timescale/:currentVersion:/schema-management/about-indexing/ [create-index]: https://docs.tigerdata.com/api/latest/hypertable/create_index/ [hypertable-create-table]: /api/:currentVersion:/hypertable/create_table/ +[alter-table]: /api/:currentVersion:/hypercore/alter_table/