Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions api/compression/compress_chunk.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,11 @@ SELECT compress_chunk('_timescaledb_internal._hyper_1_2_chunk');

## Optional arguments

| Name | Type | Default | Required | Description |
|----------------------|--|---------|--|----------------------------------------------------------------------------------------------------------------------------------------------------|
| `chunk` | REGCLASS | - |✔| Name of the chunk to add to the $COLUMNSTORE. |
| `if_not_columnstore` | BOOLEAN | `true` |✖| Set to `false` so this job fails with an error rather than a warning if `chunk` is already in the $COLUMNSTORE. |
| `recompress` | BOOLEAN | `false` |✖| Set to true to recompress. In-memory recompression will be attempted first; otherwise it will fall back to internal decompress/compress. |
| Name | Type | Default | Required | Description |
|----------------------|--|---------|--|--------------------------------------------------------------------------------------------------------------------------------|
| `chunk` | REGCLASS | - |✔| Name of the chunk to add to the $COLUMNSTORE. |
| `if_not_columnstore` | BOOLEAN | `true` |✖| Set to `false` so this job fails with an error rather than a warning if `chunk` is already in the $COLUMNSTORE. |
| `recompress` | BOOLEAN | `false` |✖| Set to true to recompress. In-memory recompression is attempted first; it falls back to internal decompress/compress. |

## Returns

Expand Down
10 changes: 5 additions & 5 deletions api/hypercore/convert_to_columnstore.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk');

## Arguments

| Name | Type | Default | Required | Description |
|----------------------|--|---------|--|----------------------------------------------------------------------------------------------------------------------------------------------------|
| `chunk` | REGCLASS | - |✔| Name of the chunk to add to the $COLUMNSTORE. |
| `if_not_columnstore` | BOOLEAN | `true` |✖| Set to `false` so this job fails with an error rather than a warning if `chunk` is already in the $COLUMNSTORE. |
| `recompress` | BOOLEAN | `false` |✖| Set to true to recompress. In-memory recompression will be attempted first; otherwise it will fall back to internal decompress/compress. |
| Name | Type | Default | Required | Description |
|----------------------|--|---------|--|---------------------------------------------------------------------------------------------------------------------------------|
| `chunk` | REGCLASS | - |✔| Name of the chunk to add to the $COLUMNSTORE. |
| `if_not_columnstore` | BOOLEAN | `true` |✖| Set to `false` so this job fails with an error rather than a warning if `chunk` is already in the $COLUMNSTORE. |
| `recompress` | BOOLEAN | `false` |✖| Set to true to recompress. In-memory recompression is attempted first; it falls back to internal decompress/compress. |

## Returns

Expand Down
6 changes: 3 additions & 3 deletions use-timescale/write-data/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ using `INSERT`, `UPDATE`, and `DELETE` statements.
* [Upsert data][upsert] into hypertables
* [Delete data][delete] from hypertables

For more information about using third-party tools to write data
into $TIMESCALE_DB, see the [Ingest data from other sources][ingest-data] section.
To find out how to add and sync data to your $SERVICE_SHORT from other sources, see
[Import and sync][ingest-data].

[about-writing-data]: /use-timescale/:currentVersion:/write-data/about-writing-data/
[delete]: /use-timescale/:currentVersion:/write-data/delete/
[ingest-data]: /use-timescale/:currentVersion:/ingest-data/
[ingest-data]: /migrate/:currentVersion:
[insert]: /use-timescale/:currentVersion:/write-data/insert/
[update]: /use-timescale/:currentVersion:/write-data/update/
[upsert]: /use-timescale/:currentVersion:/write-data/upsert/
139 changes: 101 additions & 38 deletions use-timescale/write-data/insert.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,25 @@
---
title: Insert data
excerpt: Insert single and multiple rows and return data in TimescaleDB with SQL
excerpt: Insert single and multiple rows and bulk load data into TimescaleDB with SQL
products: [cloud, mst, self_hosted]
keywords: [ingest]
tags: [insert, write, hypertables]
keywords: [ingest, bulk load]
tags: [insert, write, hypertables, copy]
---

import EarlyAccess2230 from "versionContent/_partials/_early_access_2_23_0.mdx";

# Insert data

Insert data into a hypertable with a standard [`INSERT`][postgres-insert] SQL
command.
You insert data into a $HYPERTABLE using the following standard SQL commands:

- `INSERT`: single rows or small batches
- `COPY`: bulk data loading

To improve performance, insert time series data directly to the columnstore using [direct compress][direct-compress].

## Insert a single row

To insert a single row into a hypertable, use the syntax `INSERT INTO ...
VALUES`. For example, to insert data into a hypertable named `conditions`:
To insert a single row into a $HYPERTABLE, use the syntax `INSERT INTO ... VALUES`:

```sql
INSERT INTO conditions(time, location, temperature, humidity)
Expand All @@ -25,11 +28,11 @@ INSERT INTO conditions(time, location, temperature, humidity)

## Insert multiple rows

You can also insert multiple rows into a hypertable using a single `INSERT`
call. This works even for thousands of rows at a time. This is more efficient
than inserting data row-by-row, and is recommended when possible.
A more efficient method to insert row-by-row is to insert multiple rows into a $HYPERTABLE using a single
`INSERT` call. This works even for thousands of rows at a time. $TIMESCALE_DB batches the rows by chunk, then writes to
each chunk in a single transaction.

Use the same syntax, separating rows with a comma:
You use the same syntax, separating rows with a comma:

```sql
INSERT INTO conditions
Expand All @@ -39,18 +42,13 @@ INSERT INTO conditions
(NOW(), 'garage', 77.0, 65.2);
```

<Highlight type="note">

You can insert multiple rows belonging to different
chunks within the same `INSERT` statement. Behind the scenes, $TIMESCALE_DB batches the rows by chunk, and writes to each chunk in a single
transaction.

</Highlight>
If you `INSERT` unsorted data, call [convert_to_columnstore('<chunk_name>', recompress => true)][convert_to_columnstore]
on the $CHUNK to reorder and optimize your data.

## Insert and return data

In the same `INSERT` command, you can return some or all of the inserted data by
adding a `RETURNING` clause. For example, to return all the inserted data, run:
You can return some or all of the inserted data by adding a `RETURNING` clause to the `INSERT` command. For example,
to return all the inserted data, run:

```sql
INSERT INTO conditions
Expand All @@ -67,36 +65,101 @@ time | location | temperature | humidity
(1 row)
```

## Direct compress on INSERT
If you `INSERT` unsorted data, call [convert_to_columnstore('<chunk_name>', recompress => true)][convert_to_columnstore]
on the $CHUNK to reorder and optimize your data.

This columnar format enables fast scanning and
aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the
$COLUMNSTORE conversion, $HYPERTABLE chunks are compressed by up to 98%, and organized for efficient, large-scale
queries.
## Bulk insert with COPY

To improve performance, you can compress data during `INSERT` so that it is injected directly into chunks
in the $COLUMNSTORE rather than waiting for the policy.
The `COPY` command is the most efficient way to load large amounts of data into a $HYPERTABLE. For
bulk data loading, `COPY` can be 2-3x faster or more than `INSERT`, especially when combined with
[direct compress][direct-compress].

To enable direct compress on INSERT, enable the following [GUC parameters][gucs]:
`COPY` supports loading from:

```sql
SET timescaledb.enable_compressed_insert = true;
SET timescaledb.enable_compressed_insert_sort_batches = true;
SET timescaledb.enable_compressed_insert_client_sorted = true;
```
- **CSV files**:

```sql
COPY conditions(time, location, temperature, humidity)
FROM '/path/to/data.csv'
WITH (FORMAT CSV, HEADER);
```

When you set `enable_compressed_insert_client_sorted` to `true`, you must ensure that data in the input
stream is sorted.
- **Standard input**

To load data from your application or script using standard input:

```sql
COPY conditions(time, location, temperature, humidity)
FROM STDIN
WITH (FORMAT CSV);
```

To signal the end of input, add `\.` on a new line.

- **Program output**

To load data generated by a program or script:

```sql
COPY conditions(time, location, temperature, humidity)
FROM PROGRAM 'generate_data.sh'
WITH (FORMAT CSV);
```

If you `COPY` unsorted data, call [convert_to_columnstore('<chunk_name>', recompress => true)][convert_to_columnstore]
on the $CHUNK to reorder and optimize your data.

## Improve performance with direct compress

<EarlyAccess2230 />

The columnar format in the $COLUMNSTORE enables fast scanning and aggregation, optimizing performance for
analytical workloads while also saving significant storage space. In the $COLUMNSTORE conversion, $HYPERTABLE chunks are
compressed by up to 98%, and organized for efficient, large-scale queries.

To improve performance, compress data during the `INSERT` and `COPY` operations so that it is injected
directly into chunks in the $COLUMNSTORE rather than waiting for the policy. Direct compress writes data in the
compressed format in memory, significantly reducing I/O and improving ingestion performance.

When you enable direct compress, ensure that your data is already sorted by the table's compression `order_by` columns.
Incorrectly sorted data results in poor compression and query performance.

- **Enable direct compress on `INSERT`**

Set the following [GUC parameters][gucs]:
```sql
SET timescaledb.enable_direct_compress_insert = true;
SET timescaledb.enable_direct_compress_insert_client_sorted = true;
```

- **Enable direct compress on `COPY`**

Set the following [GUC parameter][gucs]:

```sql
SET timescaledb.enable_direct_compress_copy = true;
SET timescaledb.enable_direct_compress_copy_client_sorted = true;
```

- **Optimal batch size**: best results with batches of 1,000 to 10,000 records
- **Cardinality**: high cardinality datasets do not compress well and may degrade query performance
- **Batch format**: the columnstore is optimized for 1,000 records per batch per segment
- **WAL efficiency**: compressed batches are written to WAL rather than individual tuples
- **Continuous aggregates**: not supported with direct compress
- **Unique constraints**: tables with unique constraints cannot use direct compress




[postgres-insert]: https://www.postgresql.org/docs/current/sql-insert.html
[postgres-copy]: https://www.postgresql.org/docs/current/sql-copy.html
[upsert]: /use-timescale/:currentVersion:/write-data/upsert/
[gucs]: /api/:currentVersion:/configuration/gucs/
[postgres-update]: https://www.postgresql.org/docs/current/sql-update.html
[hypertable-create-table]: /api/:currentVersion:/hypertable/create_table/
[add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/
[remove_columnstore_policy]: /api/:currentVersion:/hypercore/remove_columnstore_policy/
[create_table_arguments]: /api/:currentVersion:/hypertable/create_table/#arguments
[alter_job_samples]: /api/:currentVersion:/jobs-automation/alter_job/#samples
[convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/
[gucs]: /api/:currentVersion:/configuration/gucs/

[postgres-insert]: https://www.postgresql.org/docs/current/sql-insert.html
[direct-compress]: /use-timescale/:currentVersion:/write-data/insert/#improve-performance-with-direct-compress
Loading