You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/developer/lightweight-update.md
+15-14Lines changed: 15 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,16 +1,16 @@
1
1
---
2
2
slug: /guides/developer/lightweight-update
3
-
sidebar_label: 'Lightweight Update'
4
-
title: 'Lightweight Update'
5
-
keywords: ['lightweight update']
6
-
description: 'Provides a description of lightweight updates'
3
+
sidebar_label: 'On-the-fly mutation'
4
+
title: 'On-the-fly Mutations'
5
+
keywords: ['On-the-fly mutation']
6
+
description: 'Provides a description of on-the-fly mutations'
7
7
---
8
8
9
-
## Lightweight Update {#lightweight-update}
9
+
## On-the-fly mutations {#lightweight-update}
10
10
11
-
When lightweight updates are enabled, updated rows are marked as updated immediately and subsequent `SELECT` queries will automatically return with the changed values. When lightweight updates are not enabled, you may have to wait for your mutations to be applied via a background process to see the changed values.
11
+
When on-the-fly mutations are enabled, updated rows are marked as updated immediately and subsequent `SELECT` queries will automatically return with the changed values. When on-the-fly mutations are not enabled, you may have to wait for your mutations to be applied via a background process to see the changed values.
12
12
13
-
Lightweight updates can be enabled for `MergeTree`-family tables by enabling the query-level setting `apply_mutations_on_fly`.
13
+
On-the-fly mutations can be enabled for `MergeTree`-family tables by enabling the query-level setting `apply_mutations_on_fly`.
14
14
15
15
```sql
16
16
SET apply_mutations_on_fly =1;
@@ -24,7 +24,7 @@ CREATE TABLE test_on_fly_mutations (id UInt64, v String)
24
24
ENGINE = MergeTree ORDER BY id;
25
25
26
26
-- Disable background materialization of mutations to showcase
27
-
-- default behavior when lightweight updates are not enabled
27
+
-- default behavior when on-the-fly mutations are not enabled
28
28
SYSTEM STOP MERGES test_on_fly_mutations;
29
29
SET mutations_sync =0;
30
30
@@ -39,8 +39,9 @@ ALTER TABLE test_on_fly_mutations DELETE WHERE v = 'e';
39
39
```
40
40
41
41
Let's check the result of the updates via a `SELECT` query:
42
+
42
43
```sql
43
-
-- Explicitly disable lightweight updates
44
+
-- Explicitly disable on-the-fly-mutations
44
45
SET apply_mutations_on_fly =0;
45
46
46
47
SELECT id, v FROM test_on_fly_mutations ORDER BY id;
@@ -56,10 +57,10 @@ Note that the values of the rows have not yet been updated when we query the new
56
57
└────┴───┘
57
58
```
58
59
59
-
Let's now see what happens when we enable lightweight updates:
60
+
Let's now see what happens when we enable on-the-fly mutations:
60
61
61
62
```sql
62
-
-- Enable lightweight updates
63
+
-- Enable on-the-fly mutations
63
64
SET apply_mutations_on_fly =1;
64
65
65
66
SELECT id, v FROM test_on_fly_mutations ORDER BY id;
@@ -73,17 +74,17 @@ The `SELECT` query now returns the correct result immediately, without having to
73
74
└────┴───┘
74
75
```
75
76
76
-
## Performance Impact {#performance-impact}
77
+
## Performance impact {#performance-impact}
77
78
78
-
When lightweight updates are enabled, mutations are not materialized immediately but will only be applied during `SELECT` queries. However, please note that mutations are still being materialized asynchronously in the background, which is a heavy process.
79
+
When on-the-fly mutations are enabled, mutations are not materialized immediately but will only be applied during `SELECT` queries. However, please note that mutations are still being materialized asynchronously in the background, which is a heavy process.
79
80
80
81
If the number of submitted mutations constantly exceeds the number of mutations that are processed in the background over some time interval, the queue of unmaterialized mutations that have to be applied will continue to grow. This will result in the eventual degradation of `SELECT` query performance.
81
82
82
83
We suggest enabling the setting `apply_mutations_on_fly` together with other `MergeTree`-level settings such as `number_of_mutations_to_throw` and `number_of_mutations_to_delay` to restrict the infinite growth of unmaterialized mutations.
83
84
84
85
## Support for subqueries and non-deterministic functions {#support-for-subqueries-and-non-deterministic-functions}
85
86
86
-
Lightweight updates have limited support with subqueries and non-deterministic functions. Only scalar subqueries with a result that have a reasonable size (controlled by the setting `mutations_max_literal_size_to_replace`) are supported. Only constant non-deterministic functions are supported (e.g. the function `now()`).
87
+
On-the-fly mutations have limited support with subqueries and non-deterministic functions. Only scalar subqueries with a result that have a reasonable size (controlled by the setting `mutations_max_literal_size_to_replace`) are supported. Only constant non-deterministic functions are supported (e.g. the function `now()`).
87
88
88
89
These behaviours are controlled by the following settings:
Copy file name to clipboardExpand all lines: docs/managing-data/updating-data/overview.md
+19-8Lines changed: 19 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,14 +18,13 @@ There are several ways to update data in ClickHouse, each with its own advantage
18
18
For both operations, if the number of submitted mutations constantly exceeds the number of mutations that are processed in the background over some time interval, the queue of non-materialized mutations that have to be applied will continue to grow. This will result in the eventual degradation of `SELECT` query performance.
19
19
20
20
In summary, update operations should be issued carefully, and the mutations queue should be tracked closely using the `system.mutations` table. Do not issue updates frequently as you would in OLTP databases. If you have a requirement for frequent updates, see [ReplacingMergeTree](/engines/table-engines/mergetree-family/replacingmergetree).
|[Update mutation](/sql-reference/statements/alter/update)|`ALTER TABLE [table] UPDATE`| Use when data must be updated to disk immediately (e.g. for compliance). Negatively affects `SELECT` performance. |
25
-
|[Lightweight update](/guides/developer/lightweight-update)|`ALTER TABLE [table] UPDATE`| Enable using `SET apply_mutations_on_fly = 1;`. Use when updating small amounts of data. Rows are immediately returned with updated data in all subsequent `SELECT` queries but are initially only internally marked as updated on disk. |
24
+
|[Lightweight updates](/sql-reference/statements/update)|`UPDATE [table] SET ... WHERE`| Use for updating small amounts of data (up to ~10% of table). Creates patch parts for immediate visibility without rewriting entire columns. Adds overhead to `SELECT` queries but has predictable latency. Currently experimental. |
25
+
|[On-the-fly updates](/guides/developer/lightweight-update)|`ALTER TABLE [table] UPDATE`| Enable using `SET apply_mutations_on_fly = 1;`. Use when updating small amounts of data. Rows are immediately returned with updated data in all subsequent `SELECT` queries but are initially only internally marked as updated on disk. |
26
26
|[ReplacingMergeTree](/engines/table-engines/mergetree-family/replacingmergetree)|`ENGINE = ReplacingMergeTree`| Use when updating large amounts of data. This table engine is optimized for data deduplication on merges. |
27
27
|[CollapsingMergeTree](/engines/table-engines/mergetree-family/collapsingmergetree)|`ENGINE = CollapsingMergeTree(Sign)`| Use when updating individual rows frequently, or for scenarios where you need to maintain the latest state of objects that change over time. For example, tracking user activity or article stats. |
28
-
29
28
Here is a summary of the different ways to update data in ClickHouse:
30
29
31
30
## Update Mutations {#update-mutations}
@@ -40,9 +39,21 @@ These are extremely IO-heavy, rewriting all the parts that match the `WHERE` exp
40
39
41
40
Read more about [update mutations](/sql-reference/statements/alter/update).
42
41
43
-
## Lightweight Updates {#lightweight-updates}
42
+
## Lightweight updates {#lightweight-updates}
43
+
44
+
Lightweight updates are a ClickHouse feature that updates rows using "patch parts" - special data parts containing only the updated columns and rows, rather than rewriting entire columns like traditional mutations. The Lightweight UPDATE
45
+
Key characteristics:
46
+
47
+
- Uses the standard `UPDATE` syntax and creates patch parts immediately without waiting for merges
48
+
- Updated values are immediately visible in `SELECT` queries through patch application, but physically materialized only during subsequent merges
49
+
- Designed for small updates (up to ~10% of table) with predictable latency
50
+
- Adds overhead to `SELECT` queries that need to apply patches, but avoids rewriting entire columns
51
+
52
+
For more details see ["The Lightweight UPDATE Statement"](/sql-reference/statements/update)
53
+
54
+
## On-the-fly Updates {#on-the-fly-updates}
44
55
45
-
Lightweight updates provide a mechanism to update rows such that they are updated immediately, and subsequent `SELECT` queries will automatically return with the changed values (this incurs an overhead and will slow queries). This effectively addresses the atomicity limitation of normal mutations. We show an example below:
56
+
On-the-fly updates provide a mechanism to update rows such that they are updated immediately, and subsequent `SELECT` queries will automatically return with the changed values (this incurs an overhead and will slow queries). This effectively addresses the atomicity limitation of normal mutations. We show an example below:
46
57
47
58
```sql
48
59
SET apply_mutations_on_fly =1;
@@ -52,7 +63,7 @@ FROM posts
52
63
WHERE Id =404346
53
64
54
65
┌─ViewCount─┐
55
-
│ 26762 │
66
+
│ 26762 │
56
67
└───────────┘
57
68
58
69
1 row inset. Elapsed: 0.115 sec. Processed 59.55 million rows, 238.25 MB (517.83 million rows/s., 2.07 GB/s.)
@@ -73,9 +84,9 @@ WHERE Id = 404346
73
84
1 row inset. Elapsed: 0.149 sec. Processed 59.55 million rows, 259.91 MB (399.99 million rows/s., 1.75 GB/s.)
74
85
```
75
86
76
-
Note that for lightweight updates, a mutation is still used to update the data; it is just not materialized immediately and applied during `SELECT` queries. It will still be applied in the background as an asynchronous process and incurs the same heavy overhead as a mutation and thus is an I/O intense operation that should be used sparingly. The expressions that can be used with this operation are also limited (see here for [details](/guides/developer/lightweight-update#support-for-subqueries-and-non-deterministic-functions)).
87
+
Note that for on-the-fly updates, a mutation is still used to update the data; it is just not materialized immediately and applied during `SELECT` queries. It will still be applied in the background as an asynchronous process and incurs the same heavy overhead as a mutation and thus is an I/O intense operation that should be used sparingly. The expressions that can be used with this operation are also limited (see here for [details](/guides/developer/lightweight-update#support-for-subqueries-and-non-deterministic-functions)).
77
88
78
-
Read more about [lightweight updates](/guides/developer/lightweight-update).
89
+
Read more about [on-the-fly updates](/guides/developer/lightweight-update).
0 commit comments