add KB article explaining difference between optimize final and final

Author: Blargian

knowledgebase/optimize_final_vs_final.mdx

@@ -1,357 +1,54 @@
 ---
-date: 2023-10-25
-title: About Quotas and Query complexity
-tags: ['Managing Cloud']
-keywords: ['Quotas', 'Query Complexity']
-description: 'Quotas and Query Complexity are powerful ways to limit and restrict what users can do in ClickHouse. This KB article shows examples on how to apply these two different approaches.'
+date: 2025-07-20
+title: What is the difference between OPTIMIZE FINAL and FINAL?
+tags: ['Core Data Concepts']
+keywords: ['OPTIMIZE FINAL', 'FINAL']
+description: 'Discusses the differences between OPTIMIZE FINAL and FINAL, and when to use and avoid them.'
 ---
 
 {frontMatter.description}
 {/* truncate */}
 
-## About Quotas and Query complexity {#about-quotas-and-query-complexity}
+# What is the difference between `OPTIMIZE FINAL` and `FINAL`?
 
-[Quotas](https://clickhouse.com/docs/operations/quotas) and [query complexity](https://clickhouse.com/docs/operations/settings/query-complexity) are powerful ways to limit and restrict what users can do in ClickHouse.
+`OPTIMIZE FINAL` is a DDL command that physically and permanently reorganizes
+and optimizes data on disk. It physically merges data parts in `MergeTree` tables,
+performing data deduplication in the process by removing duplicate rows from storage.
 
-Quotas do apply restrictions within the context of a time interval, while query complexity applies regardless of time intervals.
+`FINAL` is a **query-time** modifier that provides deduplicated results without
+changing the structure of the stored data. It works by performing merge logic at
+read-time. It is temporary, only affecting the current query result.
 
-This KB article shows examples on how to apply these two different approaches.
+Users are often advised to avoid using `OPTIMIZE FINAL`, as it has a significant
+performance overhead, however they should not confuse the two. It is often necessary
+to use `FINAL` to get back results without duplicates, especially when using table
+engines like `ReplacingMergeTree` which may contain duplicate rows which have not
+been replaced during the eventual, background merge process.
 
-## The sample data {#the-sample-data}
+The table below summarises the key differences:
 
-We refer to this simple sample table for the purpose of these examples:
+|Aspect	           |`OPTIMIZE FINAL`                            | `FINAL`                                            |
+|------------------|--------------------------------------------|----------------------------------------------------|
+|Type              | DDL Command                                | Query Modifier                                     |
+|Effect            | Permanent storage optimization	            | Temporary query-time deduplication                 |
+|Performance       | Impact	High cost once, then faster queries	| Lower individual cost, but repeated for each query |
+|Data Modification | Yes - physically changes storage           | No - read-only operation                           |
+|Use Case          | Periodic maintenance/optimization          | Real-time deduplicated queries                     |
 
-```sql
-clickhouse-cloud :) CREATE TABLE default.test_table (name String, age UInt8) ENGINE=MergeTree ORDER BY tuple();
+## When to use each {#when-to-use-each}
 
--- CREATE TABLE default.test_table
--- (
---     `name` String,
---     `age` UInt8
--- )
--- ENGINE = MergeTree
--- ORDER BY tuple()
+Use `OPTIMIZE FINAL` when:
 
--- Query id: 4fd405db-a96e-4004-b1f6-e7f87def05d7
+- You want to permanently improve query performance
+- You can afford the one-time optimization cost
+- You're doing periodic table maintenance
+- You want to physically clean up duplicate data
 
--- Ok.
+Use `FINAL` when:
 
--- 0 rows in set. Elapsed: 0.313 sec.
-
-clickhouse-cloud :) INSERT INTO default.test_table SELECT * FROM generateRandom('name String, age UInt8',1,1) LIMIT 100;
-
--- INSERT INTO default.test_table SELECT *
--- FROM generateRandom('name String, age UInt8', 1, 1)
--- LIMIT 100
-
--- Query id: 6eccfdc6-d98c-4377-ae25-f18deec6c807
-
--- Ok.
-
--- 0 rows in set. Elapsed: 0.055 sec.
-
-clickhouse-cloud :) SELECT * FROM default.test_table_00006488 LIMIT 5
-
--- SELECT *
--- FROM default.test_table_00006488
--- LIMIT 5
-
--- Query id: 9fa58419-fb57-4260-886a-ccb836449f58
-
--- ┌─name─┬─age─┐
--- │      │ 200 │
--- │ 4    │  72 │
--- │ +    │ 127 │
--- │      │ 144 │
--- │ ]    │  60 │
--- └──────┴─────┘
-
--- 5 rows in set. Elapsed: 0.003 sec.
-```
-
-## Using Quotas {#using-quotas}
-
-In this example we create a role to which we'll apply a Quota that allows only 10 result rows to be retrieved for each 10 seconds interval:
-
-```sql
-# AS the privileged user
-
-# create a user
-clickhouse-cloud :) CREATE USER user_with_quota IDENTIFIED WITH sha256_password BY 'Dr6P1S8SGaQ@u!BUAnv';
-
--- CREATE USER user_with_quota IDENTIFIED WITH sha256_hash BY '2444E98ADA7433FC12F55C467D3564BF87F47B1A996E70D77496A2F1E42BAD73' SALT '129F92F8AB4AB6E56A01AA826D10D1239F14148606E197EB19D7612F8AF8BC52'
-
--- Query id: 542a4013-e34c-4776-b374-962fcfd2575a
-
--- Ok.
-
--- 0 rows in set. Elapsed: 0.097 sec.
-
-# create a role to which quotas will be applied
-clickhouse-cloud :) CREATE ROLE role_with_quota
-
--- CREATE ROLE role_with_quota
-
--- Query id: 133a843b-8619-4642-84d9-9c232539b6a0
-
--- Ok.
-
--- 0 rows in set. Elapsed: 0.096 sec.
-
-
--- grant select privileges
-clickhouse-cloud :) GRANT SELECT ON default.* TO role_with_quota;
-
--- GRANT SELECT ON default.* TO role_with_quota
-
--- Query id: 1b0e295e-597d-477f-8847-13411157fd1c
-
--- Ok.
-
--- 0 rows in set. Elapsed: 0.100 sec.
-
-
--- grant role to the user
-clickhouse-cloud :) GRANT role_with_quota TO user_with_quota
-
--- GRANT role_with_quota TO user_with_quota
-
--- Query id: 0e19ff50-8990-4c17-8f91-5c8ce4142bdd
-
--- Ok.
-
--- 0 rows in set. Elapsed: 0.099 sec.
-
-
--- create a quota that allows max 10 result rows in each 10 seconds interval and apply that to the role
-clickhouse-cloud :) CREATE QUOTA quota_max_10_result_rows_per_10_seconds FOR INTERVAL 10 second MAX result_rows = 10 TO role_with_quota
-
--- CREATE QUOTA quota_max_10_result_rows_per_10_seconds FOR INTERVAL 10 second MAX result_rows = 10 TO role_with_quota
-
--- 0 rows in set. Elapsed: 23.427 sec.
-
--- Query id: fe4d2038-2d35-415d-89ec-9eaaa2533fcd
-```
-
-Now login as the user `user_with_quota`
-
-```sql
--- login as the user where quota is applied through the role
-clickhouse-cloud :) SELECT user()
-
--- SELECT user()
-
--- Query id: 56ebd28d-0d36-4caf-9cef-c3e51d9f0b9d
-
--- ┌─currentUser()───┐
--- │ user_with_quota │
--- └─────────────────┘
-
--- 1 row in set. Elapsed: 0.002 sec.
-
-
--- list grants
-clickhouse-cloud :) SHOW GRANTS
-
--- SHOW GRANTS
-
--- Query id: cc78bada-28f4-4862-9fdf-7e68aae6fd80
-
--- ┌─GRANTS───────────────────────────────────┐
--- │ GRANT role_with_quota TO user_with_quota │
--- └──────────────────────────────────────────┘
-
--- 1 row in set. Elapsed: 0.001 sec.
-
--- check the timem
-clickhouse-cloud :) select now()
-
--- SELECT now()
-
--- Query id: bbbd54a8-6c2f-4d3b-982a-03d7bd143aa9
-
--- ┌───────────────now()─┐
--- │ 2023-10-25 14:37:38 │
--- └─────────────────────┘
-
--- 1 row in set. Elapsed: 0.001 sec.
-
-
--- query ten rows
-clickhouse-cloud :) SELECT * FROM test_table LIMIT 10
-
--- SELECT *
--- FROM test_table
--- LIMIT 10
-
--- Query id: 20f1c02f-c938-4d06-851d-824c82693eb9
-
--- ┌─name─┬─age─┐
--- │      │ 200 │
--- │ 4    │  72 │
--- │ +    │ 127 │
--- │      │ 144 │
--- │ ]    │  60 │
--- │      │ 137 │
--- │      │ 176 │
--- │      │ 147 │
--- │      │ 107 │
--- │ Q    │ 128 │
--- └──────┴─────┘
-
--- 10 rows in set. Elapsed: 0.002 sec.
-
--- attempt to get another row within the 10 seconds interval since the last query
-clickhouse-cloud :) SELECT * FROM test_table LIMIT 1
-
--- SELECT *
--- FROM test_table
--- LIMIT 1
-
--- Query id: 48ae46ef-7b33-4765-affa-e47e889f48e5
-
-
--- 0 rows in set. Elapsed: 0.094 sec.
-
--- Received exception from server (version 23.8.1):
--- Code: 201. DB::Exception: Received from dxqjx1s5lt.eu-west-1.aws.clickhouse.cloud:9440. DB::Exception: Quota for user `user_with_quota` for 10s has been exceeded: result_rows = 11/10. 
--- Interval will end at 2023-10-25 14:37:50. Name of quota template: `quota_max_10_result_rows_per_10_seconds`. (QUOTA_EXCEEDED)
-
-
--- check the time
-clickhouse-cloud :) select now()
-
--- SELECT now()
-
--- Query id: 87f190f6-3f75-4fe6-bf9c-c80ed88e179f
-
--- ┌───────────────now()─┐
--- │ 2023-10-25 14:37:45 │
--- └─────────────────────┘
-
--- 1 row in set. Elapsed: 0.001 sec.
-```
-
-Note that the user will need to wait another 5 seconds before can get a new 10 rows resultset "allowance".
-
-
-## Using Query Complexity {#using-query-complexity}
-
-In this example we create a role to which we'll apply a Query Complexity `SETTING` that allows only 1 rows to be returned for each query.
-
-```sql
--- AS the privileged user
--- create a user
-clickhouse-cloud :) CREATE USER user_with_query_complexity IDENTIFIED WITH sha256_password BY 'Dr6P1S8SGaQ@u!BUAnv';
-
--- CREATE USER user_with_query_complexity IDENTIFIED WITH sha256_hash BY '99AB4976077304554286C43AA47C3BEDA5758EF56282C2FC90C0787DC6FE72BC' SALT '5A50D2B9B1DF7E8A1AA9A2CC00BCF802B7F605281A09E18E237447509B5C7A7C'
-
--- Query id: 91856182-f2bb-40cc-8902-2786beeeb93d
-
--- Ok.
-
--- 0 rows in set. Elapsed: 0.104 sec.
-
-
--- create a role with query complexity SETTINGS that allows only one role in resultset
-clickhouse-cloud :) CREATE ROLE role_with_query_complexity SETTINGS max_result_rows=1;
-
--- CREATE ROLE role_with_query_complexity SETTINGS max_result_rows = 1
-
--- Query id: ec3d89fe-cab8-4cc3-9180-da5c93519643
-
--- Ok.
-
--- 0 rows in set. Elapsed: 0.097 sec.
-
-
--- grant select privileges
-clickhouse-cloud :) GRANT SELECT ON default.* TO role_with_query_complexity;
-
--- GRANT SELECT ON default.* TO role_with_query_complexity
-
--- Query id: 230774ad-8073-4e2e-9530-3e90bce41cb1
-
--- Ok.
-
--- 0 rows in set. Elapsed: 0.097 sec.
-
-
--- grant role to the user
-clickhouse-cloud :) GRANT role_with_query_complexity TO user_with_query_complexity
-
--- GRANT role_with_query_complexity TO user_with_query_complexity
-
--- Query id: f28c7c7b-61f7-48a8-a281-1f3784764b47
-
--- Ok.
-
--- 0 rows in set. Elapsed: 0.096 sec.
-```
-
-
-Now login as the user `user_with_query_complexity`:
-
-```sql
-
--- login as the user where query complexity is applied through the role
-clickhouse-cloud :) SELECT user();
-
--- SELECT user()
-
--- Query id: 196c91fc-abff-464d-acce-6af961c233a3
-
--- ┌─currentUser()──────────────┐
--- │ user_with_query_complexity │
--- └────────────────────────────┘
-
--- 1 row in set. Elapsed: 0.001 sec.
-
-
--- list grants
-clickhouse-cloud :) SHOW GRANTS
-
--- SHOW GRANTS
-
--- Query id: 87657b99-c3d9-4ffd-90e8-488f04f7f93b
-
--- ┌─GRANTS─────────────────────────────────────────────────────────┐
--- │ GRANT role_with_query_complexity TO user_with_query_complexity │
--- └────────────────────────────────────────────────────────────────┘
-
--- 1 row in set. Elapsed: 0.001 sec.
-
--- attempt to query with 1 row in resultset
-clickhouse-cloud :) SELECT * FROM default.test_table LIMIT 1;
-
--- SELECT *
--- FROM default.test_table
--- LIMIT 1
-
--- Query id: 7266891b-8611-4342-81b0-fe04766e62fa
-
--- ┌─name─┬─age─┐
--- │      │ 200 │
--- └──────┴─────┘
-
--- 1 row in set. Elapsed: 0.002 sec.
-
-
--- attempt to query with more than 1 row in resultset
-clickhouse-cloud :) SELECT * FROM default.test_table LIMIT 2;
-
--- SELECT *
--- FROM default.test_table
--- LIMIT 2
-
--- Query id: ec8ecff3-f731-45bd-bb27-894ba358c7c8
-
--- 0 rows in set. Elapsed: 0.091 sec.
-
---Received exception from server (version 23.8.1):
---Code: 396. DB::Exception: Received from dxqjx1s5lt.eu-west-1.aws.clickhouse.cloud:9440. 
---DB::Exception: Limit for result exceeded, max rows: 1.00, current rows: 2.00. (TOO_MANY_ROWS_OR_BYTES)
-```
-
-Whenever attempting to get more than 1 row in resultset the query complexity constraint will kick in.
+- You need deduplicated results immediately
+- You can't wait for or don't want permanent optimization
+- You only occasionally need deduplicated data
+- You're working with frequently changing data
 
+Both are valuable tools, but they serve different purposes in ClickHouse's deduplication strategy.