You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Performance tuning with ordered clustered columnstore index
2
+
title: Performance tuning with ordered clustered columnstore index
3
3
description: Recommendations and considerations you should know as you use ordered clustered columnstore index to improve your query performance in dedicated SQL pools.
4
4
author: XiaoyuMSFT
5
+
ms.author: xiaoyul
5
6
manager: craigg
7
+
ms.reviewer: nibruno; wiassaf
8
+
ms.date: 10/17/2022
6
9
ms.service: synapse-analytics
10
+
ms.subservice: sql-dw
7
11
ms.topic: conceptual
8
-
ms.subservice: sql-dw
9
-
ms.date: 07/14/2022
10
-
ms.author: xiaoyul
11
-
ms.reviewer: nibruno; wiassaf
12
-
ms.custom: seo-lt-2019, azure-synapse
12
+
ms.custom:
13
+
- seo-lt-2019
14
+
- azure-synapse
13
15
---
14
16
15
-
# Performance tuning with ordered clustered columnstore index
17
+
# Performance tuning with ordered clustered columnstore index
16
18
17
19
**Applies to:** Azure Synapse Analytics dedicated SQL pools, SQL Server 2022 (16.x) and later
18
20
19
-
When users query a columnstore table in dedicated SQL pool, the optimizer checks the minimum and maximum values stored in each segment. Segments that are outside the bounds of the query predicate aren't read from disk to memory. A query can finish faster if the number of segments to read and their total size are small.
21
+
When users query a columnstore table in dedicated SQL pool, the optimizer checks the minimum and maximum values stored in each segment. Segments that are outside the bounds of the query predicate aren't read from disk to memory. A query can finish faster if the number of segments to read and their total size are small.
20
22
21
23
## Ordered vs. non-ordered clustered columnstore index
22
24
23
-
By default, for each table created without an index option, an internal component (index builder) creates a non-ordered clustered columnstore index (CCI) on it. Data in each column is compressed into a separate CCI rowgroup segment. There's metadata on each segment's value range, so segments that are outside the bounds of the query predicate aren't read from disk during query execution. CCI offers the highest level of data compression and reduces the size of segments to read so queries can run faster. However, because the index builder doesn't sort data before compressing them into segments, segments with overlapping value ranges could occur, causing queries to read more segments from disk and take longer to finish.
25
+
By default, for each table created without an index option, an internal component (index builder) creates a non-ordered clustered columnstore index (CCI) on it. Data in each column is compressed into a separate CCI rowgroup segment. There's metadata on each segment's value range, so segments that are outside the bounds of the query predicate aren't read from disk during query execution. CCI offers the highest level of data compression and reduces the size of segments to read so queries can run faster. However, because the index builder doesn't sort data before compressing them into segments, segments with overlapping value ranges could occur, causing queries to read more segments from disk and take longer to finish.
24
26
25
-
When creating an ordered CCI, the dedicated SQL pool engine sorts the existing data in memory by the order key(s) before the index builder compresses them into index segments. With sorted data, segment overlapping is reduced allowing queries to have a more efficient segment elimination and thus faster performance because the number of segments to read from disk is smaller. If all data can be sorted in memory at once, then segment overlapping can be avoided. Due to large tables in data warehouses, this scenario doesn't happen often.
27
+
When creating an ordered CCI, the dedicated SQL pool engine sorts the existing data in memory by the order key(s) before the index builder compresses them into index segments. With sorted data, segment overlapping is reduced allowing queries to have a more efficient segment elimination and thus faster performance because the number of segments to read from disk is smaller. If all data can be sorted in memory at once, then segment overlapping can be avoided. Due to large tables in data warehouses, this scenario doesn't happen often.
26
28
27
29
To check the segment ranges for a column, run the following command with your table name and column name:
@@ -43,28 +45,28 @@ WHERE o.name = '<Table Name>' and cols.name = '<Column Name>' and TMap.physical
43
45
ORDER BYo.name, pnp.distribution_id, cls.min_data_id;
44
46
```
45
47
46
-
> [!NOTE]
47
-
> In an ordered CCI table, the new data resulting from the same batch of DML or data loading operations are sorted within that batch, there is no global sorting across all data in the table. Users can REBUILD the ordered CCI to sort all data in the table. In dedicated SQL pool, the columnstore index REBUILD is an offline operation. For a partitioned table, the REBUILD is done one partition at a time. Data in the partition that is being rebuilt is "offline" and unavailable until the REBUILD is complete for that partition.
48
+
> [!NOTE]
49
+
> In an ordered CCI table, the new data resulting from the same batch of DML or data loading operations are sorted within that batch, there is no global sorting across all data in the table. Users can REBUILD the ordered CCI to sort all data in the table. In dedicated SQL pool, the columnstore index REBUILD is an offline operation. For a partitioned table, the REBUILD is done one partition at a time. Data in the partition that is being rebuilt is "offline" and unavailable until the REBUILD is complete for that partition.
48
50
49
51
## Query performance
50
52
51
53
A query's performance gain from an ordered CCI depends on the query patterns, the size of data, how well the data is sorted, the physical structure of segments, and the DWU and resource class chosen for the query execution. Users should review all these factors before choosing the ordering columns when designing an ordered CCI table.
52
54
53
55
Queries with all these patterns typically run faster with ordered CCI.
54
56
1. The queries have equality, inequality, or range predicates
55
-
1. The predicate columns and the ordered CCI columns are the same.
56
-
57
+
1. The predicate columns and the ordered CCI columns are the same.
58
+
57
59
In this example, table T1 has a clustered columnstore index ordered in the sequence of Col_C, Col_B, and Col_A.
58
60
59
61
```sql
60
62
CREATE CLUSTERED COLUMNSTORE INDEX MyOrderedCCI ON T1
61
63
ORDER (Col_C, Col_B, Col_A);
62
64
```
63
65
64
-
The performance of query 1 and query 2 can benefit most from ordered CCI than the other queries as they reference all the ordered CCI columns.
66
+
The performance of query 1 and query 2 can benefit more from ordered CCI than the other queries, as they reference all the ordered CCI columns.
65
67
66
68
```sql
67
-
-- Query #1:
69
+
-- Query #1:
68
70
69
71
SELECT*FROM T1 WHERE Col_C ='c'AND Col_B ='b'AND Col_A ='a';
70
72
@@ -77,55 +79,55 @@ SELECT * FROM T1 WHERE Col_B = 'b' AND Col_A = 'a';
77
79
78
80
-- Query #4
79
81
SELECT*FROM T1 WHERE Col_A ='a'AND Col_C ='c';
80
-
81
82
```
82
83
83
84
## Data loading performance
84
85
85
-
The performance of data loading into an ordered CCI table is similar to a partitioned table. Loading data into an ordered CCI table can take longer than a non-ordered CCI table because of the data sorting operation, however queries can run faster afterwards with ordered CCI.
86
+
The performance of data loading into an ordered CCI table is similar to a partitioned table. Loading data into an ordered CCI table can take longer than a non-ordered CCI table because of the data sorting operation, however queries can run faster afterwards with ordered CCI.
86
87
87
-
Here is an example performance comparison of loading data into tables with different schemas.
88
+
Here's an example performance comparison of loading data into tables with different schemas.
88
89
89
-
90
+
:::image type="content" source="./media/performance-tuning-ordered-cci/cci-data-loading-performance.png" alt-text="Bar graph that shows the performance comparison of loading data into tables with different schemas.":::
90
91
92
+
Here's an example query performance comparison between CCI and ordered CCI.
91
93
92
-
Here is an example query performance comparison between CCI and ordered CCI.
94
+
:::image type="content" source="./media/performance-tuning-ordered-cci/occi_query_performance.png" alt-text="Bar graph comparing performance during data_loading. An ordered clustered columnstore index has lower duration.":::
The number of overlapping segments depends on the size of data to sort, the available memory, and the maximum degree of parallelism (MAXDOP) setting during ordered CCI creation. Below are options to reduce segment overlapping when creating ordered CCI.
100
99
101
-
- Use `xlargerc` resource class on a higher DWU to allow more memory for data sorting before the index builder compresses the data into segments. Once in an index segment, the physical location of the data cannot be changed. There's no data sorting within a segment or across segments.
100
+
- Use `xlargerc` resource class on a higher DWU to allow more memory for data sorting before the index builder compresses the data into segments. Once in an index segment, the physical location of the data cannot be changed. There's no data sorting within a segment or across segments.
102
101
103
-
- Create ordered CCI with `OPTION (MAXDOP = 1)`. Each thread used for ordered CCI creation works on a subset of data and sorts it locally. There's no global sorting across data sorted by different threads. Using parallel threads can reduce the time to create an ordered CCI but will generate more overlapping segments than using a single thread. Currently, the MAXDOP option is only supported in creating an ordered CCI table using CREATE TABLE AS SELECT command. Creating an ordered CCI via CREATE INDEX or CREATE TABLE commands does not support the MAXDOP option. For example:
102
+
- Create ordered CCI with `OPTION (MAXDOP = 1)`. Each thread used for ordered CCI creation works on a subset of data and sorts it locally. There's no global sorting across data sorted by different threads. Using parallel threads can reduce the time to create an ordered CCI but will generate more overlapping segments than using a single thread. Using a single threaded operation delivers the highest compression quality. For example:
104
103
105
104
```sql
106
105
CREATETABLETable1 WITH (DISTRIBUTION = HASH(c1), CLUSTERED COLUMNSTORE INDEX ORDER(c1) )
107
106
ASSELECT*FROM ExampleTable
108
107
OPTION (MAXDOP 1);
109
108
```
110
109
110
+
> [!NOTE]
111
+
> Currently, in dedicated SQL pools in Azure Synapse Analytics, the MAXDOP option is only supported in creating an ordered CCI table using `CREATE TABLE AS SELECT` command. Creating an ordered CCI via `CREATE INDEX` or `CREATE TABLE` commands doesn't support the MAXDOP option. This limitation does not apply to SQL Server 2022 and later versions, where you can specify MAXDOP with the `CREATE INDEX` or `CREATE TABLE` commands.
112
+
111
113
- Pre-sort the data by the sort key(s) before loading them into tables.
112
114
113
-
Here is an example of an ordered CCI table distribution that has zero segment overlapping following above recommendations. The ordered CCI table is created in a DWU1000c database via CTAS from a 20-GB heap table using MAXDOP 1 and `xlargerc`. The CCI is ordered on a BIGINT column with no duplicates.
115
+
Here's an example of an ordered CCI table distribution that has zero segment overlapping following above recommendations. The ordered CCI table is created in a DWU1000c database via CTAS from a 20-GB heap table using MAXDOP 1 and `xlargerc`. The CCI is ordered on a BIGINT column with no duplicates.
:::image type="content" source="./media/performance-tuning-ordered-cci/perfect-sorting-example.png" alt-text="A screenshot of text data showing no segment overlapping.":::
116
118
117
119
## Create ordered CCI on large tables
118
120
119
-
Creating an ordered CCI is an offline operation. For tables with no partitions, the data won't be accessible to users until the ordered CCI creation process completes. For partitioned tables, since the engine creates the ordered CCI partition by partition, users can still access the data in partitions where ordered CCI creation isn't in process. You can use this option to minimize the downtime during ordered CCI creation on large tables:
121
+
Creating an ordered CCI is an offline operation. For tables with no partitions, the data won't be accessible to users until the ordered CCI creation process completes. For partitioned tables, since the engine creates the ordered CCI partition by partition, users can still access the data in partitions where ordered CCI creation isn't in process. You can use this option to minimize the downtime during ordered CCI creation on large tables:
120
122
121
-
1.Create partitions on the target large table (called `Table_A`).
122
-
2. Create an empty ordered CCI table (called `Table_B`) with the same table and partition schema as `Table_A`.
123
-
3. Switch one partition from `Table_A` to `Table_B`.
124
-
4. Run `ALTER INDEX <Ordered_CCI_Index> ON <Table_B> REBUILD PARTITION = <Partition_ID>` to rebuild the switched-in partition on `Table_B`.
125
-
5. Repeat step 3 and 4 for each partition in `Table_A`.
126
-
6.Once all partitions are switched from `Table_A` to `Table_B` and have been rebuilt, drop `Table_A`, and rename `Table_B` to `Table_A`.
123
+
1. Create partitions on the target large table (called `Table_A`).
124
+
1. Create an empty ordered CCI table (called `Table_B`) with the same table and partition schema as `Table_A`.
125
+
1. Switch one partition from `Table_A` to `Table_B`.
126
+
1. Run `ALTER INDEX <Ordered_CCI_Index> ON <Table_B> REBUILD PARTITION = <Partition_ID>` to rebuild the switched-in partition on `Table_B`.
127
+
1. Repeat step 3 and 4 for each partition in `Table_A`.
128
+
1.Once all partitions are switched from `Table_A` to `Table_B` and have been rebuilt, drop `Table_A`, and rename `Table_B` to `Table_A`.
127
129
128
-
>[!TIP]
130
+
>[!TIP]
129
131
> For a dedicated SQL pool table with an ordered CCI, ALTER INDEX REBUILD will re-sort the data using `tempdb`. Monitor `tempdb` during rebuild operations. If you need more `tempdb` space, scale up the pool. Scale back down once the index rebuild is complete.
130
132
>
131
133
> For a dedicated SQL pool table with an ordered CCI, ALTER INDEX REORGANIZE does not re-sort the data. To resort data, use ALTER INDEX REBUILD.
@@ -134,10 +136,10 @@ Creating an ordered CCI is an offline operation. For tables with no partitions,
134
136
135
137
## Feature differences in SQL Server 2022 capabilities
136
138
137
-
SQL Server 2022 (16.x) introduced ordered clustered columnstore indexes similar to the feature in Azure Synapse dedicated SQL pools.
139
+
SQL Server 2022 (16.x) introduced ordered clustered columnstore indexes similar to the feature in Azure Synapse dedicated SQL pools.
138
140
139
-
- Currently, only SQL Server 2022 (16.x) and later support clustered columnstore enhanced rowgroup elimination capabilities for string, binary, and guid data types, and the datetimeoffset data type for scale greater than two.
140
-
- Currently, only SQL Server 2022 (16.x) supports clustered columnstore rowgroup elimination for the prefix of `LIKE` predicates.
141
+
- Currently, only SQL Server 2022 (16.x) and later versions support clustered columnstore enhanced [segment elimination](/sql/relational-databases/indexes/columnstore-indexes-design-guidance) capabilities for string, binary, and guid data types, and the datetimeoffset data type for scale greater than two. Previously, this segment elimination applies to numeric, date, and time data types, and the datetimeoffset data type with scale less than or equal to two.
142
+
- Currently, only SQL Server 2022 (16.x) and later versions support clustered columnstore rowgroup elimination for the prefix of `LIKE` predicates, for example `column LIKE 'string%'`. Segment elimination is not supported for non-prefix use of LIKE such as `column LIKE '%string'`.
141
143
142
144
For more information, see [What's New in Columnstore Indexes](/sql/relational-databases/indexes/columnstore-indexes-what-s-new).
143
145
@@ -146,8 +148,8 @@ For more information, see [What's New in Columnstore Indexes](/sql/relational-da
146
148
**A. To check for ordered columns and order ordinal:**
0 commit comments