You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/best-practices/sparse-primary-indexes.md
+17-17Lines changed: 17 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -331,7 +331,7 @@ ClickHouse is a <a href="https://clickhouse.com/docs/introduction/distinctive-fe
331
331
- then by `URL`,
332
332
- and lastly by `EventTime`:
333
333
334
-
<Imageimg={sparsePrimaryIndexes01}size="lg"alt="Sparse Primary Indices 01"background="white"/>
334
+
<Imageimg={sparsePrimaryIndexes01}size="md"alt="Sparse Primary Indices 01"background="white"/>
335
335
336
336
`UserID.bin`, `URL.bin`, and `EventTime.bin` are the data files on disk where the values of the `UserID`, `URL`, and `EventTime` columns are stored.
337
337
@@ -355,7 +355,7 @@ Column values are not physically stored inside granules: granules are just a log
355
355
The following diagram shows how the (column values of) 8.87 million rows of our table
356
356
are organized into 1083 granules, as a result of the table's DDL statement containing the setting `index_granularity` (set to its default value of 8192).
357
357
358
-
<Imageimg={sparsePrimaryIndexes02}size="lg"alt="Sparse Primary Indices 02"background="white"/>
358
+
<Imageimg={sparsePrimaryIndexes02}size="md"alt="Sparse Primary Indices 02"background="white"/>
359
359
360
360
The first (based on physical order on disk) 8192 rows (their column values) logically belong to granule 0, then the next 8192 rows (their column values) belong to granule 1 and so on.
361
361
@@ -390,7 +390,7 @@ For example
390
390
391
391
In total the index has 1083 entries for our table with 8.87 million rows and 1083 granules:
392
392
393
-
<Imageimg={sparsePrimaryIndexes03b}size="lg"alt="Sparse Primary Indices 03b"background="white"/>
393
+
<Imageimg={sparsePrimaryIndexes03b}size="md"alt="Sparse Primary Indices 03b"background="white"/>
394
394
395
395
:::note
396
396
- For tables with [adaptive index granularity](/whats-new/changelog/2019.md/#experimental-features-1), there is also one "final" additional mark stored in the primary index that records the values of the primary key columns of the last table row, but because we disabled adaptive index granularity (in order to simplify the discussions in this guide, as well as make the diagrams and results reproducible), the index of our example table doesn't include this final mark.
@@ -615,7 +615,7 @@ We discuss that second stage in more detail in the following section.
615
615
616
616
The following diagram illustrates a part of the primary index file for our table.
617
617
618
-
<Imageimg={sparsePrimaryIndexes04}size="lg"alt="Sparse Primary Indices 04"background="white"/>
618
+
<Imageimg={sparsePrimaryIndexes04}size="md"alt="Sparse Primary Indices 04"background="white"/>
619
619
620
620
As discussed above, via a binary search over the index’s 1083 UserID marks, mark 176 was identified. Its corresponding granule 176 can therefore possibly contain rows with a UserID column value of 749.927.693.
621
621
@@ -637,7 +637,7 @@ In ClickHouse the physical locations of all granules for our table are stored in
637
637
638
638
The following diagram shows the three mark files `UserID.mrk`, `URL.mrk`, and `EventTime.mrk` that store the physical locations of the granules for the table’s `UserID`, `URL`, and `EventTime` columns.
639
639
640
-
<Imageimg={sparsePrimaryIndexes05}size="lg"alt="Sparse Primary Indices 05"background="white"/>
640
+
<Imageimg={sparsePrimaryIndexes05}size="md"alt="Sparse Primary Indices 05"background="white"/>
641
641
642
642
We have discussed how the primary index is a flat uncompressed array file (primary.idx), containing index marks that are numbered starting at 0.
643
643
@@ -688,7 +688,7 @@ The indirection provided by mark files avoids storing, directly within the prima
688
688
689
689
The following diagram and the text below illustrate how for our example query ClickHouse locates granule 176 in the UserID.bin data file.
690
690
691
-
<Imageimg={sparsePrimaryIndexes06}size="lg"alt="Sparse Primary Indices 06"background="white"/>
691
+
<Imageimg={sparsePrimaryIndexes06}size="md"alt="Sparse Primary Indices 06"background="white"/>
692
692
693
693
We discussed earlier in this guide that ClickHouse selected the primary index mark 176 and therefore granule 176 as possibly containing matching rows for our query.
694
694
@@ -802,7 +802,7 @@ We have marked the key column values for the first table rows for each granule i
802
802
803
803
Suppose UserID had low cardinality. In this case it would be likely that the same UserID value is spread over multiple table rows and granules and therefore index marks. For index marks with the same UserID, the URL values for the index marks are sorted in ascending order (because the table rows are ordered first by UserID and then by URL). This allows efficient filtering as described below:
804
804
805
-
<Imageimg={sparsePrimaryIndexes07}size="lg"alt="Sparse Primary Indices 06"background="white"/>
805
+
<Imageimg={sparsePrimaryIndexes07}size="md"alt="Sparse Primary Indices 06"background="white"/>
806
806
807
807
There are three different scenarios for the granule selection process for our abstract sample data in the diagram above:
808
808
@@ -816,7 +816,7 @@ There are three different scenarios for the granule selection process for our ab
816
816
817
817
When the UserID has high cardinality then it is unlikely that the same UserID value is spread over multiple table rows and granules. This means the URL values for the index marks are not monotonically increasing:
818
818
819
-
<Imageimg={sparsePrimaryIndexes08}size="lg"alt="Sparse Primary Indices 06"background="white"/>
819
+
<Imageimg={sparsePrimaryIndexes08}size="md"alt="Sparse Primary Indices 06"background="white"/>
820
820
821
821
As we can see in the diagram above, all shown marks whose URL values are smaller than W3 are getting selected for streaming its associated granule's rows into the ClickHouse engine.
822
822
@@ -850,7 +850,7 @@ ALTER TABLE hits_UserID_URL MATERIALIZE INDEX url_skipping_index;
850
850
```
851
851
ClickHouse now created an additional index that is storing - per group of 4 consecutive [granules](#data-is-organized-into-granules-for-parallel-data-processing) (note the `GRANULARITY 4` clause in the `ALTER TABLE` statement above) - the minimum and maximum URL value:
852
852
853
-
<Imageimg={sparsePrimaryIndexes13a}size="lg"alt="Sparse Primary Indices 13a"background="white"/>
853
+
<Imageimg={sparsePrimaryIndexes13a}size="md"alt="Sparse Primary Indices 13a"background="white"/>
854
854
855
855
The first index entry (‘mark 0’ in the diagram above) is storing the minimum and maximum URL values for the [rows belonging to the first 4 granules of our table](#data-is-organized-into-granules-for-parallel-data-processing).
Because we switched the order of the columns in the primary key, the inserted rows are now stored on disk in a different lexicographical order (compared to our [original table](#a-table-with-a-primary-key)) and therefore also the 1083 granules of that table are containing different values than before:
947
947
948
-
<Imageimg={sparsePrimaryIndexes10}size="lg"alt="Sparse Primary Indices 10"background="white"/>
948
+
<Imageimg={sparsePrimaryIndexes10}size="md"alt="Sparse Primary Indices 10"background="white"/>
949
949
950
950
This is the resulting primary key:
951
951
952
-
<Imageimg={sparsePrimaryIndexes11}size="lg"alt="Sparse Primary Indices 11"background="white"/>
952
+
<Imageimg={sparsePrimaryIndexes11}size="md"alt="Sparse Primary Indices 11"background="white"/>
953
953
954
954
That can now be used to significantly speed up the execution of our example query filtering on the URL column in order to calculate the top 10 users that most frequently clicked on the URL "http://public_search":
955
955
```sql
@@ -1097,7 +1097,7 @@ Ok.
1097
1097
- if new rows are inserted into the source table hits_UserID_URL, then that rows are automatically also inserted into the implicitly created table
1098
1098
- Effectively the implicitly created table has the same row order and primary index as the [secondary table that we created explicitly](/guides/best-practices/sparse-primary-indexes#option-1-secondary-tables):
1099
1099
1100
-
<Imageimg={sparsePrimaryIndexes12b1}size="lg"alt="Sparse Primary Indices 12b1"background="white"/>
1100
+
<Imageimg={sparsePrimaryIndexes12b1}size="md"alt="Sparse Primary Indices 12b1"background="white"/>
1101
1101
1102
1102
ClickHouse is storing the [column data files](#data-is-stored-on-disk-ordered-by-primary-key-columns) (*.bin), the [mark files](#mark-files-are-used-for-locating-granules) (*.mrk2) and the [primary index](#the-primary-index-has-one-entry-per-granule) (primary.idx) of the implicitly created table in a special folder withing the ClickHouse server's data directory:
1103
1103
@@ -1181,7 +1181,7 @@ ALTER TABLE hits_UserID_URL
1181
1181
- please note that projections do not make queries that use ORDER BY more efficient, even if the ORDER BY matches the projection's ORDER BY statement (see https://github.com/ClickHouse/ClickHouse/issues/47333)
1182
1182
- Effectively the implicitly created hidden table has the same row order and primary index as the [secondary table that we created explicitly](/guides/best-practices/sparse-primary-indexes#option-1-secondary-tables):
1183
1183
1184
-
<Imageimg={sparsePrimaryIndexes12c1}size="lg"alt="Sparse Primary Indices 12c1"background="white"/>
1184
+
<Imageimg={sparsePrimaryIndexes12c1}size="md"alt="Sparse Primary Indices 12c1"background="white"/>
1185
1185
1186
1186
ClickHouse is storing the [column data files](#data-is-stored-on-disk-ordered-by-primary-key-columns) (*.bin), the [mark files](#mark-files-are-used-for-locating-granules) (*.mrk2) and the [primary index](#the-primary-index-has-one-entry-per-granule) (primary.idx) of the hidden table in a special folder (marked in orange in the screenshot below) next to the source table's data files, mark files, and primary index files:
1187
1187
@@ -1449,7 +1449,7 @@ In the following we illustrate why it's beneficial for the compression ratio of
1449
1449
1450
1450
The diagram below sketches the on-disk order of rows for a primary key where the key columns are ordered by cardinality in ascending order:
1451
1451
1452
-
<Imageimg={sparsePrimaryIndexes14a}size="lg"alt="Sparse Primary Indices 14a"background="white"/>
1452
+
<Imageimg={sparsePrimaryIndexes14a}size="md"alt="Sparse Primary Indices 14a"background="white"/>
1453
1453
1454
1454
We discussed that [the table's row data is stored on disk ordered by primary key columns](#data-is-stored-on-disk-ordered-by-primary-key-columns).
1455
1455
@@ -1461,7 +1461,7 @@ and locality (the more similar the data is, the better the compression ratio is)
1461
1461
1462
1462
In contrast to the diagram above, the diagram below sketches the on-disk order of rows for a primary key where the key columns are ordered by cardinality in descending order:
1463
1463
1464
-
<Imageimg={sparsePrimaryIndexes14b}size="lg"alt="Sparse Primary Indices 14b"background="white"/>
1464
+
<Imageimg={sparsePrimaryIndexes14b}size="md"alt="Sparse Primary Indices 14b"background="white"/>
1465
1465
1466
1466
Now the table's rows are first ordered by their `ch` value, and rows that have the same `ch` value are ordered by their `cl` value.
1467
1467
But because the first key column `ch` has high cardinality, it is unlikely that there are rows with the same `ch` value. And because of that is is also unlikely that `cl` values are ordered (locally - for rows with the same `ch` value).
@@ -1504,7 +1504,7 @@ The following diagram shows
1504
1504
- the insert order of rows when the content changes (for example because of keystrokes typing the text into the text-area) and
1505
1505
- the on-disk order of the data from the inserted rows when the `PRIMARY KEY (hash)` is used:
1506
1506
1507
-
<Imageimg={sparsePrimaryIndexes15a}size="lg"alt="Sparse Primary Indices 15a"background="white"/>
1507
+
<Imageimg={sparsePrimaryIndexes15a}size="md"alt="Sparse Primary Indices 15a"background="white"/>
1508
1508
1509
1509
Because the `hash` column is used as the primary key column
1510
1510
- specific rows can be retrieved [very quickly](#the-primary-index-is-used-for-selecting-granules), but
@@ -1519,7 +1519,7 @@ The following diagram shows
1519
1519
- the insert order of rows when the content changes (for example because of keystrokes typing the text into the text-area) and
1520
1520
- the on-disk order of the data from the inserted rows when the compound `PRIMARY KEY (fingerprint, hash)` is used:
1521
1521
1522
-
<Imageimg={sparsePrimaryIndexes15b}size="lg"alt="Sparse Primary Indices 15b"background="white"/>
1522
+
<Imageimg={sparsePrimaryIndexes15b}size="md"alt="Sparse Primary Indices 15b"background="white"/>
1523
1523
1524
1524
Now the rows on disk are first ordered by `fingerprint`, and for rows with the same fingerprint value, their `hash` value determines the final order.
0 commit comments