Skip to content

Commit bc864c6

Browse files
mneedhammneedham
committed
Dynamic column selection
1 parent 6e09691 commit bc864c6

File tree

1 file changed

+205
-0
lines changed

1 file changed

+205
-0
lines changed

‎docs/guides/developer/dynamic-column-selection.md

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
---
2+
slug: /guides/developer/dynamic-column-selection
3+
sidebar_label: 'Dynamic column selection'
4+
title: 'Dynamic column selection'
5+
description: 'Use alternative query languages in ClickHouse'
6+
---
7+
8+
[Dynamic column selection](/docs/sql-reference/statements/select#dynamic-column-selection) is a powerful but underutilized ClickHouse feature that allows you to select columns using regular expressions instead of naming each column individually. You can also apply functions to matching columns using the APPLY modifier, making it incredibly useful for data analysis and transformation tasks.
9+
10+
We're going to learn how to use this feature with help from the [New York taxis dataset](/docs/getting-started/example-datasets/nyc-taxi), which you can find also find in the [ClickHouse SQL playground](https://sql.clickhouse.com?query=LS0gRGF0YXNldCBjb250YWluaW5nIHRheGkgcmlkZSBkYXRhIGluIE5ZQyBmcm9tIDIwMDkuIE1vcmUgaW5mbyBoZXJlOiBodHRwczovL2NsaWNraG91c2UuY29tL2RvY3MvZW4vZ2V0dGluZy1zdGFydGVkL2V4YW1wbGUtZGF0YXNldHMvbnljLXRheGkKU0VMRUNUICogRlJPTSBueWNfdGF4aS50cmlwcyBMSU1JVCAxMDA).
11+
12+
<iframe width="768" height="432" src="https://www.youtube.com/embed/moabRqqHNo4?si=jgmInV-u3UxtLvMS" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
13+
14+
## Selecting columns that match a pattern
15+
16+
Let's start with a common scenario: selecting only the columns that contain `_amount` from the NYC taxi dataset. Instead of manually typing each column name, we can use the `COLUMNS` expression with a regular expression:
17+
18+
```sql
19+
FROM nyc_taxi.trips
20+
SELECT COLUMNS('.*_amount')
21+
LIMIT 10;
22+
```
23+
24+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudCcpCkZST00gbnljX3RheGkudHJpcHMKTElNSVQgMTA7&run_query=true)
25+
26+
This query returns the first 10 rows, but only for columns whose names match the pattern `.*_amount` (any characters followed by "_amount").
27+
28+
```text
29+
┌─fare_amount─┬─tip_amount─┬─tolls_amount─┬─total_amount─┐
30+
1. │ 9 │ 0 │ 0 │ 9.8 │
31+
2. │ 9 │ 0 │ 0 │ 9.8 │
32+
3. │ 3.5 │ 0 │ 0 │ 4.8 │
33+
4. │ 3.5 │ 0 │ 0 │ 4.8 │
34+
5. │ 3.5 │ 0 │ 0 │ 4.3 │
35+
6. │ 3.5 │ 0 │ 0 │ 4.3 │
36+
7. │ 2.5 │ 0 │ 0 │ 3.8 │
37+
8. │ 2.5 │ 0 │ 0 │ 3.8 │
38+
9. │ 5 │ 0 │ 0 │ 5.8 │
39+
10. │ 5 │ 0 │ 0 │ 5.8 │
40+
└─────────────┴────────────┴──────────────┴──────────────┘
41+
```
42+
43+
Let’s say we also want to return columns that contain the terms `fee` or `tax`.
44+
We can update the regular expression to include those:
45+
46+
```sql
47+
SELECT COLUMNS('.*_amount|fee|tax')
48+
FROM nyc_taxi.trips
49+
ORDER BY rand()
50+
LIMIT 3;
51+
```
52+
53+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykKRlJPTSBueWNfdGF4aS50cmlwcwpPUkRFUiBCWSByYW5kKCkgCkxJTUlUIDM7&run_query=true)
54+
55+
```text
56+
┌─fare_amount─┬─mta_tax─┬─tip_amount─┬─tolls_amount─┬─ehail_fee─┬─total_amount─┐
57+
1. │ 5 │ 0.5 │ 1 │ 0 │ 0 │ 7.8 │
58+
2. │ 12.5 │ 0.5 │ 0 │ 0 │ 0 │ 13.8 │
59+
3. │ 4.5 │ 0.5 │ 1.66 │ 0 │ 0 │ 9.96 │
60+
└─────────────┴─────────┴────────────┴──────────────┴───────────┴──────────────┘
61+
```
62+
63+
## Selecting multiple patterns
64+
65+
We can combine multiple column patterns in a single query:
66+
67+
```sql
68+
SELECT
69+
COLUMNS('.*_amount'),
70+
COLUMNS('.*_date.*')
71+
FROM nyc_taxi.trips
72+
LIMIT 5;
73+
```
74+
75+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIAogICAgQ09MVU1OUygnLipfYW1vdW50JyksCiAgICBDT0xVTU5TKCcuKl9kYXRlLionKQpGUk9NIG55Y190YXhpLnRyaXBzCkxJTUlUIDU7&run_query=true)
76+
77+
```text
78+
┌─fare_amount─┬─tip_amount─┬─tolls_amount─┬─total_amount─┬─pickup_date─┬─────pickup_datetime─┬─dropoff_date─┬────dropoff_datetime─┐
79+
1. │ 9 │ 0 │ 0 │ 9.8 │ 2001-01-01 │ 2001-01-01 00:01:48 │ 2001-01-01 │ 2001-01-01 00:15:47 │
80+
2. │ 9 │ 0 │ 0 │ 9.8 │ 2001-01-01 │ 2001-01-01 00:01:48 │ 2001-01-01 │ 2001-01-01 00:15:47 │
81+
3. │ 3.5 │ 0 │ 0 │ 4.8 │ 2001-01-01 │ 2001-01-01 00:02:08 │ 2001-01-01 │ 2001-01-01 01:00:02 │
82+
4. │ 3.5 │ 0 │ 0 │ 4.8 │ 2001-01-01 │ 2001-01-01 00:02:08 │ 2001-01-01 │ 2001-01-01 01:00:02 │
83+
5. │ 3.5 │ 0 │ 0 │ 4.3 │ 2001-01-01 │ 2001-01-01 00:02:26 │ 2001-01-01 │ 2001-01-01 00:04:49 │
84+
└─────────────┴────────────┴──────────────┴──────────────┴─────────────┴─────────────────────┴──────────────┴─────────────────────┘
85+
```
86+
87+
## Apply functions to all columns
88+
89+
We can also use the [`APPLY`](https://clickhouse.com/docs/sql-reference/statements/select#apply) modifier to apply functions across every column.
90+
For example, if we wanted to find the maximum value of each of those columns, we could run the following query:
91+
92+
```sql
93+
SELECT COLUMNS('.*_amount|fee|tax') APPLY(max)
94+
FROM nyc_taxi.trips;
95+
```
96+
97+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkobWF4KQpGUk9NIG55Y190YXhpLnRyaXBzOw&run_query=true)
98+
99+
100+
```text
101+
┌─max(fare_amount)─┬─max(mta_tax)─┬─max(tip_amount)─┬─max(tolls_amount)─┬─max(ehail_fee)─┬─max(total_amount)─┐
102+
1. │ 998310 │ 500000.5 │ 3950588.8 │ 7999.92 │ 1.95 │ 3950611.5 │
103+
└──────────────────┴──────────────┴─────────────────┴───────────────────┴────────────────┴───────────────────┘
104+
```
105+
106+
Or maybe, we’d like to see the average instead:
107+
108+
```sql
109+
SELECT COLUMNS('.*_amount|fee|tax') APPLY(avg)
110+
FROM nyc_taxi.trips
111+
```
112+
113+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkoYXZnKQpGUk9NIG55Y190YXhpLnRyaXBzOw&run_query=true)
114+
115+
116+
```text
117+
┌─avg(fare_amount)─┬───────avg(mta_tax)─┬────avg(tip_amount)─┬──avg(tolls_amount)─┬──────avg(ehail_fee)─┬──avg(total_amount)─┐
118+
1. │ 11.8044154834777 │ 0.4555942672733423 │ 1.3469850969211845 │ 0.2256511991414463 │ 3.37600560437412e-9 │ 14.423323722271563 │
119+
└──────────────────┴────────────────────┴────────────────────┴────────────────────┴─────────────────────┴────────────────────┘
120+
```
121+
122+
123+
Those values contain a lot of decimal places, but luckily we can fix that by chaining functions. In this case, we’ll apply the avg function, followed by the round function:
124+
125+
```sql
126+
SELECT COLUMNS('.*_amount|fee|tax') APPLY(avg) APPLY(round)
127+
FROM nyc_taxi.trips;
128+
```
129+
130+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkoYXZnKSBBUFBMWShyb3VuZCkKRlJPTSBueWNfdGF4aS50cmlwczs&run_query=true)
131+
132+
133+
```text
134+
┌─round(avg(fare_amount))─┬─round(avg(mta_tax))─┬─round(avg(tip_amount))─┬─round(avg(tolls_amount))─┬─round(avg(ehail_fee))─┬─round(avg(total_amount))─┐
135+
1. │ 12 │ 0 │ 1 │ 0 │ 0 │ 14 │
136+
└─────────────────────────┴─────────────────────┴────────────────────────┴──────────────────────────┴───────────────────────┴──────────────────────────┘
137+
```
138+
139+
140+
But that rounds the averages to whole numbers. If we want to round to, say, 2 decimal places, we can do that as well. As well as taking in functions, the APPLY function takes in a lambda, which gives us the flexibility to have the round function round our average values to 2 decimal places:
141+
142+
```sql
143+
SELECT COLUMNS('.*_amount|fee|tax') APPLY(avg) APPLY(x -> round(x, 2))
144+
FROM nyc_taxi.trips;
145+
```
146+
147+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkgYXZnIEFQUExZIHggLT4gcm91bmQoeCwgMikKRlJPTSBueWNfdGF4aS50cmlwcw&run_query=true)
148+
149+
150+
```text
151+
┌─round(avg(fare_amount), 2)─┬─round(avg(mta_tax), 2)─┬─round(avg(tip_amount), 2)─┬─round(avg(tolls_amount), 2)─┬─round(avg(ehail_fee), 2)─┬─round(avg(total_amount), 2)─┐
152+
1. │ 11.8 │ 0.46 │ 1.35 │ 0.23 │ 0 │ 14.42 │
153+
└────────────────────────────┴────────────────────────┴───────────────────────────┴─────────────────────────────┴──────────────────────────┴─────────────────────────────┘
154+
```
155+
156+
## Replacing columns
157+
158+
So far so good. But let’s say we want to adjust one of the values, while leaving the other ones as they are. For example, maybe we want to double the total amount and divide the MTA tax by 1.1. We can do that by using the REPLACE clause, which will replace a column while leaving the other ones as they are.
159+
160+
```sql
161+
FROM nyc_taxi.trips
162+
SELECT
163+
COLUMNS('.*_amount|fee|tax')
164+
REPLACE(
165+
total_amount*2 AS total_amount,
166+
mta_tax/1.1 AS mta_tax
167+
)
168+
APPLY(avg)
169+
APPLY(col -> round(col, 2));
170+
```
171+
172+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=RlJPTSBueWNfdGF4aS50cmlwcyAKU0VMRUNUIAogIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykKICBSRVBMQUNFKAogICAgdG90YWxfYW1vdW50KjIgQVMgdG90YWxfYW1vdW50LAogICAgbXRhX3RheC8xLjEgQVMgbXRhX3RheAogICkgCiAgQVBQTFkoYXZnKQogIEFQUExZKGNvbCAtPiByb3VuZChjb2wsIDIpKTs&run_query=true)
173+
174+
175+
```text
176+
┌─round(avg(fare_amount), 2)─┬─round(avg(di⋯, 1.1)), 2)─┬─round(avg(tip_amount), 2)─┬─round(avg(tolls_amount), 2)─┬─round(avg(ehail_fee), 2)─┬─round(avg(mu⋯nt, 2)), 2)─┐
177+
1. │ 11.8 │ 0.41 │ 1.35 │ 0.23 │ 0 │ 28.85 │
178+
└────────────────────────────┴──────────────────────────┴───────────────────────────┴─────────────────────────────┴──────────────────────────┴──────────────────────────┘
179+
```
180+
181+
## Excluding columns
182+
183+
We can also choose to exclude a field by using the EXCEPT clause. For example, to remove the tolls_amount column, we would write the following query:
184+
185+
```sql
186+
FROM nyc_taxi.trips
187+
SELECT
188+
COLUMNS('.*_amount|fee|tax') EXCEPT(tolls_amount)
189+
REPLACE(
190+
total_amount*2 AS total_amount,
191+
mta_tax/1.1 AS mta_tax
192+
)
193+
APPLY(avg)
194+
APPLY(col -> round(col, 2));
195+
```
196+
197+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=RlJPTSBueWNfdGF4aS50cmlwcyAKU0VMRUNUIAogIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgRVhDRVBUKHRvbGxzX2Ftb3VudCkKICBSRVBMQUNFKAogICAgdG90YWxfYW1vdW50KjIgQVMgdG90YWxfYW1vdW50LAogICAgbXRhX3RheC8xLjEgQVMgbXRhX3RheAogICkgCiAgQVBQTFkoYXZnKQogIEFQUExZKGNvbCAtPiByb3VuZChjb2wsIDIpKTs&run_query=true)
198+
199+
200+
201+
```text
202+
┌─round(avg(fare_amount), 2)─┬─round(avg(di⋯, 1.1)), 2)─┬─round(avg(tip_amount), 2)─┬─round(avg(ehail_fee), 2)─┬─round(avg(mu⋯nt, 2)), 2)─┐
203+
1. │ 11.8 │ 0.41 │ 1.35 │ 0 │ 28.85 │
204+
└────────────────────────────┴──────────────────────────┴───────────────────────────┴──────────────────────────┴──────────────────────────┘
205+
```

0 commit comments

Comments
 (0)