Skip to content

Commit a4a6446

Browse files
authored
Add documentation for Kafka engine virtual columns
Added documentation for Kafka engine virtual columns, including built-in and extra columns for parse-error streaming.
1 parent 17460fa commit a4a6446

File tree

1 file changed

+40
-0
lines changed

1 file changed

+40
-0
lines changed
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
title: "Kafka engine Virtual columns"
3+
linkTitle: "Kafka virtual columns"
4+
weight: 100
5+
description: >-
6+
Kafka virtual columns
7+
---
8+
9+
## Kafka engine virtual columns (built-in)
10+
11+
[From the Kafka engine docs](https://clickhouse.com/docs/engines/table-engines/integrations/kafka?utm_source=chatgpt.com#virtual-columns), the supported virtual columns are:
12+
13+
- `_topic` — Kafka topic (LowCardinality(String))
14+
15+
- `_key` — message key (String)
16+
17+
- `_offset` — message offset (UInt64)
18+
19+
- `_timestamp` — message timestamp (Nullable(DateTime))
20+
21+
- `_timestamp_ms` — timestamp with millisecond precision (Nullable(DateTime64(3)))
22+
23+
- `_partition` — partition (UInt64)
24+
25+
- `_headers.name` — header keys (Array(String))
26+
27+
- `_headers.value` — header values (Array(String))
28+
29+
Extra virtual columns when you enable parse-error streaming:
30+
31+
If you set `kafka_handle_error_mode='stream'`, ClickHouse adds:
32+
33+
- `_raw_message` — the raw message that failed to parse (String)
34+
35+
- `_error` — the exception message from parsing failure (String)
36+
37+
Note: `_raw_message` and `_error` are populated only when parsing fails; otherwise they’re empty.
38+
39+
We can use these columns in a materialized view like this for example:
40+

0 commit comments

Comments
 (0)