You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/data-ingestion/clickpipes/kafka.md
+8-8Lines changed: 8 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -101,7 +101,7 @@ without an embedded schema id, then the specific schema ID or subject must be sp
101
101
102
102
11.**Congratulations!** you have successfully set up your first ClickPipe. If this is a streaming ClickPipe it will be continuously running, ingesting data in real-time from your remote data source.
103
103
104
-
## Supported Data Sources {#supported-data-sources}
104
+
## Supported data sources {#supported-data-sources}
#### Standard types support {#standard-types-support}
127
127
The following standard ClickHouse data types are currently supported in ClickPipes:
@@ -169,7 +169,7 @@ Note that you will have to manually change the destination column to the desired
169
169
170
170
ClickPipes supports all Avro Primitive and Complex types, and all Avro Logical types except `time-millis`, `time-micros`, `local-timestamp-millis`, `local_timestamp-micros`, and `duration`. Avro `record` types are converted to Tuple, `array` types to Array, and `map` to Map (string keys only). In general the conversions listed [here](/interfaces/formats/Avro#data-types-matching) are available. We recommend using exact type matching for Avro numeric types, as ClickPipes does not check for overflow or precision loss on type conversion.
171
171
172
-
#### Nullable Types and Avro Unions {#nullable-types-and-avro-unions}
172
+
#### Nullable types and Avro unions {#nullable-types-and-avro-unions}
173
173
174
174
Nullable types in Avro are defined by using a Union schema of `(T, null)` or `(null, T)` where T is the base Avro type. During schema inference, such unions will be mapped to a ClickHouse "Nullable" column. Note that ClickHouse does not support
175
175
`Nullable(Array)`, `Nullable(Map)`, or `Nullable(Tuple)` types. Avro null unions for these types will be mapped to non-nullable versions (Avro Record types are mapped to a ClickHouse named Tuple). Avro "nulls" for these types will be inserted as:
@@ -179,7 +179,7 @@ Nullable types in Avro are defined by using a Union schema of `(T, null)` or `(n
179
179
180
180
ClickPipes does not currently support schemas that contain other Avro Unions (this may change in the future with the maturity of the new ClickHouse Variant and JSON data types). If the Avro schema contains a "non-null" union, ClickPipes will generate an error when attempting to calculate a mapping between the Avro schema and Clickhouse column types.
ClickPipes dynamically retrieves and applies the Avro schema from the configured Schema Registry using the schema ID embedded in each message/event. Schema updates are detected and processed automatically.
185
185
@@ -190,7 +190,7 @@ The following rules are applied to the mapping between the retrieved Avro schema
190
190
- If the Avro schema is missing a field defined in the ClickHouse destination mapping, the ClickHouse column will be populated with a "zero" value, such as 0 or an empty string. Note that [DEFAULT](/sql-reference/statements/create/table#default) expressions are not currently evaluated for ClickPipes inserts (this is temporary limitation pending updates to the ClickHouse server default processing).
191
191
- If the Avro schema field and the ClickHouse column are incompatible, inserts of that row/message will fail, and the failure will be recorded in the ClickPipes errors table. Note that several implicit conversions are supported (like between numeric types), but not all (for example, an Avro `record` field can not be inserted into an `Int32` ClickHouse column).
192
192
193
-
## Kafka Virtual Columns {#kafka-virtual-columns}
193
+
## Kafka virtual columns {#kafka-virtual-columns}
194
194
195
195
The following virtual columns are supported for Kafka compatible streaming data sources. When creating a new destination table virtual columns can be added by using the `Add Column` button.
196
196
@@ -208,7 +208,7 @@ The following virtual columns are supported for Kafka compatible streaming data
208
208
Note that the _raw_message column is only recommended for JSON data. For use cases where only the JSON string is required (such as using ClickHouse [`JsonExtract*`](/sql-reference/functions/json-functions#jsonextract-functions) functions to populate a downstream materialized
209
209
view), it may improve ClickPipes performance to delete all the "non-virtual" columns.
210
210
211
-
## Best Practices {#best-practices}
211
+
## Best practices {#best-practices}
212
212
213
213
### Message Compression {#compression}
214
214
We strongly recommend using compression for your Kafka topics. Compression can result in a significant saving in data transfer costs with virtually no performance hit.
@@ -275,7 +275,7 @@ Below is an example of the required IAM policy for Apache Kafka APIs for MSK:
275
275
}
276
276
```
277
277
278
-
#### Configuring a Trusted Relationship {#configuring-a-trusted-relationship}
278
+
#### Configuring a trusted relationship {#configuring-a-trusted-relationship}
279
279
280
280
If you are authenticating to MSK with a IAM role ARN, you will need to add a trusted relationship between your ClickHouse Cloud instance so the role can be assumed.
0 commit comments