You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/data-ingestion/clickpipes/index.md
+8-3Lines changed: 8 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -70,8 +70,13 @@ Steps:
70
70
<Imageimg={cp_custom_role}alt="Assign a custom role"size="lg"border/>
71
71
72
72
## Error reporting {#error-reporting}
73
-
ClickPipes will create a table next to your destination table with the postfix `<destination_table_name>_clickpipes_error`. This table will contain any errors from malformed data or mismatched schema and will include the entirety of the invalid message. In addition to this, error not related to data will be stored in the `system.clickpipes_log` table, this will store all other errors related to the operation of your ClickPipe (network, connectivity, etc.). Both error tables have a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.
74
-
If ClickPipes cannot connect to a data source or destination after 15min., ClickPipes instance stops and stores an appropriate message in the system error table (providing the ClickHouse instance is available).
73
+
ClickPipes will store errors in two separate tables depending on the type of error encountered during the ingestion process.
74
+
### Record Errors
75
+
ClickPipes will create a table next to your destination table with the postfix `<destination_table_name>_clickpipes_error`. This table will contain any errors from malformed data or mismatched schema and will include the entirety of the invalid message. This table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.
76
+
### System Errors
77
+
Errors related to the operation of the ClickPipe will be stored in the `system.clickpipes_log` table. This will store all other errors related to the operation of your ClickPipe (network, connectivity, etc.). This table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.
78
+
79
+
If ClickPipes cannot connect to a data source or destination after 15 min., the ClickPipes instance stops and stores an appropriate message in the system error table (providing the ClickHouse instance is available).
75
80
76
81
## F.A.Q {#faq}
77
82
-**What is ClickPipes?**
@@ -88,4 +93,4 @@ If ClickPipes cannot connect to a data source or destination after 15min., Click
88
93
89
94
-**Is there a way to handle errors or failures when using ClickPipes for Kafka?**
90
95
91
-
Yes, ClickPipes for Kafka will automatically retry case of failures when consuming data from Kafka. ClickPipes also creates a dedicated error table that will hold errors and malformed data for 7 days.
96
+
Yes, ClickPipes for Kafka will automatically retry in the event of failures when consuming data from Kafka for any operational issue including network issues, connectivity issues, etc. In the event of malformed data or invalid schema, ClickPipes will store the record in the record_error table and continue processing.
0 commit comments