Skip to content

Commit 75f176a

Browse files
authored
Merge pull request #3550 from ClickHouse/tp/system-table-clickpipes-blurb
Update clickpipes error reporting section to mention the system table
2 parents 7fcaf39 + d2b9939 commit 75f176a

File tree

1 file changed

+8
-3
lines changed
  • docs/integrations/data-ingestion/clickpipes

1 file changed

+8
-3
lines changed

docs/integrations/data-ingestion/clickpipes/index.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -82,8 +82,13 @@ Steps:
8282
<Image img={cp_custom_role} alt="Assign a custom role" size="lg" border/>
8383

8484
## Error reporting {#error-reporting}
85-
ClickPipes will create a table next to your destination table with the postfix `<destination_table_name>_clickpipes_error`. This table will contain any errors from the operations of your ClickPipe (network, connectivity, etc.) and also any data that don't conform to the schema. The error table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.
86-
If ClickPipes cannot connect to a data source or destination after 15min., ClickPipes instance stops and stores an appropriate message in the error table (providing the ClickHouse instance is available).
85+
ClickPipes will store errors in two separate tables depending on the type of error encountered during the ingestion process.
86+
### Record Errors {#record-errors}
87+
ClickPipes will create a table next to your destination table with the postfix `<destination_table_name>_clickpipes_error`. This table will contain any errors from malformed data or mismatched schema and will include the entirety of the invalid message. This table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.
88+
### System Errors {#system-errors}
89+
Errors related to the operation of the ClickPipe will be stored in the `system.clickpipes_log` table. This will store all other errors related to the operation of your ClickPipe (network, connectivity, etc.). This table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.
90+
91+
If ClickPipes cannot connect to a data source after 15 min or to a destination after 1 hr, the ClickPipes instance stops and stores an appropriate message in the system error table (provided the ClickHouse instance is available).
8792

8893
## F.A.Q {#faq}
8994
- **What is ClickPipes?**
@@ -100,4 +105,4 @@ If ClickPipes cannot connect to a data source or destination after 15min., Click
100105

101106
- **Is there a way to handle errors or failures when using ClickPipes for Kafka?**
102107

103-
Yes, ClickPipes for Kafka will automatically retry case of failures when consuming data from Kafka. ClickPipes also supports enabling a dedicated error table that will hold errors and malformed data for 7 days.
108+
Yes, ClickPipes for Kafka will automatically retry in the event of failures when consuming data from Kafka for any operational issue including network issues, connectivity issues, etc. In the event of malformed data or invalid schema, ClickPipes will store the record in the record_error table and continue processing.

0 commit comments

Comments
 (0)