Merge remote-tracking branch 'upstream/main' into docs-kafka_table_engine

Author: morsapaes

docs/guides/developer/deduplicating-inserts-on-retries.md

@@ -9,6 +9,16 @@ Insert operations can sometimes fail due to errors such as timeouts. When insert
 
 When an insert is retried, ClickHouse tries to determine whether the data has already been successfully inserted. If the inserted data is marked as a duplicate, ClickHouse does not insert it into the destination table. However, the user will still receive a successful operation status as if the data had been inserted normally.
 
+## Limitations {#limitations}
+
+### Uncertain insert status {#uncertain-insert-status}
+
+The user must retry the insert operation until it succeeds. If all retries fail, it is impossible to determine whether the data was inserted or not. When materialized views are involved, it is also unclear in which tables the data may have appeared. The materialized views could be out of sync with the source table.
+
+### Deduplication window limit {#deduplication-window-limit}
+
+If more than `*_deduplication_window` other insert operations occur during the retry sequence, deduplication may not work as intended. In this case, the same data can be inserted multiple times.
+
 ## Enabling insert deduplication on retries {#enabling-insert-deduplication-on-retries}
 
 ### Insert deduplication for tables {#insert-deduplication-for-tables}
@@ -45,7 +55,8 @@ You can control this process using the following settings for the source table:
 - [`replicated_deduplication_window_seconds`](/operations/settings/merge-tree-settings#replicated_deduplication_window_seconds)
 - [`non_replicated_deduplication_window`](/operations/settings/merge-tree-settings#non_replicated_deduplication_window)
 
-You can also use the user profile setting [`deduplicate_blocks_in_dependent_materialized_views`](/operations/settings/settings#deduplicate_blocks_in_dependent_materialized_views).
+You have to also enable the user profile setting [`deduplicate_blocks_in_dependent_materialized_views`](/operations/settings/settings#deduplicate_blocks_in_dependent_materialized_views).
+With enabled setting `insert_deduplicate=1` an inserted data is deduplicated in source table. The setting `deduplicate_blocks_in_dependent_materialized_views=1` additionally enables deduplication in dependant tables. You have to enable both if full deduplication is desired.
 
 When inserting blocks into tables under materialized views, ClickHouse calculates the `block_id` by hashing a string that combines the `block_id`s from the source table and additional identifiers. This ensures accurate deduplication within materialized views, allowing data to be distinguished based on its original insertion, regardless of any transformations applied before reaching the destination table under the materialized view.
 

docs/integrations/data-ingestion/clickpipes/index.md

@@ -82,8 +82,13 @@ Steps:
 <Image img={cp_custom_role} alt="Assign a custom role" size="lg" border/>
 
 ## Error reporting {#error-reporting}
-ClickPipes will create a table next to your destination table with the postfix `<destination_table_name>_clickpipes_error`. This table will contain any errors from the operations of your ClickPipe (network, connectivity, etc.) and also any data that don't conform to the schema. The error table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.
-If ClickPipes cannot connect to a data source or destination after 15min., ClickPipes instance stops and  stores an appropriate message in the error table (providing the ClickHouse instance is available).
+ClickPipes will store errors in two separate tables depending on the type of error encountered during the ingestion process.
+### Record Errors {#record-errors}
+ClickPipes will create a table next to your destination table with the postfix `<destination_table_name>_clickpipes_error`. This table will contain any errors from malformed data or mismatched schema and will include the entirety of the invalid message. This table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.
+### System Errors {#system-errors}
+Errors related to the operation of the ClickPipe will be stored in the `system.clickpipes_log` table. This will store all other errors related to the operation of your ClickPipe (network, connectivity, etc.). This table has a [TTL](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.
+
+If ClickPipes cannot connect to a data source after 15 min or to a destination after 1 hr, the ClickPipes instance stops and stores an appropriate message in the system error table (provided the ClickHouse instance is available).
 
 ## F.A.Q {#faq}
 - **What is ClickPipes?**
@@ -100,4 +105,4 @@ If ClickPipes cannot connect to a data source or destination after 15min., Click
 
 - **Is there a way to handle errors or failures when using ClickPipes for Kafka?**
 
-  Yes, ClickPipes for Kafka will automatically retry case of failures when consuming data from Kafka. ClickPipes also supports enabling a dedicated error table that will hold errors and malformed data for 7 days.
+  Yes, ClickPipes for Kafka will automatically retry in the event of failures when consuming data from Kafka for any operational issue including network issues, connectivity issues, etc. In the event of malformed data or invalid schema, ClickPipes will store the record in the record_error table and continue processing.

docs/integrations/data-ingestion/kafka/kafka-table-engine.md

@@ -6,7 +6,6 @@ description: 'Using the Kafka Table Engine'
 title: 'Using the Kafka table engine'
 ---
 
-import CloudAvailableBadge from '@theme/badges/CloudAvailableBadge';
 import Image from '@theme/IdealImage';
 import kafka_01 from '@site/static/images/integrations/data-ingestion/kafka/kafka_01.png';
 import kafka_02 from '@site/static/images/integrations/data-ingestion/kafka/kafka_02.png';
@@ -15,8 +14,6 @@ import kafka_04 from '@site/static/images/integrations/data-ingestion/kafka/kafk
 
 # Using the Kafka table engine
 
-<CloudAvailableBadge/>
-
 The Kafka table engine can be used to [**read** data from](#kafka-to-clickhouse) and [**write** data to](#clickhouse-to-kafka) Apache Kafka and other Kafka API-compatible brokers (e.g., Redpanda, Amazon MSK).
 
 ### Kafka to ClickHouse {#kafka-to-clickhouse}

scripts/badger.sh

@@ -1,5 +1,4 @@
 #!/bin/bash
-
 # This script is used to generate a list of feature badges in use in the docs
 # It runs as part of a weekly github action and the result is sent as a notification
 # to the #docs channel in Slack.
@@ -19,33 +18,41 @@ components=(
     "ExperimentalBadge"
     "PrivatePreviewBadge"
     "CloudNotSupportedBadge"
-    "CloudAvailableBadge"
+    "CloudOnlyBadge"
+)
+
+# Custom display names (must match order of components array)
+display_names=(
+    "Beta Features"
+    "Experimental Features"
+    "Private Preview Features"
+    "Cloud Unsupported Features"
+    "Cloud Only Features"
 )
 
 # Function to extract slug from a file's frontmatter
 extract_slug_from_file() {
     local filepath="$1"
     local slug=""
-
     # Look for "slug: some/path/slug" in the file
     slug=$(grep -m 1 "^slug:" "$filepath" 2>/dev/null | sed 's/^slug:[[:space:]]*//' | tr -d '"' | tr -d "'")
-
     # If no slug found, return the filepath as fallback
     if [ -z "$slug" ]; then
         slug="[no slug] $filepath"
     fi
-
     echo "$slug"
 }
 
 # Search for each component and collect all slugs
-for component in "${components[@]}"; do
-    echo "$component:"
-
+for i in "${!components[@]}"; do
+    component="${components[$i]}"
+    display_name="${display_names[$i]}"
+    
+    echo "$display_name:"
     # Get unique files containing the component
     files=$(grep -rl --include="*.md" --include="*.mdx" --include="*.jsx" --include="*.tsx" \
         -E "<$component[[:space:]/>]|</$component>" "$DOCS_DIR" 2>/dev/null | sort -u)
-
+    
     if [ -z "$files" ]; then
         echo "  (none)"
     else
@@ -60,6 +67,5 @@ for component in "${components[@]}"; do
             fi
         done <<< "$files"
     fi
-
     echo
 done

scripts/settings/session-settings.sql

@@ -58,7 +58,7 @@ WITH
         name,
         ' {#'||name||'} \n\n',
         multiIf(tier == 'Experimental', '<ExperimentalBadge/>\n\n', tier == 'Beta', '<BetaBadge/>\n\n', ''),
-        if(description LIKE '%Only has an effect in ClickHouse Cloud%', '<CloudAvailableBadge/>\n\n', ''),
+        if(description LIKE '%Only has an effect in ClickHouse Cloud%', '<CloudOnlyBadge/>\n\n', ''),
         if(
             type != '' AND default != '',
             format(
@@ -83,7 +83,7 @@ description: ''Settings which are found in the ``system.settings`` table.''
 
 import ExperimentalBadge from \'@theme/badges/ExperimentalBadge\';
 import BetaBadge from \'@theme/badges/BetaBadge\';
-import CloudAvailableBadge from \'@theme/badges/CloudAvailableBadge\';
+import CloudOnlyBadge from \'@theme/badges/CloudOnlyBadge\';
 import SettingsInfoBlock from \'@theme/SettingsInfoBlock/SettingsInfoBlock\';
 import VersionHistory from \'@theme/VersionHistory/VersionHistory\';
 

src/theme/badges/CloudAvailableBadge/index.js renamed to src/theme/badges/CloudOnlyBadge/index.js

@@ -12,12 +12,12 @@ const Icon = () => {
     )
 }
 
-const CloudAvailableBadge = () => {
+const CloudOnlyBadge = () => {
     return (
         <div className={styles.cloudBadge}>
             <Icon />{'ClickHouse Cloud only'}
         </div>
     )
 }
 
-export default CloudAvailableBadge
+export default CloudOnlyBadge