Merge pull request #4941 from morsapaes/docs-s3-pipe

ClickPipes: restructure documentation for Object Storage pipes

Author: morsapaes

…1_create_clickpipe_for_object_storage.md …ipes/object-storage/_create_clickpipe.mddocs/integrations/data-ingestion/clickpipes/object-storage/01_create_clickpipe_for_object_storage.md renamed to docs/_snippets/clickpipes/object-storage/_create_clickpipe.md docs/integrations/data-ingestion/clickpipes/object-storage/01_create_clickpipe_for_object_storage.md renamed to docs/_snippets/clickpipes/object-storage/_create_clickpipe.md

@@ -1,17 +1,6 @@
----
-sidebar_label: 'Create your first object storage ClickPipe'
-description: 'Seamlessly connect your object storage to ClickHouse Cloud.'
-slug: /integrations/clickpipes/object-storage
-title: 'Creating your first object-storage ClickPipe'
-doc_type: 'guide'
-integration:
-  - support_level: 'core'
-  - category: 'clickpipes'
----
-
+import Image from '@theme/IdealImage';
 import cp_step0 from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step0.png';
-import cp_step1 from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step1.png';
-import cp_step2_object_storage from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step2_object_storage.png';
+import cp_step2 from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/cp_step2.png';
 import cp_step3_object_storage from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step3_object_storage.png';
 import cp_step4a from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step4a.png';
 import cp_step4a3 from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step4a3.png';
@@ -21,41 +10,38 @@ import cp_success from '@site/static/images/integrations/data-ingestion/clickpip
 import cp_remove from '@site/static/images/integrations/data-ingestion/clickpipes/cp_remove.png';
 import cp_destination from '@site/static/images/integrations/data-ingestion/clickpipes/cp_destination.png';
 import cp_overview from '@site/static/images/integrations/data-ingestion/clickpipes/cp_overview.png';
-import Image from '@theme/IdealImage';
-
-Object Storage ClickPipes provide a simple and resilient way to ingest data from Amazon S3, Google Cloud Storage, Azure Blob Storage, and DigitalOcean Spaces into ClickHouse Cloud. Both one-time and continuous ingestion are supported with exactly-once semantics.
 
-# Creating your first object storage ClickPipe {#creating-your-first-clickpipe}
+import S3DataSource from '@site/docs/_snippets/clickpipes/object-storage/amazon-s3/_1-data-source.md';
+import GCSSDataSource from '@site/docs/_snippets/clickpipes/object-storage/google-cloud-storage/_1-data-source.md';
+import ABSDataSource from '@site/docs/_snippets/clickpipes/object-storage/azure-blob-storage/_1-data-source.md';
 
-## Prerequisite {#prerequisite}
+<VerticalStepper type="numbered" headerLevel="h2">
 
-- You have familiarized yourself with the [ClickPipes intro](../index.md).
+## Select the data source {#1-select-the-data-source}
 
-## Navigate to data sources {#1-load-sql-console}
+**1.** In ClickHouse Cloud, select **Data sources** in the main navigation menu and click **Create ClickPipe**.
 
-In the cloud console, select the `Data Sources` button on the left-side menu and click on "Set up a ClickPipe"
+    <Image img={cp_step0} alt="Select imports" size="lg" border/>
 
-<Image img={cp_step0} alt="Select imports" size="lg" border/>
+{props.provider === 's3' && <S3DataSource />}
+{props.provider === 'gcs' && <GCSSDataSource />}
+{props.provider === 'abs' && <ABSDataSource />}
 
-## Select a data source {#2-select-data-source}
+## Setup your ClickPipe connection {#2-setup-your-clickpipe-connection}
 
-Select your data source.
+**1.** To setup a new ClickPipe, you must provide details on how to connect to and authenticate with your object storage service.
 
-<Image img={cp_step1} alt="Select data source type" size="lg" border/>
+{props.provider === 's3' && <S3DataSource />}
+{props.provider === 'gcs' && <GCSSDataSource />}
+{props.provider === 'abs' && <ABSDataSource />}
 
-## Configure the ClickPipe {#3-configure-clickpipe}
+**2.** Click **Incoming data**. ClickPipes will fetch metadata from your bucket for the next step.
 
-Fill out the form by providing your ClickPipe with a name, a description (optional), your IAM role or credentials, and bucket URL.
-You can specify multiple files using bash-like wildcards.
-For more information, [see the documentation on using wildcards in path](/integrations/clickpipes/object-storage/reference/#limitations).
-
-<Image img={cp_step2_object_storage} alt="Fill out connection details" size="lg" border/>
-
-## Select data format {#4-select-format}
+## Select data format {#3-select-data-format}
 
 The UI will display a list of files in the specified bucket.
 Select your data format (we currently support a subset of ClickHouse formats) and if you want to enable continuous ingestion.
-([More details below](/integrations/clickpipes/object-storage/reference/#continuous-ingest)).
+See the "continuous ingest" section in the overview page for more details.
 
 <Image img={cp_step3_object_storage} alt="Set data format and topic" size="lg" border/>
 
@@ -77,7 +63,7 @@ In that case, the UI will allow you to map fields from the source to the ClickHo
 <Image img={cp_step4b} alt="Use an existing table" size="lg" border/>
 
 :::info
-You can also map [virtual columns](../../sql-reference/table-functions/s3#virtual-columns), like `_path` or `_size`, to fields.
+You can also map [virtual columns](/sql-reference/table-functions/s3#virtual-columns), like `_path` or `_size`, to fields.
 :::
 
 ## Configure permissions {#6-configure-permissions}
@@ -107,5 +93,7 @@ As well as controls to remove the ClickPipe and display a summary of the ingest
 <Image img={cp_overview} alt="View overview" size="lg" border/>
 
 **Congratulations!** you have successfully set up your first ClickPipe.
-If this is a streaming ClickPipe, it will be continuously running, ingesting data in real-time from your remote data source.
+If this is a ClickPipe configure for continuous ingestion, it will be continuously running, ingesting data in real-time from your remote data source.
 Otherwise, it will ingest the batch and complete.
+
+</VerticalStepper>

docs/_snippets/clickpipes/object-storage/amazon-s3/_1-data-source.md

@@ -0,0 +1,10 @@
+import cp_step1 from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/cp_step1.png';
+import Image from '@theme/IdealImage';
+
+**2.** Click the **Amazon S3** tile. You can also use this tile to connect to other S3-compatible services not listed in the ClickPipes UI.
+
+    <Image img={cp_step1} alt="Select imports" size="lg" border/>
+
+    :::tip
+    Due to differences in URL formats and API implementations across object storage service providers, not all S3-compatible services are supported out-of-the-box. If you're running into issues with a service that is not listed under [supported data sources](/integrations/clickpipes/object-storage/s3/overview#supported-data-sources), please [reach out to our team](https://clickhouse.com/company/contact?loc=clickpipes).
+    :::

docs/_snippets/clickpipes/object-storage/amazon-s3/_2-connection.md

@@ -0,0 +1,14 @@
+import cp_step2 from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/cp_step2.png';
+import Image from '@theme/IdealImage';
+
+    <Image img={cp_step2} alt="Fill out connection details" size="lg" border/>
+
+    * **Authentication method**: the S3 ClickPipe supports [IAM credentials](/integrations/clickpipes/object-storage/amazon-s3/overview/#iam-credentials) (`Credentials`) and [IAM role-based authentication](/integrations/clickpipes/object-storage/amazon-s3/overview/#iam-role) (`IAM role`). See the [reference documentation](/integrations/clickpipes/object-storage/overview/#access-control) for guidance on authentication and permissions.
+
+    * **S3 file path**: the S3 ClickPipe expects a [virtual-hosted-style URI](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access).
+
+        ```bash
+        https://bucket-name.s3.region-code.amazonaws.com/key-name
+        ```
+
+        You can use POSIX wildcards to match multiple files or prefixes. See the [reference documentation](/integrations/clickpipes/object-storage/overview/#file-pattern-matching) for guidance on supported patterns.

docs/_snippets/clickpipes/object-storage/azure-blob-storage/_1-data-source.md

@@ -0,0 +1,6 @@
+import cp_step1 from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/cp_step1.png';
+import Image from '@theme/IdealImage';
+
+**2.** Click the **Azure Blob Storage** tile.
+
+    <Image img={cp_step1} alt="Select imports" size="lg" border/>

docs/_snippets/clickpipes/object-storage/google-cloud-storage/_1-data-source.md

@@ -0,0 +1,6 @@
+import cp_step1 from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/cp_step1.png';
+import Image from '@theme/IdealImage';
+
+**2.** Click the **Google Cloud Storage** tile.
+
+    <Image img={cp_step1} alt="Select imports" size="lg" border/>

docs/_snippets/clickpipes/object-storage/google-cloud-storage/_2-connection.md

@@ -0,0 +1,14 @@
+import cp_step2 from '@site/static/images/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/cp_step2.png';
+import Image from '@theme/IdealImage';
+
+    <Image img={cp_step2} alt="Fill out connection details" size="lg" border/>
+
+    * **Authentication method**: the ABS ClickPipe supports [HMAC credentials](/integrations/clickpipes/object-storage/azure-blob-storage/overview/#) (`Credentials`). See the [reference documentation](/integrations/clickpipes/object-storage/azure-blob-storage/overview/#access-control) for guidance on authentication and permissions.
+
+    * **GCS file path**: The GCS ClickPipe uses the Cloud Storage [XML API](https://docs.cloud.google.com/storage/docs/interoperability) for interoperability, which requires the `storage.googleapis.com` endpoint:
+
+        ```bash
+        https://storage.googleapis.com/bucket-name/key-name
+        ```
+
+        You can use POSIX wildcards to match multiple files or prefixes. See the [reference documentation](/integrations/clickpipes/object-storage/overview/#file-pattern-matching) for guidance on supported patterns.

docs/integrations/data-ingestion/clickpipes/kafka/index.md

@@ -15,9 +15,9 @@ keywords: ['Kafka ClickPipes', 'Apache Kafka', 'streaming ingestion', 'real-time
 <!--AUTOGENERATED_START-->
 | Page | Description |
 |-----|-----|
-| [Creating your first Kafka ClickPipe](/integrations/clickpipes/kafka/create-your-first-kafka-clickpipe) | Step-by-step guide to creating your first Kafka ClickPipe. |
-| [Schema registries for Kafka ClickPipe](/integrations/clickpipes/kafka/schema-registries) | How to integrate for ClickPipes with a schema registry for schema management |
 | [Reference](/integrations/clickpipes/kafka/reference) | Details supported formats, sources, delivery semantics, authentication and experimental features supported by Kafka ClickPipes |
-| [Best practices](/integrations/clickpipes/kafka/best-practices) | Details best practices to follow when working with Kafka ClickPipes |
+| [Schema registries for Kafka ClickPipe](/integrations/clickpipes/kafka/schema-registries) | How to integrate for ClickPipes with a schema registry for schema management |
+| [Creating your first Kafka ClickPipe](/integrations/clickpipes/kafka/create-your-first-kafka-clickpipe) | Step-by-step guide to creating your first Kafka ClickPipe. |
 | [Kafka ClickPipes FAQ](/integrations/clickpipes/kafka/faq) | Frequently asked questions about ClickPipes for Kafka |
+| [Best practices](/integrations/clickpipes/kafka/best-practices) | Details best practices to follow when working with Kafka ClickPipes |
 <!--AUTOGENERATED_END-->