Skip to content

Commit dc22894

Browse files
RoRoJRoRoJ
authored
feat(kafka): add doc for Kafka private beta (#5443)
1 parent ff9d32d commit dc22894

File tree

9 files changed

+541
-0
lines changed

9 files changed

+541
-0
lines changed

‎pages/clusters-for-kafka/concepts.mdx‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
---
2+
title: Clusters for Apache Kafka® - Concepts
3+
description: Understand the key concepts for Clusters for Apache Kafka®.
4+
tags: billing account pricing payment
5+
dates:
6+
posted: 2025-09-15
7+
validation: 2025-09-15
8+
noindex: true
9+
---
10+
11+
<Message type="note">
12+
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
13+
</Message>
14+
15+
## Cluster
16+
17+
A cluster is a deployment of Apache Kafka®. It consists of one or more nodes (called **brokers**) that work together to store, manage and distribute event streams. [Data replication](#data-replication) across the cluster ensures high availability and robustness.
18+
19+
## Clusters for Apache Kafka®
20+
21+
Clusters for Apache Kafka® is Scaleway's product to provide a secure, fully-managed infrastructure for your Apache Kafka® deployments. With our clusters, you can create data pipelines and event-driven architectures that run natively on Apache Kafka®, without having to manage the underlying infrastructure.
22+
23+
## Credentials
24+
25+
Credentials consist of a username and password that allow you to connect to your cluster. These are defined during cluster creation. During Private Beta, a limit of one user (i.e. one set of credentials) per cluster is applied.
26+
27+
## Data replication
28+
29+
Data replication is the mechanism that ensures event data is copied across multiple brokers in a cluster, for fault tolerance and high availability. If a node fails, data replication means Apache Kafka® can keep serving data without downtime or data loss.
30+
31+
## Endpoint
32+
33+
During Private Beta, access to your Cluster for Apache Kafka® is via private endpoints only. When creating your cluster, you must attach it to a [Private Network](/vpc/concepts/#private-networks), and all nodes in the cluster receive a private IPv4 endpoint on this network. Endpoints are in a format similar to `bf610fb0-f7eb-4ec3-a893...634aea.kafka.fr-par.scw.cloud`.
34+
35+
## Event streaming
36+
37+
Clusters for Apache Kafka® is specifically designed for implementing event streaming at scale. Event streaming is the process of capturing, storing and processing a continuous flow of events in real time. Each event represents something that happened, e.g.`User added item to cart`, `User clicked 'Pay now'`, `Payment was rejected by bank`. Events are stored in topics, e.g. `Inventory`, `Payment`, `Orders`. Typically, programs and microservices subscribe to these topics to consume their events and take action in relation to them, e.g. `Generate invoice`, `Check for fraud`, `Program delivery`.
38+
39+
## Node
40+
41+
A Cluster for Apache Kafka® consists of one or more nodes, also called **brokers**. Nodes work together to store, manage, and distribute event streams. When creating a cluster, you can choose between **dedicated** nodes or **shared** nodes, based on your workload requirements, budget, and use case:
42+
43+
- **Dedicated node**: Runs only your cluster, with no other customers sharing the broker's resources.
44+
- **Shared node**: Runs your cluster along with other customers', on the same broker.
45+
46+
## Storage
47+
48+
A cluster's storage is used to persist events, so they can be reliably consumed by services and applications. During Private Beta, each Cluster for Apache Kafka® has a fixed amount of 100 GB of storage per node.
49+
50+
## TLS certificate
51+
52+
A TLS certificate (aka SSL certificate) is a digital credential that encrypts data in transit and verifies the identity of a server or client. Scaleway generates a TLS certificate for each Cluster for Apache Kafka®, and all clients connecting to the cluster must install this certificate in order to be authenticated. This ensures secure and encrypted communication between nodes (brokers) and clients.
53+
54+
## Users
55+
56+
See [credentials](#credentials).
57+

‎pages/clusters-for-kafka/faq.mdx‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: Clusters for Apache Kafka® FAQ
3+
description: Get answers to all your questions about Scaleway Clusters for Apache Kafka®
4+
dates:
5+
validation: 2025-09-15
6+
posted: 2025-09-15
7+
productIcon: KafkaProductIcon
8+
noindex: true
9+
---
10+
11+
<Message type="note">
12+
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
13+
</Message>
14+
15+
## General
16+
17+
### What is Clusters for Apache Kafka®?
18+
19+
Clusters for Apache Kafka® is Scaleway's product to provide a secure, fully-managed infrastructure for your Apache Kafka® deployments. With our clusters, you can create data pipelines and event-driven architectures that run natively on Apache Kafka®, without having to manage the underlying infrastructure. This product is currently in Private Beta.
20+
21+
### How do I use my cluster, and how can I connect to it?
22+
23+
To use your cluster, you first [connect](/clusters-for-kafka/how-to/connect-to-cluster/) via the Kafka CLI or any Kafka-supported programming language. The connection is secured, and requires the appropriate credentials, certificate, and endpoint provided by Scaleway. All configuration is done through a Kafka client, allowing you to manage topics, producers, and consumers easily. This flexibility ensures seamless integration with your applications and workflows.
24+
25+
### What limitations apply during the Private Beta?
26+
27+
Private Beta is aimed at providing an early version of a future product, to allow users to test and validate it during its development. The following limitations apply to Clusters for Apache Kafka® in its current Private Beta phase:
28+
29+
- Only selected users can test the product ([request an invitation](https://www.scaleway.com/en/betas/)).
30+
- Only one Kafka version is supported.
31+
- Clusters can only be provisioned with either one or three nodes (brokers).
32+
- The number of nodes cannot be modified after creation of the cluster.
33+
- Storage is fixed at 100 GB per node.
34+
- Connectivity to the cluster is facilitated exclusively through Private Networks.
35+
- Only one set of credentials (username and password) can be created per cluster.
36+
- Each Scaleway Organization in the Private Beta may have a maximum of 5 clusters at one time.
37+
38+
We aim to improve and develop the product based on feedback received during the beta stage.
39+
40+
### Is Scaleway's Clusters for Apache Kafka® based on the open-source Kafka framework?
41+
42+
Yes, Scaleway's Clusters for Apache Kafka is built on the open-source Apache Kafka framework. We use the Kafka API to ensure compatibility and leverage the robust features of the open-source ecosystem.
43+
44+
## How is Apache Kafka® different to RabbitMQ and NATS?
45+
46+
Apache Kafka, RabbitMQ, and NATS are all messaging systems but serve different purposes and have distinct architectures.
47+
48+
- Apache Kafka is designed for high-throughput, real-time data streaming, and is topic-based, using the pull model. It can process large amounts of data in real-time with high concurrency and fault tolerance, and is ideal for stream processing, event sourcing, and big data applications.
49+
- RabbitMQ is a versatile message broker that supports multiple protocols, and offers strong support for message delivery guarantees and reliability. It uses the push model, and is well-suited for traditional message queuing and task distribution.
50+
- [NATS](/nats/) is a lightweight, high-performance subject-based messaging system focused on speed and simplicity, making it great for microservices.
51+
52+
Each system has its own strengths and is suited to different use cases depending on the requirements of the application.
53+
54+
## Billing
55+
56+
### How am I billed for Clusters for Apache Kafka®?
57+
58+
Clusters for Apache Kafka® are free of charge during Private Beta, and you will not be billed.
59+
After the Private Beta, we are considering a usage-based pricing model that will be competitive with current market offerings.
60+

‎pages/clusters-for-kafka/how-to/connect-to-cluster.mdx‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
title: How to connect to a Cluster for Apache Kafka®
3+
description: Learn how to connect to your Scaleway Cluster for Apache Kafka® with Scaleway, and understand how to securely establish a connection using TLS certificates and credentials.
4+
dates:
5+
validation: 2025-09-15
6+
posted: 2025-09-15
7+
noindex: true
8+
---
9+
import Requirements from '@macros/iam/requirements.mdx'
10+
11+
<Message type="note">
12+
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
13+
</Message>
14+
15+
You can use different frameworks to connect to your Cluster for Apache Kafka®, and integrate it with your applications.
16+
17+
All clients that want to connect to your cluster must:
18+
19+
- Have downloaded the cluster's [TLS certificate](/clusters-for-kafka/how-to/manage-cluster/) and configured the client to recognize it
20+
- Be attached to a Private Network in the same VPC as the cluster, or connect via a Public Gateway in the VPC
21+
- Know the cluster's username and password
22+
23+
<Requirements />
24+
25+
- A Scaleway account logged into the [console](https://console.scaleway.com)
26+
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
27+
- A Cluster for Apache Kafka®
28+
29+
## How to download the TLS certificate
30+
31+
1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.
32+
33+
2. Click the cluster you want to connect to. Its **Overview** page displays.
34+
35+
3. In the **Connectivity** panel, click **Download** next to the TLS certificate.
36+
37+
The TLS certificate in `.pem` format is downloaded to your local device. If using Java, import it into a truststore and configure your `client.properties` appropriately. For other types of client (e.g. Python, Go, CLI), you may be able to use the certificate directly by setting its location in your Kafka consumer configuration code.
38+
39+
<Message type="note">
40+
TLS is server-side only, and no client certificate is required.
41+
</Message>
42+
43+
## How to ensure clients have network access
44+
45+
Any clients wanting to connect to the cluster must be [attached to a Private Network](/vpc/how-to/attach-resources-to-pn/) in the same VPC as the cluster.
46+
47+
## How to access the username and password
48+
49+
The username and password for a Cluster for Apache Kafka® are defined when [creating the cluster](/clusters-for-kafka/how-to/create-cluster/). During Private Beta, only one user can be created per cluster. The username cannot be modified after creation, but the password can be changed at any time.
50+
51+
If you need a reminder of the username after creating the cluster, follow the steps below. You cannot view the password you set, but you can change it to a new one.
52+
53+
1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.
54+
55+
2. Click the cluster you want to connect to. Its **Overview** page displays.
56+
57+
3. Click the **Users** tab.
58+
59+
The username displays in the list of users. Click **Change password** if you have forgotten the password and need to change it to a new one.
60+
61+
## How to connect to the cluster
62+
63+
Once you have ensured that the client in question has the cluster's TLS certificate, appropriate network access and the correct username and password, you can facilitate connection via the Kafka-compatible framework of your choice. Code snippets are provided for Go, Python and Java; access them by following the steps below.
64+
65+
1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.
66+
67+
2. Click the cluster you want to connect to. Its **Overview** page displays.
68+
69+
3. In the **Connectivity** panel, click **Connect**.
70+
71+
A pop-up displays, with tabs for each sample framework: **Go**, **Python**, **Java**, **Node.js**, and **Mongoose**.
72+
73+
4. Click the tab of your choice to view the code sample.
74+

‎pages/clusters-for-kafka/how-to/create-cluster.mdx‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
title: How to create a Cluster for Apache Kafka®
3+
description: Learn how to create a Cluster for Apache Kafka® with Scaleway, and get your hosted, fully-managed Kafka infrastructure up and running with just a few clicks.
4+
dates:
5+
validation: 2025-09-15
6+
posted: 2025-09-15
7+
noindex: true
8+
---
9+
import Requirements from '@macros/iam/requirements.mdx'
10+
11+
<Message type="note">
12+
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
13+
</Message>
14+
15+
This page shows you how to create a Cluster for Apache Kafka® via the Scaleway console.
16+
17+
<Requirements />
18+
19+
- A Scaleway account logged into the [console](https://console.scaleway.com)
20+
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
21+
22+
## How to create a Cluster for Apache Kafka®
23+
24+
1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. The **Clusters for Apache Kafka®a** product page displays.
25+
26+
2. Click **Create cluster**. The creation wizard displays.
27+
28+
3. Select a **region** for the cluster. During Private Beta, clusters are only available in the Paris region.
29+
30+
4. Select a **Kafka® version** to run on the cluster. Only one pre-selected Kafka® version is available during Private Beta.
31+
32+
5. Select the **number of nodes** (brokers) you want in the cluster, either **3 nodes** for a highly-available cluster enabling data replication, or **1 node** for a cluster suitable for testing and development. During Private Beta, the number of nodes cannot be changed after cluster creation.
33+
34+
6. Select a **node type**. Choose between [dedicated or shared nodes](/clusters-for-kafka/concepts/#node), depending on your workload requirements and use case. For shared nodes, various types are available, each offering different cost-performance ratios.
35+
36+
7. Configure **network connectivity**. During Private Beta, connectivity to clusters is facilitated exclusively through **Private Networks**. Select a Private Network from the dropdown list to attach your cluster to.
37+
38+
8. Create **credentials** by defining a username and password. These will be used when connecting to your cluster. During Private Beta, only one user can be created per cluster, and the username is not modifiable after creation (though the password can be changed).
39+
40+
9. Enter a **name**, and (optionally), **tags** to identify your cluster.
41+
42+
10. Check the estimated cost summary, then click **Create cluster**.
43+
44+
Your cluster is created, and you are taken to its **Overview** page in the console.

‎pages/clusters-for-kafka/how-to/delete-cluster.mdx‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
title: How to delete to a Cluster for Apache Kafka®
3+
description: Learn how to safely delete your Scaleway Cluster for Apache Kafka® with Scaleway in just a few clicks.
4+
dates:
5+
validation: 2025-09-15
6+
posted: 2025-09-15
7+
noindex: true
8+
---
9+
import Requirements from '@macros/iam/requirements.mdx'
10+
11+
<Message type="note">
12+
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
13+
</Message>
14+
15+
You can delete your cluster at any time by following the steps on this page.
16+
17+
<Requirements />
18+
19+
- A Scaleway account logged into the [console](https://console.scaleway.com)
20+
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
21+
- A Cluster for Apache Kafka®
22+
23+
## How to delete a cluster
24+
25+
1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.
26+
27+
2. Click the cluster you want to delete. Its **Overview** page displays.
28+
29+
3. Click the **Settings** tab.
30+
31+
4. Click **Delete cluster**.
32+
33+
A pop-up displays, informing you that this action will permanently delete the cluster, and all data stored on its nodes will be lost.
34+
35+
5. Type **DELETE** in the box, and click **Delete cluster**.
36+
37+
The cluster is deleted, and you are returned to the product page.

‎pages/clusters-for-kafka/how-to/index.mdx‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
title: Clusters for Apache Kafka® - How Tos
3+
description: Clusters for Apache Kafka® How Tos
4+
noindex: true
5+
---

‎pages/clusters-for-kafka/how-to/manage-cluster.mdx‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
---
2+
title: How to manage to a Cluster for Apache Kafka®
3+
description: Learn how to manage to your Scaleway Cluster for Apache Kafka® with Scaleway, including editing its details, renewing its TLS certificate and more.
4+
dates:
5+
validation: 2025-09-15
6+
posted: 2025-09-15
7+
noindex: true
8+
---
9+
import Requirements from '@macros/iam/requirements.mdx'
10+
11+
<Message type="note">
12+
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
13+
</Message>
14+
15+
After creating your cluster, you can renew its TLS certificate, change the attached Private Network, and change its password.
16+
17+
Other management actions for internally configuring your cluster, such as creating topics and events, and configuring partitions and replication factors, can be carried out via a Kafka-compatible client library or a Kafka CLI.
18+
19+
<Requirements />
20+
21+
- A Scaleway account logged into the [console](https://console.scaleway.com)
22+
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
23+
- A Cluster for Apache Kafka®
24+
25+
## How to renew the TLS certificate
26+
27+
You may want or need to renew the cluster's TLS certificate for reasons of security, compliance or if the certificate expires. Follow the steps below to renew the certificate:
28+
29+
1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.
30+
31+
2. Click the cluster whose certificate you want to renew. Its **Overview** page displays.
32+
33+
3. In the **Connectivity** panel, click **Renew** next to the TLS certificate.
34+
35+
A pop-up displays, warning you that you will not be able to connect to your cluster using the previous certificate, and you will need to download the new certificate for all clients.
36+
37+
4. Click **Renew TLS certificate** to confirm.
38+
39+
The certificate is renewed, and you are returned to the cluster's **Overview** page. Remember to download the new certificate for all clients that want to connect to the cluster.
40+
41+
## How to change the attached Private Network
42+
43+
In order to facilitate connectivity to your cluster, it must always be attached to a Private Network. A cluster can only be attached to one Private Network at a time, but you can change the attached Private Network as follows:
44+
45+
1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.
46+
47+
2. Click the cluster whose certificate you want to renew. Its **Overview** page displays.
48+
49+
3. In the **Network** panel, click **Detach Private Network**.
50+
51+
A pop-up displays, asking you to confirm the action.
52+
53+
4. Click **Detach resource** to confirm.
54+
55+
The cluster is detached, and you are returned to its **Overview** page. Click **Attach Private Network** to choose a new Private Network to attach it to. Remember that the cluster is not accessible unless it is attached to a Private Network.
56+
57+
## How to change the password
58+
59+
During Private Beta, each cluster can have only one set of user credentials, which are set during cluster creation. You cannot change the username after creating the cluster, but you can change the password at any time:
60+
61+
1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.
62+
63+
2. Click the cluster whose password you want to change. Its **Overview** page displays.
64+
65+
3. Click the **Users** tab.
66+
67+
4. Click **Change password** next to the cluster's username credential.
68+
69+
A pop-up displays, prompting you to change the password. Either enter the new password of your choice, or click **Generate password** to generate a new random password.
70+
71+
Ensure that you copy or remember the password, as it will not be displayed again after this screen.
72+
73+
5. Click **Update**.
74+
75+
The password is changed and you are returned to your cluster's **Overview** page.

0 commit comments

Comments
 (0)