Skip to content
57 changes: 57 additions & 0 deletions pages/clusters-for-kafka/concepts.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
title: Clusters for Apache Kafka® - Concepts
description: Understand the key concepts for Clusters for Apache Kafka®.
tags: billing account pricing payment
dates:
posted: 2025-09-15
validation: 2025-09-15
noindex: true
---

<Message type="note">
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
</Message>

## Cluster

A cluster is a deployment of Apache Kafka®. It consists of one or more nodes (called **brokers**) that work together to store, manage and distribute event streams. [Data replication](#data-replication) across the cluster ensures high availability and robustness.

## Clusters for Apache Kafka®

Clusters for Apache Kafka® is Scaleway's product to provide a secure, fully-managed infrastructure for your Apache Kafka® deployments. With our clusters, you can create data pipelines and event-driven architectures that run natively on Apache Kafka®, without having to manage the underlying infrastructure.

## Credentials

Credentials consist of a username and password that allow you to connect to your cluster. These are defined during cluster creation. During Private Beta, a limit of one user (i.e. one set of credentials) per cluster is applied.

## Data replication

Data replication is the mechanism that ensures event data is copied across multiple brokers in a cluster, for fault tolerance and high availability. If a node fails, data replication means Apache Kafka® can keep serving data without downtime or data loss.

## Endpoint

During Private Beta, access to your Cluster for Apache Kafka® is via private endpoints only. When creating your cluster, you must attach it to a [Private Network](/vpc/concepts/#private-networks), and all nodes in the cluster receive a private IPv4 endpoint on this network. Endpoints are in a format similar to `bf610fb0-f7eb-4ec3-a893...634aea.kafka.fr-par.scw.cloud`.

## Event streaming

Clusters for Apache Kafka® is specifically designed for implementing event streaming at scale. Event streaming is the process of capturing, storing and processing a continuous flow of events in real time. Each event represents something that happened, e.g.`User added item to cart`, `User clicked 'Pay now'`, `Payment was rejected by bank`. Events are stored in topics, e.g. `Inventory`, `Payment`, `Orders`. Typically, programs and microservices subscribe to these topics to consume their events and take action in relation to them, e.g. `Generate invoice`, `Check for fraud`, `Program delivery`.

## Node

A Cluster for Apache Kafka® consists of one or more nodes, also called **brokers**. Nodes work together to store, manage, and distribute event streams. When creating a cluster, you can choose between **dedicated** nodes or **shared** nodes, based on your workload requirements, budget, and use case:

- **Dedicated node**: Runs only your cluster, with no other customers sharing the broker's resources.
- **Shared node**: Runs your cluster along with other customers', on the same broker.

## Storage

A cluster's storage is used to persist events, so they can be reliably consumed by services and applications. During Private Beta, each Cluster for Apache Kafka® has a fixed amount of 100 GB of storage per node.

## TLS certificate

A TLS certificate (aka SSL certificate) is a digital credential that encrypts data in transit and verifies the identity of a server or client. Scaleway generates a TLS certificate for each Cluster for Apache Kafka®, and all clients connecting to the cluster must install this certificate in order to be authenticated. This ensures secure and encrypted communication between nodes (brokers) and clients.

## Users

See [credentials](#credentials).

60 changes: 60 additions & 0 deletions pages/clusters-for-kafka/faq.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
title: Clusters for Apache Kafka® FAQ
description: Get answers to all your questions about Scaleway Clusters for Apache Kafka®
dates:
validation: 2025-09-15
posted: 2025-09-15
productIcon: KafkaProductIcon
noindex: true
---

<Message type="note">
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
</Message>

## General

### What is Clusters for Apache Kafka®?

Clusters for Apache Kafka® is Scaleway's product to provide a secure, fully-managed infrastructure for your Apache Kafka® deployments. With our clusters, you can create data pipelines and event-driven architectures that run natively on Apache Kafka®, without having to manage the underlying infrastructure. This product is currently in Private Beta.

### How do I use my cluster, and how can I connect to it?

To use your cluster, you first [connect](/clusters-for-kafka/how-to/connect-to-cluster/) via the Kafka CLI or any Kafka-supported programming language. The connection is secured, and requires the appropriate credentials, certificate, and endpoint provided by Scaleway. All configuration is done through a Kafka client, allowing you to manage topics, producers, and consumers easily. This flexibility ensures seamless integration with your applications and workflows.

### What limitations apply during the Private Beta?

Private Beta is aimed at providing an early version of a future product, to allow users to test and validate it during its development. The following limitations apply to Clusters for Apache Kafka® in its current Private Beta phase:

- Only selected users can test the product ([request an invitation](https://www.scaleway.com/en/betas/)).
- Only one Kafka version is supported.
- Clusters can only be provisioned with either one or three nodes (brokers).
- The number of nodes cannot be modified after creation of the cluster.
- Storage is fixed at 100 GB per node.
- Connectivity to the cluster is facilitated exclusively through Private Networks.
- Only one set of credentials (username and password) can be created per cluster.
- Each Scaleway Organization in the Private Beta may have a maximum of 5 clusters at one time.

We aim to improve and develop the product based on feedback received during the beta stage.

### Is Scaleway's Clusters for Apache Kafka® based on the open-source Kafka framework?

Yes, Scaleway's Clusters for Apache Kafka is built on the open-source Apache Kafka framework. We use the Kafka API to ensure compatibility and leverage the robust features of the open-source ecosystem.

## How is Apache Kafka® different to RabbitMQ and NATS?

Apache Kafka, RabbitMQ, and NATS are all messaging systems but serve different purposes and have distinct architectures.

- Apache Kafka is designed for high-throughput, real-time data streaming, and is topic-based, using the pull model. It can process large amounts of data in real-time with high concurrency and fault tolerance, and is ideal for stream processing, event sourcing, and big data applications.
- RabbitMQ is a versatile message broker that supports multiple protocols, and offers strong support for message delivery guarantees and reliability. It uses the push model, and is well-suited for traditional message queuing and task distribution.
- [NATS](/nats/) is a lightweight, high-performance subject-based messaging system focused on speed and simplicity, making it great for microservices.

Each system has its own strengths and is suited to different use cases depending on the requirements of the application.

## Billing

### How am I billed for Clusters for Apache Kafka®?

Clusters for Apache Kafka® are free of charge during Private Beta, and you will not be billed.
After the Private Beta, we are considering a usage-based pricing model that will be competitive with current market offerings.

74 changes: 74 additions & 0 deletions pages/clusters-for-kafka/how-to/connect-to-cluster.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
---
title: How to connect to a Cluster for Apache Kafka®
description: Learn how to connect to your Scaleway Cluster for Apache Kafka® with Scaleway, and understand how to securely establish a connection using TLS certificates and credentials.
dates:
validation: 2025-09-15
posted: 2025-09-15
noindex: true
---
import Requirements from '@macros/iam/requirements.mdx'

<Message type="note">
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
</Message>

You can use different frameworks to connect to your Cluster for Apache Kafka®, and integrate it with your applications.

All clients that want to connect to your cluster must:

- Have downloaded the cluster's [TLS certificate](/clusters-for-kafka/how-to/manage-cluster/) and configured the client to recognize it
- Be attached to a Private Network in the same VPC as the cluster, or connect via a Public Gateway in the VPC
- Know the cluster's username and password

<Requirements />

- A Scaleway account logged into the [console](https://console.scaleway.com)
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
- A Cluster for Apache Kafka®

## How to download the TLS certificate

1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.

2. Click the cluster you want to connect to. Its **Overview** page displays.

3. In the **Connectivity** panel, click **Download** next to the TLS certificate.

The TLS certificate in `.pem` format is downloaded to your local device. If using Java, import it into a truststore and configure your `client.properties` appropriately. For other types of client (e.g. Python, Go, CLI), you may be able to use the certificate directly by setting its location in your Kafka consumer configuration code.

<Message type="note">
TLS is server-side only, and no client certificate is required.
</Message>

## How to ensure clients have network access

Any clients wanting to connect to the cluster must be [attached to a Private Network](/vpc/how-to/attach-resources-to-pn/) in the same VPC as the cluster.

## How to access the username and password

The username and password for a Cluster for Apache Kafka® are defined when [creating the cluster](/clusters-for-kafka/how-to/create-cluster/). During Private Beta, only one user can be created per cluster. The username cannot be modified after creation, but the password can be changed at any time.

If you need a reminder of the username after creating the cluster, follow the steps below. You cannot view the password you set, but you can change it to a new one.

1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.

2. Click the cluster you want to connect to. Its **Overview** page displays.

3. Click the **Users** tab.

The username displays in the list of users. Click **Change password** if you have forgotten the password and need to change it to a new one.

## How to connect to the cluster

Once you have ensured that the client in question has the cluster's TLS certificate, appropriate network access and the correct username and password, you can facilitate connection via the Kafka-compatible framework of your choice. Code snippets are provided for Go, Python and Java; access them by following the steps below.

1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.

2. Click the cluster you want to connect to. Its **Overview** page displays.

3. In the **Connectivity** panel, click **Connect**.

A pop-up displays, with tabs for each sample framework: **Go**, **Python**, **Java**, **Node.js**, and **Mongoose**.

4. Click the tab of your choice to view the code sample.

44 changes: 44 additions & 0 deletions pages/clusters-for-kafka/how-to/create-cluster.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
title: How to create a Cluster for Apache Kafka®
description: Learn how to create a Cluster for Apache Kafka® with Scaleway, and get your hosted, fully-managed Kafka infrastructure up and running with just a few clicks.
dates:
validation: 2025-09-15
posted: 2025-09-15
noindex: true
---
import Requirements from '@macros/iam/requirements.mdx'

<Message type="note">
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
</Message>

This page shows you how to create a Cluster for Apache Kafka® via the Scaleway console.

<Requirements />

- A Scaleway account logged into the [console](https://console.scaleway.com)
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization

## How to create a Cluster for Apache Kafka®

1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. The **Clusters for Apache Kafka®a** product page displays.

2. Click **Create cluster**. The creation wizard displays.

3. Select a **region** for the cluster. During Private Beta, clusters are only available in the Paris region.

4. Select a **Kafka® version** to run on the cluster. Only one pre-selected Kafka® version is available during Private Beta.

5. Select the **number of nodes** (brokers) you want in the cluster, either **3 nodes** for a highly-available cluster enabling data replication, or **1 node** for a cluster suitable for testing and development. During Private Beta, the number of nodes cannot be changed after cluster creation.

6. Select a **node type**. Choose between [dedicated or shared nodes](/clusters-for-kafka/concepts/#node), depending on your workload requirements and use case. For shared nodes, various types are available, each offering different cost-performance ratios.

7. Configure **network connectivity**. During Private Beta, connectivity to clusters is facilitated exclusively through **Private Networks**. Select a Private Network from the dropdown list to attach your cluster to.

8. Create **credentials** by defining a username and password. These will be used when connecting to your cluster. During Private Beta, only one user can be created per cluster, and the username is not modifiable after creation (though the password can be changed).

9. Enter a **name**, and (optionally), **tags** to identify your cluster.

10. Check the estimated cost summary, then click **Create cluster**.

Your cluster is created, and you are taken to its **Overview** page in the console.
37 changes: 37 additions & 0 deletions pages/clusters-for-kafka/how-to/delete-cluster.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
---
title: How to delete to a Cluster for Apache Kafka®
description: Learn how to safely delete your Scaleway Cluster for Apache Kafka® with Scaleway in just a few clicks.
dates:
validation: 2025-09-15
posted: 2025-09-15
noindex: true
---
import Requirements from '@macros/iam/requirements.mdx'

<Message type="note">
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
</Message>

You can delete your cluster at any time by following the steps on this page.

<Requirements />

- A Scaleway account logged into the [console](https://console.scaleway.com)
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
- A Cluster for Apache Kafka®

## How to delete a cluster

1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.

2. Click the cluster you want to delete. Its **Overview** page displays.

3. Click the **Settings** tab.

4. Click **Delete cluster**.

A pop-up displays, informing you that this action will permanently delete the cluster, and all data stored on its nodes will be lost.

5. Type **DELETE** in the box, and click **Delete cluster**.

The cluster is deleted, and you are returned to the product page.
5 changes: 5 additions & 0 deletions pages/clusters-for-kafka/how-to/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
title: Clusters for Apache Kafka® - How Tos
description: Clusters for Apache Kafka® How Tos
noindex: true
---
75 changes: 75 additions & 0 deletions pages/clusters-for-kafka/how-to/manage-cluster.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
---
title: How to manage to a Cluster for Apache Kafka®
description: Learn how to manage to your Scaleway Cluster for Apache Kafka® with Scaleway, including editing its details, renewing its TLS certificate and more.
dates:
validation: 2025-09-15
posted: 2025-09-15
noindex: true
---
import Requirements from '@macros/iam/requirements.mdx'

<Message type="note">
Clusters for Apache Kafka® is currently in [Private Beta](https://www.scaleway.com/en/betas/) and available to selected testers only.
</Message>

After creating your cluster, you can renew its TLS certificate, change the attached Private Network, and change its password.

Other management actions for internally configuring your cluster, such as creating topics and events, and configuring partitions and replication factors, can be carried out via a Kafka-compatible client library or a Kafka CLI.

<Requirements />

- A Scaleway account logged into the [console](https://console.scaleway.com)
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
- A Cluster for Apache Kafka®

## How to renew the TLS certificate

You may want or need to renew the cluster's TLS certificate for reasons of security, compliance or if the certificate expires. Follow the steps below to renew the certificate:

1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.

2. Click the cluster whose certificate you want to renew. Its **Overview** page displays.

3. In the **Connectivity** panel, click **Renew** next to the TLS certificate.

A pop-up displays, warning you that you will not be able to connect to your cluster using the previous certificate, and you will need to download the new certificate for all clients.

4. Click **Renew TLS certificate** to confirm.

The certificate is renewed, and you are returned to the cluster's **Overview** page. Remember to download the new certificate for all clients that want to connect to the cluster.

## How to change the attached Private Network

In order to facilitate connectivity to your cluster, it must always be attached to a Private Network. A cluster can only be attached to one Private Network at a time, but you can change the attached Private Network as follows:

1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.

2. Click the cluster whose certificate you want to renew. Its **Overview** page displays.

3. In the **Network** panel, click **Detach Private Network**.

A pop-up displays, asking you to confirm the action.

4. Click **Detach resource** to confirm.

The cluster is detached, and you are returned to its **Overview** page. Click **Attach Private Network** to choose a new Private Network to attach it to. Remember that the cluster is not accessible unless it is attached to a Private Network.

## How to change the password

During Private Beta, each cluster can have only one set of user credentials, which are set during cluster creation. You cannot change the username after creating the cluster, but you can change the password at any time:

1. Click **Apache Kafka®** in the **Data and Analytics** section of the [console](https://console.scaleway.com/) side menu. A list of your clusters displays.

2. Click the cluster whose password you want to change. Its **Overview** page displays.

3. Click the **Users** tab.

4. Click **Change password** next to the cluster's username credential.

A pop-up displays, prompting you to change the password. Either enter the new password of your choice, or click **Generate password** to generate a new random password.

Ensure that you copy or remember the password, as it will not be displayed again after this screen.

5. Click **Update**.

The password is changed and you are returned to your cluster's **Overview** page.
Loading
Loading