Skip to content

Commit 62b1a8b

Browse files
Update README (#40)
* Update README Motivation: This PR closes issue #28. Modifications: * add instructions on how to build librdkafka from source * add documentation for public interface * Linux: Change librdkafka installation Modifications: * changed Docker image Ubuntu release to Jammy * use the Confluence repository for installing librdkafka on Linux systems
1 parent 2ca5142 commit 62b1a8b

File tree

2 files changed

+127
-23
lines changed

2 files changed

+127
-23
lines changed

README.md

Lines changed: 119 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
1-
# 🚧WIP🚧: swift-kafka-gsoc
1+
# 🚧WIP🚧: SwiftKafka
22

3-
This package is currently under development as part of the Google Summer of Code 2022.
4-
It aims to provide a library to interact with Apache Kafka.
3+
SwiftKafka is a Swift Package in development that provides a convenient way to communicate with [Apache Kafka](https://kafka.apache.org) servers. The main goal was to create an API that leverages [Swift's new concurrency features](https://docs.swift.org/swift-book/LanguageGuide/Concurrency.html). Under the hood, this package uses the [`librdkafka`](github.com/edenhill/librdkafka) C library.
54

6-
## Installing Dependencies
5+
## Getting Started
6+
7+
### Installing Dependencies
78

89
Please make sure to have the [`librdkafka`](https://github.com/edenhill/librdkafka) library installed before building.
910

@@ -15,11 +16,15 @@ brew install librdkafka
1516

1617
#### Linux
1718

19+
The default `apt-get` package for the library is outdated. Therefore, we recommend installing [`librdkafka`](https://github.com/edenhill/librdkafka) from the [official Confluence package repository](https://docs.confluent.io/platform/current/installation/installing_cp/deb-ubuntu.html#get-the-software).
20+
1821
```bash
22+
wget -qO - http://packages.confluent.io/deb/7.2/archive.key | sudo apt-key add -
23+
sudo add-apt-repository "deb http://packages.confluent.io/deb/ $(lsb_release -cs) main"
1924
apt-get install librdkafka-dev
2025
```
2126

22-
## Building the Project
27+
### Building the Project
2328

2429
#### macOS
2530

@@ -47,12 +52,117 @@ env PKG_CONFIG_PATH=$(brew --prefix)/opt/[email protected]/lib/pkgconfig xed .
4752
swift build
4853
```
4954

50-
## 🚧WIP🚧: Docker
51-
52-
As part of developing for Linux, we also provide a Docker environment for this package. It only contains a proof-of-concept program but will be expanded in the future.
55+
### Docker
5356

54-
To run the proof-of-concept program, use:
57+
We also provide a Docker environment for this package. This will automatically start a local Kafka server and run the package tests.
5558

5659
```bash
5760
docker-compose -f docker/docker-compose.yaml run swift-kafka-gsoc
5861
```
62+
63+
## Overview
64+
65+
### Producer API
66+
67+
The `sendAsync(_:)` method of `KafkaProducer` returns a message-id that can later be used to identify the corresponding acknowledgement. Acknowledgements are received through the `acknowledgements` [`AsyncSequence`](https://developer.apple.com/documentation/swift/asyncsequence). Each acknowledgement indicates that producing a message was successful or returns an error.
68+
69+
```swift
70+
var config = KafkaConfig()
71+
try config.set("localhost:9092", forKey: "bootstrap.servers")
72+
73+
let producer = try await KafkaProducer(
74+
config: config,
75+
logger: .kafkaTest // Your logger here
76+
)
77+
78+
let messageID = try await producer.sendAsync(
79+
KafkaProducerMessage(
80+
topic: "topic-name",
81+
value: "Hello, World!"
82+
)
83+
)
84+
85+
for await acknowledgement in producer.acknowledgements {
86+
// Check if acknowledgement belongs to the sent message
87+
}
88+
89+
// Required
90+
await producer.shutdownGracefully()
91+
```
92+
93+
### Consumer API
94+
95+
After initializing the `KafkaConsumer` with a topic-partition pair to read from, messages can be consumed using the `messages` [`AsyncSequence`](https://developer.apple.com/documentation/swift/asyncsequence).
96+
97+
```swift
98+
var config = KafkaConfig()
99+
try config.set("localhost:9092", forKey: "bootstrap.servers")
100+
101+
let consumer = try KafkaConsumer(
102+
topic: "topic-name",
103+
partition: KafkaPartition(rawValue: 0),
104+
config: config,
105+
logger: .kafkaTest // Your logger here
106+
)
107+
108+
for await messageResult in consumer.messages {
109+
switch messageResult {
110+
case .success(let message):
111+
// Do something with message
112+
case .failure(let error):
113+
// Handle error
114+
}
115+
}
116+
```
117+
118+
#### Consumer Groups
119+
120+
SwiftKafka also allows users to subscribe to an array of topics as part of a consumer group.
121+
122+
```swift
123+
var config = KafkaConfig()
124+
try config.set("localhost:9092", forKey: "bootstrap.servers")
125+
126+
let consumer = try KafkaConsumer(
127+
topics: ["topic-name"],
128+
groupID: "example-group",
129+
config: config,
130+
logger: .kafkaTest // Your logger here
131+
)
132+
133+
for await messageResult in consumer.messages {
134+
switch messageResult {
135+
case .success(let message):
136+
// Do something with message
137+
case .failure(let error):
138+
// Handle error
139+
}
140+
}
141+
```
142+
143+
#### Manual commits
144+
145+
By default, the `KafkaConsumer` automatically commits message offsets after receiving the corresponding message. However, we allow users to disable this setting and commit message offsets manually.
146+
147+
```swift
148+
var config = KafkaConfig()
149+
try config.set("localhost:9092", forKey: "bootstrap.servers")
150+
try config.set("false", forKey: "enable.auto.commit")
151+
152+
let consumer = try KafkaConsumer(
153+
topics: ["topic-name"],
154+
groupID: "example-group",
155+
config: config,
156+
logger: .kafkaTest // Your logger here
157+
)
158+
159+
for await messageResult in consumer.messages {
160+
switch messageResult {
161+
case .success(let message):
162+
// Do something with message
163+
try await consumer.commitSync(message)
164+
case .failure(let error):
165+
// Handle error
166+
}
167+
}
168+
```

docker/Dockerfile

Lines changed: 8 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,15 @@
11
ARG swift_version=5.7
2-
ARG ubuntu_version=focal
2+
ARG ubuntu_version=jammy
33
ARG base_image=swift:$swift_version-$ubuntu_version
44
FROM $base_image
55

6-
# Build librdkafka from source because apt-get
7-
# only supports librdkafka-dev version 1.2.1
8-
9-
# Install librdkafka dependencies
10-
RUN apt-get update && apt-get install -y --no-install-recommends \
11-
make \
12-
gcc \
13-
libzstd-dev \
14-
libsasl2-dev \
15-
libssl-dev \
16-
zlib1g-dev
17-
RUN git clone --depth 1 https://github.com/edenhill/librdkafka
18-
RUN cd librdkafka && ./configure && make && make install && ldconfig
6+
# Use Confluent repository for downloading librdkafka-dev because
7+
# apt-get only supports librdkafka-dev version 1.2.1
8+
RUN apt-get update
9+
RUN apt-get install wget software-properties-common gnupg -y
10+
RUN wget -qO - http://packages.confluent.io/deb/7.2/archive.key | apt-key add -
11+
RUN add-apt-repository "deb http://packages.confluent.io/deb/ $(lsb_release -cs) main" -y
12+
RUN apt-get install librdkafka-dev -y
1913

2014
WORKDIR /swift-kafka-gsoc
2115

0 commit comments

Comments
 (0)