Skip to content

Commit c2686df

Browse files
mimischimimischi
committed
Update Benchmark set up and persist benchmarks
We have recently updated our toolchain version to 5.9, but have never updated the benchmark suite to use 5.9 or newer. Also, it does not look like we ever commited a baseline to this repository. While at it, wiring up GitHub actions to run the benchmarks for every pull request, as well as on a merge into `main`. Re-uses a lot of the heavy lifting that was defined in `apple/swift-nio` already. Note that `nightly-main` appears broken at the moment. The Benchmark tool gets stuck, and initial debugging point to it never finishing `_sendAndAcknowledgeMessages` when producing a set of messages before every benchmark.
1 parent 4a74297 commit c2686df

File tree

33 files changed

+458
-134
lines changed

33 files changed

+458
-134
lines changed

‎.github/workflows/benchmarks.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Benchmarks
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
benchmark_package_path:
7+
type: string
8+
description: "Path to the directory containing the benchmarking package. Defaults to ."
9+
default: "."
10+
swift_package_arguments:
11+
type: string
12+
description: "Arguments to the switch package command invocation e.g. `--disable-sandbox`"
13+
linux_5_9_enabled:
14+
type: boolean
15+
description: "Boolean to enable the Linux 5.9 Swift version matrix job. Defaults to true."
16+
default: true
17+
linux_5_10_enabled:
18+
type: boolean
19+
description: "Boolean to enable the Linux 5.10 Swift version matrix job. Defaults to true."
20+
default: true
21+
linux_6_0_enabled:
22+
type: boolean
23+
description: "Boolean to enable the Linux 6.0 Swift version matrix job. Defaults to true."
24+
default: true
25+
linux_nightly_6_0_enabled:
26+
type: boolean
27+
description: "Boolean to enable the Linux nightly 6.0 Swift version matrix job. Defaults to true."
28+
default: true
29+
linux_nightly_main_enabled:
30+
type: boolean
31+
description: "Boolean to enable the Linux nightly main Swift version matrix job. Defaults to true."
32+
default: true
33+
34+
jobs:
35+
benchmarks:
36+
name: Benchmarks
37+
# Workaround https://github.com/nektos/act/issues/1875
38+
uses: ./.github/workflows/swift_matrix.yml
39+
with:
40+
name: "Benchmarks"
41+
matrix_linux_command: "apt-get update -y -q && apt-get install -y -q libjemalloc-dev && apt-get -y install libsasl2-dev && swift package --package-path ${{ inputs.benchmark_package_path }} ${{ inputs.swift_package_arguments }} benchmark baseline check --check-absolute-path ${{ inputs.benchmark_package_path }}/Thresholds/${SWIFT_VERSION}/"
42+
matrix_linux_5_9_enabled: ${{ inputs.linux_5_9_enabled }}
43+
matrix_linux_5_10_enabled: ${{ inputs.linux_5_10_enabled }}
44+
matrix_linux_6_0_enabled: ${{ inputs.linux_6_0_enabled }}
45+
matrix_linux_nightly_6_0_enabled: ${{ inputs.linux_nightly_6_0_enabled }}
46+
matrix_linux_nightly_main_enabled: ${{ inputs.linux_nightly_main_enabled }}

‎.github/workflows/cxx_interop.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
name: Cxx interop
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
linux_5_9_enabled:
7+
type: boolean
8+
description: "Boolean to enable the Linux 5.9 Swift version matrix job. Defaults to true."
9+
default: true
10+
linux_5_10_enabled:
11+
type: boolean
12+
description: "Boolean to enable the Linux 5.10 Swift version matrix job. Defaults to true."
13+
default: true
14+
linux_6_0_enabled:
15+
type: boolean
16+
description: "Boolean to enable the Linux 6.0 Swift version matrix job. Defaults to true."
17+
default: true
18+
linux_nightly_6_0_enabled:
19+
type: boolean
20+
description: "Boolean to enable the Linux nightly 6.0 Swift version matrix job. Defaults to true."
21+
default: true
22+
linux_nightly_main_enabled:
23+
type: boolean
24+
description: "Boolean to enable the Linux nightly main Swift version matrix job. Defaults to true."
25+
default: true
26+
27+
jobs:
28+
cxx-interop:
29+
name: Cxx interop
30+
# Workaround https://github.com/nektos/act/issues/1875
31+
uses: ./.github/workflows/swift_matrix.yml
32+
with:
33+
name: "Cxx interop"
34+
matrix_linux_command: "apt-get update -y -q && apt-get install -y -q jq && apt-get -y install libsasl2-dev && curl -s https://raw.githubusercontent.com/apple/swift-nio/main/scripts/check-cxx-interop-compatibility.sh | bash"
35+
matrix_linux_5_9_enabled: ${{ inputs.linux_5_9_enabled }}
36+
matrix_linux_5_10_enabled: ${{ inputs.linux_5_10_enabled }}
37+
matrix_linux_6_0_enabled: ${{ inputs.linux_6_0_enabled }}
38+
matrix_linux_nightly_6_0_enabled: ${{ inputs.linux_nightly_6_0_enabled }}
39+
matrix_linux_nightly_main_enabled: ${{ inputs.linux_nightly_main_enabled }}

‎.github/workflows/main.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,23 @@ on:
99
jobs:
1010
unit-tests:
1111
name: Unit tests
12+
# Workaround https://github.com/nektos/act/issues/1875
1213
uses: ./.github/workflows/unit_tests.yml
1314
with:
1415
linux_5_9_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -strict-concurrency=complete"
1516
linux_5_10_arguments_override: "--explicit-target-dependency-import-check error -Xswiftc -strict-concurrency=complete"
1617
linux_6_0_arguments_override: "--explicit-target-dependency-import-check error"
1718
linux_nightly_6_0_arguments_override: "--explicit-target-dependency-import-check error"
1819
linux_nightly_main_arguments_override: "--explicit-target-dependency-import-check error"
20+
21+
benchmarks:
22+
name: Benchmarks
23+
# Workaround https://github.com/nektos/act/issues/1875
24+
uses: ./.github/workflows/benchmarks.yml@main
25+
with:
26+
benchmark_package_path: "Benchmarks"
27+
28+
cxx-interop:
29+
name: Cxx interop
30+
# Workaround https://github.com/nektos/act/issues/1875
31+
uses: ./.github/workflows/cxx_interop.yml

‎.github/workflows/pull_request.yml‎

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -23,17 +23,14 @@ jobs:
2323
linux_nightly_6_0_arguments_override: "--explicit-target-dependency-import-check error"
2424
linux_nightly_main_arguments_override: "--explicit-target-dependency-import-check error"
2525

26+
benchmarks:
27+
name: Benchmarks
28+
# Workaround https://github.com/nektos/act/issues/1875
29+
uses: ./.github/workflows/benchmarks.yml
30+
with:
31+
benchmark_package_path: "Benchmarks"
32+
2633
cxx-interop:
2734
name: Cxx interop
28-
uses: apple/swift-nio/.github/workflows/swift_matrix.yml@main
29-
with:
30-
name: "Cxx interop"
31-
matrix_linux_command: "apt-get update -y -q && apt-get install -y -q jq && apt-get -y install libsasl2-dev && curl -s https://raw.githubusercontent.com/apple/swift-nio/main/scripts/check-cxx-interop-compatibility.sh | bash"
32-
matrix_linux_5_9_enabled: true
33-
matrix_linux_5_10_enabled: true
34-
matrix_linux_6_0_enabled: true
35-
matrix_linux_nightly_6_0_enabled: true
36-
matrix_linux_nightly_main_enabled: true
37-
matrix_windows_6_0_enabled: false
38-
matrix_windows_nightly_6_0_enabled: false
39-
matrix_windows_nightly_main_enabled: false
35+
# Workaround https://github.com/nektos/act/issues/1875
36+
uses: ./.github/workflows/cxx_interop.yml

‎.github/workflows/swift_matrix.yml‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
name: Matrix
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
name:
7+
type: string
8+
description: "The name of the workflow used for the concurrency group."
9+
required: true
10+
matrix_linux_command:
11+
type: string
12+
description: "The command of the current Swift version linux matrix job to execute."
13+
required: true
14+
matrix_linux_5_9_enabled:
15+
type: boolean
16+
description: "Boolean to enable the 5.9 Swift version matrix job. Defaults to true."
17+
default: true
18+
matrix_linux_5_9_container_image:
19+
type: string
20+
description: "Container image for the 5.9 Swift version matrix job. Defaults to matching Swift Ubuntu image."
21+
default: "swift:5.9-jammy"
22+
matrix_linux_5_9_command_override:
23+
type: string
24+
description: "The command of the 5.9 Swift version linux matrix job to execute."
25+
matrix_linux_5_10_enabled:
26+
type: boolean
27+
description: "Boolean to enable the 5.10 Swift version matrix job. Defaults to true."
28+
default: true
29+
matrix_linux_5_10_container_image:
30+
type: string
31+
description: "Container image for the 5.10 Swift version matrix job. Defaults to matching Swift Ubuntu image."
32+
default: "swift:5.10-jammy"
33+
matrix_linux_5_10_command_override:
34+
type: string
35+
description: "The command of the 5.10 Swift version linux matrix job to execute."
36+
matrix_linux_6_0_enabled:
37+
type: boolean
38+
description: "Boolean to enable the 6.0 Swift version matrix job. Defaults to true."
39+
default: true
40+
matrix_linux_6_0_container_image:
41+
type: string
42+
description: "Container image for the 6.0 Swift version matrix job. Defaults to matching Swift Ubuntu image."
43+
default: "swift:6.0-jammy"
44+
matrix_linux_6_0_command_override:
45+
type: string
46+
description: "The command of the 6.0 Swift version linux matrix job to execute."
47+
matrix_linux_nightly_6_0_enabled:
48+
type: boolean
49+
description: "Boolean to enable the nightly 6.0 Swift version matrix job. Defaults to true."
50+
default: true
51+
matrix_linux_nightly_6_0_container_image:
52+
type: string
53+
description: "Container image for the nightly 6.0 Swift version matrix job. Defaults to matching Swift Ubuntu image."
54+
default: "swiftlang/swift:nightly-6.0-jammy"
55+
matrix_linux_nightly_6_0_command_override:
56+
type: string
57+
description: "The command of the nightly 6.0 Swift version linux matrix job to execute."
58+
matrix_linux_nightly_main_enabled:
59+
type: boolean
60+
description: "Boolean to enable the nightly main Swift version matrix job. Defaults to true."
61+
default: true
62+
matrix_linux_nightly_main_container_image:
63+
type: string
64+
description: "Container image for the nightly main Swift version matrix job. Defaults to matching Swift Ubuntu image."
65+
default: "swiftlang/swift:nightly-main-jammy"
66+
matrix_linux_nightly_main_command_override:
67+
type: string
68+
description: "The command of the nightly main Swift version linux matrix job to execute."
69+
70+
# We are cancelling previously triggered workflow runs
71+
concurrency:
72+
group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.name }}
73+
cancel-in-progress: true
74+
75+
jobs:
76+
linux:
77+
name: Linux (${{ matrix.swift.swift_version }})
78+
runs-on: ubuntu-latest
79+
strategy:
80+
fail-fast: false
81+
matrix:
82+
# We are specifying only the major and minor of the docker images to automatically pick up the latest patch release
83+
swift:
84+
- image: ${{ inputs.matrix_linux_5_9_container_image }}
85+
swift_version: "5.9"
86+
enabled: ${{ inputs.matrix_linux_5_9_enabled }}
87+
- image: ${{ inputs.matrix_linux_5_10_container_image }}
88+
swift_version: "5.10"
89+
enabled: ${{ inputs.matrix_linux_5_10_enabled }}
90+
- image: ${{ inputs.matrix_linux_6_0_container_image }}
91+
swift_version: "6.0"
92+
enabled: ${{ inputs.matrix_linux_6_0_enabled }}
93+
- image: ${{ inputs.matrix_linux_nightly_6_0_container_image }}
94+
swift_version: "nightly-6.0"
95+
enabled: ${{ inputs.matrix_linux_nightly_6_0_enabled }}
96+
- image: ${{ inputs.matrix_linux_nightly_main_container_image }}
97+
swift_version: "nightly-main"
98+
enabled: ${{ inputs.matrix_linux_nightly_main_enabled }}
99+
container:
100+
image: ${{ matrix.swift.image }}
101+
steps:
102+
- name: Checkout repository
103+
if: ${{ matrix.swift.enabled }}
104+
uses: actions/checkout@v4
105+
with:
106+
persist-credentials: false
107+
submodules: true
108+
- name: Mark the workspace as safe
109+
if: ${{ matrix.swift.enabled }}
110+
# https://github.com/actions/checkout/issues/766
111+
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
112+
- name: Run matrix job
113+
if: ${{ matrix.swift.enabled }}
114+
env:
115+
SWIFT_VERSION: ${{ matrix.swift.swift_version }}
116+
COMMAND: ${{ inputs.matrix_linux_command }}
117+
COMMAND_OVERRIDE_5_9: ${{ inputs.matrix_linux_5_9_command_override }}
118+
COMMAND_OVERRIDE_5_10: ${{ inputs.matrix_linux_5_10_command_override }}
119+
COMMAND_OVERRIDE_6_0: ${{ inputs.matrix_linux_6_0_command_override }}
120+
COMMAND_OVERRIDE_NIGHTLY_6_0: ${{ inputs.matrix_linux_nightly_6_0_command_override }}
121+
COMMAND_OVERRIDE_NIGHTLY_MAIN: ${{ inputs.matrix_linux_nightly_main_command_override }}
122+
run: |
123+
apt-get -qq update && apt-get -qq -y install curl && apt-get -y install libsasl2-dev
124+
curl -s https://raw.githubusercontent.com/apple/swift-nio/main/scripts/check-matrix-job.sh | bash
125+
services:
126+
zookeeper:
127+
image: ubuntu/zookeeper
128+
kafka:
129+
image: ubuntu/kafka
130+
env:
131+
ZOOKEEPER_HOST: zookeeper
132+
env:
133+
KAFKA_HOST: kafka

‎.github/workflows/unit_tests.yml‎

Lines changed: 15 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -47,59 +47,18 @@ on:
4747
jobs:
4848
unit-tests:
4949
name: Unit tests
50-
runs-on: ubuntu-latest
51-
strategy:
52-
fail-fast: false
53-
matrix:
54-
# We are specifying only the major and minor of the docker images to automatically pick up the latest patch release
55-
swift:
56-
- image: "swift:5.9-jammy"
57-
swift_version: "5.9"
58-
enabled: ${{ inputs.linux_5_9_enabled }}
59-
- image: "swift:5.10-jammy"
60-
swift_version: "5.10"
61-
enabled: ${{ inputs.linux_5_10_enabled }}
62-
- image: "swift:6.0-jammy"
63-
swift_version: "6.0"
64-
enabled: ${{ inputs.linux_6_0_enabled }}
65-
- image: "swiftlang/swift:nightly-6.0-jammy"
66-
swift_version: "nightly-6.0"
67-
enabled: ${{ inputs.linux_nightly_6_0_enabled }}
68-
- image: "swiftlang/swift:nightly-main-jammy"
69-
swift_version: "nightly-main"
70-
enabled: ${{ inputs.linux_nightly_main_enabled }}
71-
steps:
72-
- name: Checkout repository
73-
if: ${{ matrix.swift.enabled }}
74-
uses: actions/checkout@v4
75-
with:
76-
persist-credentials: false
77-
submodules: true
78-
- name: Mark the workspace as safe
79-
if: ${{ matrix.swift.enabled }}
80-
# https://github.com/actions/checkout/issues/766
81-
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
82-
- name: Run matrix job
83-
if: ${{ matrix.swift.enabled }}
84-
env:
85-
SWIFT_VERSION: ${{ matrix.swift.swift_version }}
86-
COMMAND: "swift test"
87-
COMMAND_OVERRIDE_5_9: "swift test ${{ inputs.linux_5_9_arguments_override }}"
88-
COMMAND_OVERRIDE_5_10: "swift test ${{ inputs.linux_5_10_arguments_override }}"
89-
COMMAND_OVERRIDE_6_0: "swift test ${{ inputs.linux_6_0_arguments_override }}"
90-
COMMAND_OVERRIDE_NIGHTLY_6_0: "swift test ${{ inputs.linux_nightly_6_0_arguments_override }}"
91-
COMMAND_OVERRIDE_NIGHTLY_MAIN: "swift test ${{ inputs.linux_nightly_main_arguments_override }}"
92-
run: |
93-
apt-get -qq update && apt-get -qq -y install curl && apt-get -y install libsasl2-dev
94-
curl -s https://raw.githubusercontent.com/apple/swift-nio/main/scripts/check-matrix-job.sh | bash
95-
container:
96-
image: ${{ matrix.swift.image }}
97-
services:
98-
zookeeper:
99-
image: ubuntu/zookeeper
100-
kafka:
101-
image: ubuntu/kafka
102-
env:
103-
ZOOKEEPER_HOST: zookeeper
104-
env:
105-
KAFKA_HOST: kafka
50+
# Workaround https://github.com/nektos/act/issues/1875
51+
uses: ./.github/workflows/swift_matrix.yml
52+
with:
53+
name: "Unit tests"
54+
matrix_linux_command: "swift test"
55+
matrix_linux_5_9_enabled: ${{ inputs.linux_5_9_enabled }}
56+
matrix_linux_5_9_command_override: "swift test ${{ inputs.linux_5_9_arguments_override }}"
57+
matrix_linux_5_10_enabled: ${{ inputs.linux_5_10_enabled }}
58+
matrix_linux_5_10_command_override: "swift test ${{ inputs.linux_5_10_arguments_override }}"
59+
matrix_linux_6_0_enabled: ${{ inputs.linux_6_0_enabled }}
60+
matrix_linux_6_0_command_override: "swift test ${{ inputs.linux_6_0_arguments_override }}"
61+
matrix_linux_nightly_6_0_enabled: ${{ inputs.linux_nightly_6_0_enabled }}
62+
matrix_linux_nightly_6_0_command_override: "swift test ${{ inputs.linux_nightly_6_0_arguments_override }}"
63+
matrix_linux_nightly_main_enabled: ${{ inputs.linux_nightly_main_enabled }}
64+
matrix_linux_nightly_main_command_override: "swift test ${{ inputs.linux_nightly_main_arguments_override }}"

‎Benchmarks/Package.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// swift-tools-version: 5.7
1+
// swift-tools-version: 5.9
22
//===----------------------------------------------------------------------===//
33
//
44
// This source file is part of the swift-kafka-client open source project

‎Benchmarks/Thresholds/5.10/SwiftKafkaConsumerBenchmarks.SwiftKafkaConsumer_basic_consumer_messages_1000.p90.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"allocatedResidentMemory" : 77266944,
3+
"cpuTotal" : 200000000,
4+
"objectAllocCount" : 5549,
5+
"releaseCount" : 15168,
6+
"retainCount" : 7108,
7+
"retainReleaseDelta" : 2511,
8+
"throughput" : 2,
9+
"wallClock" : 695307500
10+
}

‎Benchmarks/Thresholds/5.10/SwiftKafkaConsumerBenchmarks.SwiftKafkaConsumer_with_offset_commit_messages_1000.p90.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
3+
}

‎Benchmarks/Thresholds/5.10/SwiftKafkaConsumerBenchmarks.librdkafka_basic_consumer_messages_1000.p90.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"allocatedResidentMemory" : 47382528,
3+
"cpuTotal" : 10000000,
4+
"objectAllocCount" : 16,
5+
"releaseCount" : 48,
6+
"retainCount" : 2,
7+
"retainReleaseDelta" : 30,
8+
"throughput" : 2,
9+
"wallClock" : 640572501
10+
}

0 commit comments

Comments
 (0)