Skip to content

Commit ea965d5

Browse files
KidswissKidswiss
committed
Add evaluation of minio alternatives
1 parent 532d8cb commit ea965d5

File tree

4 files changed

+244
-2
lines changed

4 files changed

+244
-2
lines changed

‎docs/modules/ROOT/pages/adr/0049-alternatives-to-minio-for-s3-compatible-object-storage.adoc‎

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
= ADR 0049 - Alternatives to Minio for S3 compatible Object Storage
2+
:adr_author: Simon Beck
3+
:adr_owner: Schedar
4+
:adr_reviewers:
5+
:adr_date: 2026-01-26
6+
:adr_upd_date: 2026-01-26
7+
:adr_status: draft
8+
:adr_tags: s3,object-storage
9+
10+
include::partial$adr-meta.adoc[]
11+
12+
[NOTE]
13+
.Summary
14+
====
15+
Garage provides good performance and simplicity.
16+
Thanks to the operator the bootstrapping and forming of the cluster can be fully managed by K8s CRs and no additional provider is necessary.
17+
====
18+
19+
== Context
20+
21+
https://github.com/minio/minio/issues/21714[Minio] as an open-source project is effectively unmaintained at this point.
22+
23+
The main use-case for Minio are OpenShift clusters which reside on CSPs that don't provide their own S3 compatible ObjectStorage.
24+
It's mostly used for backups and logs.
25+
26+
Features we need from the new solution:
27+
- IAM with ACLs
28+
- Basic S3 compatibility (compatible with Restic)
29+
- Clustered/HA mode, optional with EC instead of replication
30+
- Storage on local block devices
31+
- Object LifeCycle to delete files older than x days, for log retention
32+
- Kubernetes readiness, are the charts and operators to simplify operations on K8s
33+
34+
Additionally to the mentioned points above we will also evaluate the complexity and ease of AppCat integration.
35+
36+
Complexity is how many moving parts a solution has.
37+
To get an objective measure for this we check how many running pods are required for an HA cluster.
38+
This includes any auxiliary operators or controllers.
39+
40+
AppCat is about how it could get integrated into AppCat.
41+
It will be checked if a full fledged provider is necessary or if a composition would suffice.
42+
If the solution can be configured via K8s objects, then usually a provider would not be necessary.
43+
However, if API access is needed then a provider is required.
44+
45+
Every solution will also undergo two different benchmarks done with `minio-warp`:
46+
47+
- The default mixed benchmark, which will stress test the clusters with a mixed selection of operations for 5 minutes
48+
- An extreme list test with 1 million objects, this test checks how good the solution can handle a large amount of objects
49+
50+
=== Solutions
51+
52+
These solutions will be looked at:
53+
54+
- https://github.com/seaweedfs/seaweedfs[SeaweedFS]
55+
- https://git.deuxfleurs.fr/Deuxfleurs/garage[Garage]
56+
- https://github.com/rook/rook[Rook-Ceph]
57+
- https://github.com/apache/ozone[Apache Ozone]
58+
59+
Honorable mentions that don't meet the clustered/HA requirement:
60+
61+
- RustFS, still a very alpha solution
62+
- VersityGW, could only do HA via RWX
63+
64+
[cols=5]
65+
|===
66+
|Criteria
67+
|SeaweedFS
68+
|Garage
69+
|Rook-Ceph
70+
|Apache Ozone
71+
72+
|IAM
73+
|✅
74+
|✅ footnote:[It's not standard S3 IAM, they have their own simplified system, but sufficient for our use-cases]
75+
|✅
76+
|⚠️ (beta state)
77+
78+
|S3 comp
79+
|✅
80+
|✅
81+
|✅
82+
|✅
83+
84+
|HA
85+
|✅ (10+4 EC)
86+
|✅ (no EC)
87+
|✅
88+
|✅
89+
90+
|Storage
91+
|✅
92+
|✅
93+
|✅
94+
|✅
95+
96+
|LifeCycle
97+
|✅
98+
|✅
99+
|✅
100+
|⚠️ (on the road map)
101+
102+
|K8s readiness
103+
|✅ Charts
104+
|✅ Community Operator
105+
|✅ Rook is an Operator
106+
|✅ Chart, but rudimentary
107+
108+
|Complexity
109+
|13 pods
110+
|4 pods
111+
|12 pods (no HA)
112+
|12 pods
113+
114+
|AppCat integration
115+
|Provider
116+
|Composition thanks to operator
117+
|Composition thanks to operator
118+
|Provider
119+
120+
|===
121+
122+
=== Performance Benchmarks
123+
124+
For completeness sake the benchmarks were also done with a Minio 4 node cluster.
125+
126+
All these benchmarks were done on an M2 Macbook Pro with kind.
127+
Except for Rook Ceph, as it needs dedicated block storage, so minikube was used.
128+
129+
==== Mixed
130+
Ran default `minio-warp mixed` against each cluster.
131+
The table contains the averages of each individual test.
132+
133+
[cols=6]
134+
|===
135+
|Solution
136+
|Delete
137+
|Get
138+
|Put
139+
|Stat
140+
|Total
141+
142+
|Seaweedfs
143+
|6.70 obj/s
144+
|301.72 MiB/s
145+
|100.88 MiB/s
146+
|20.12 obj/s
147+
|402.60 MiB/s, 67.08 obj/s
148+
149+
|Garage
150+
|11.95 obj/s
151+
|538.23 MiB/s
152+
|179.31 MiB/s
153+
|35.89 obj/s
154+
|717.55 MiB/s, 119.60 obj/s
155+
156+
|Rook footnoteref:[rook, During the benchmark the OSD crashed and restarted. It still came back up. But it's evident that it would require more tweaking.]
157+
|0.15 obj/s
158+
|6.82 MiB/s
159+
|5.92 MiB/s
160+
|0.45 obj/s
161+
|9.10 MiB/s, 1.51 obj/s
162+
163+
|Ozone footnoteref:[ozone, The cluster crashed unrecoverably]
164+
|Cluster crashed
165+
|Cluster crashed
166+
|Cluster crashed
167+
|Cluster crashed
168+
|Cluster crashed
169+
170+
|Minio
171+
|10.26 obj/s
172+
|459.90 MiB/s
173+
|153.44 MiB/s
174+
|30.70 obj/s
175+
|613.34 MiB/s
176+
177+
|===
178+
179+
==== List
180+
181+
This test was to see if the solutions can handle a large amount of objects without failing.
182+
The test first creates 1 million small objects and then it will list them all.
183+
184+
The command used was: `warp list --obj.size="1Ki" --objects=1000000 --concurrent=16`
185+
186+
[cols=3]
187+
|===
188+
|Solution
189+
|Creation AVG
190+
|List AVG
191+
192+
|Seaweedfs
193+
|4572 obj/s
194+
|224930.38 obj/s
195+
196+
|Garage
197+
|2877 obj/s
198+
|27694.61 obj/s
199+
200+
|Rook footnoteref:[rook]
201+
|Did not run
202+
|Did not run
203+
204+
|Ozone footnoteref:[ozone]
205+
|Did not run
206+
|Did not run
207+
208+
|Minio
209+
|498 obj/s
210+
|4573 obj/s
211+
212+
|===
213+
214+
While for mixed operations Garage and Seaweedfs provide solid performance, Garage is a clear winner overtaking Seaweedfs.
215+
216+
Seaweedfs really shines with a lot of small objects.
217+
There it takes the crown by passing Garage almost 2 times for put and 10 times for list.
218+
219+
==== Resource Usage
220+
221+
While no in-depth analysis of the resource usage was made during the benchmarks, here are a few observations:
222+
223+
- Generally all solutions ate all the CPU they could get during the benchmark stress testing
224+
- Garage was by far the least memory hungry with less than 200Mb usage during the stress test, idling at less than 20Mb
225+
- SeaweedFS and Rook ceph where somewhat on par with around 500Mb memory usage. Although Rook was not deployed with an HA cluster config
226+
- Ozone takes the last place with over 2Gb memory usage before crashing footnote:[The crashes don't seem memory related, there were stacktraces about missing objects.]
227+
228+
== Decision
229+
Garage with the community operator.
230+
231+
Garage's performance is overall pretty good and it can handle 1 million files.
232+
It's the least complex solution and offers good integration into AppCat via their operator.
233+
234+
235+
== Consequences
236+
A new composition for Garage needs to be implemented.
237+
Because there's deep integration into AppCat itself it will usually not be a self-service product.

‎docs/modules/ROOT/pages/adr/index.adoc‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,4 +197,8 @@
197197

198198
`database,service`
199199
|draft | |2026-01-14
200+
|xref:adr/0049-alternatives-to-minio-for-s3-compatible-object-storage.adoc[]
201+
202+
`s3,object-storage`
203+
|draft | |2026-01-26
200204
|===

‎docs/modules/ROOT/partials/nav-adrs.adoc‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,4 +45,5 @@
4545
** xref:adr/0045-service-orchestration-crossplane-2-0.adoc[]
4646
** xref:adr/0046-secret-management-framework-2-0.adoc[]
4747
** xref:adr/0047-service-maintenance-and-upgrades-framework-2-0.adoc[]
48-
** xref:adr/0048-evaluating-vector-databases-as-appcat-services.adoc[]
48+
** xref:adr/0048-evaluating-vector-databases-as-appcat-services.adoc[]
49+
** xref:adr/0049-alternatives-to-minio-for-s3-compatible-object-storage.adoc[]

‎templates/adr/cookiecutter.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"adr_number": "0049",
2+
"adr_number": "0050",
33
"full_name": "VSHNeer Name",
44
"adr_title": "Title",
55
"adr_reviewers": "",

0 commit comments

Comments
 (0)