|
| 1 | += ADR 0050 - Alternatives to Minio for S3 compatible Object Storage |
| 2 | +:adr_author: Simon Beck |
| 3 | +:adr_owner: Schedar |
| 4 | +:adr_reviewers: |
| 5 | +:adr_date: 2026-01-26 |
| 6 | +:adr_upd_date: 2026-01-26 |
| 7 | +:adr_status: draft |
| 8 | +:adr_tags: s3,object-storage |
| 9 | + |
| 10 | +include::partial$adr-meta.adoc[] |
| 11 | + |
| 12 | +[NOTE] |
| 13 | +.Summary |
| 14 | +==== |
| 15 | +Garage provides good performance and simplicity. |
| 16 | +Thanks to the operator the bootstrapping and forming of the cluster can be fully managed by K8s CRs and no additional provider is necessary. |
| 17 | +==== |
| 18 | + |
| 19 | +== Context |
| 20 | + |
| 21 | +https://github.com/minio/minio/issues/21714[Minio] as an open-source project is effectively unmaintained at this point. |
| 22 | + |
| 23 | +The main use-case for Minio is OpenShift clusters which reside on CSPs that don't provide their own S3 compatible ObjectStorage. |
| 24 | +It's mostly used for backups and logs. |
| 25 | + |
| 26 | +Features we need from the new solution: |
| 27 | +- IAM with ACLs |
| 28 | +- Basic S3 compatibility (compatible with Restic) |
| 29 | +- Clustered/HA mode, optional with EC instead of replication |
| 30 | +- Storage on local block devices |
| 31 | +- Object LifeCycle to delete files older than x days, for log retention |
| 32 | +- Kubernetes readiness: are there charts and operators to simplify operations on K8s? |
| 33 | + |
| 34 | +Additionally to the mentioned points above we will also evaluate the complexity and ease of AppCat integration. |
| 35 | + |
| 36 | +Complexity is how many moving parts a solution has. |
| 37 | +To get an objective measure for this, we check how many running pods are required for an HA cluster. |
| 38 | +This includes any auxiliary operators or controllers. |
| 39 | + |
| 40 | +AppCat Integration is about how it could get integrated into AppCat. |
| 41 | +It will be checked if a full fledged provider is necessary or if a composition would suffice. |
| 42 | +If the solution can be configured via K8s objects, then usually a provider would not be necessary. |
| 43 | +However, if API access is needed then a provider is required. |
| 44 | + |
| 45 | +Every solution will also undergo two different benchmarks done with `minio-warp`: |
| 46 | + |
| 47 | +- The default mixed benchmark, which will stress test the clusters with a mixed selection of operations for 5 minutes |
| 48 | +- An extreme list test with 1 million objects, this test checks how good the solution can handle a large amount of objects |
| 49 | + |
| 50 | +=== Solutions |
| 51 | + |
| 52 | +These solutions will be looked at: |
| 53 | + |
| 54 | +- https://github.com/seaweedfs/seaweedfs[SeaweedFS] |
| 55 | +- https://git.deuxfleurs.fr/Deuxfleurs/garage[Garage] |
| 56 | +- https://github.com/rook/rook[Rook-Ceph] |
| 57 | +- https://github.com/apache/ozone[Apache Ozone] |
| 58 | + |
| 59 | +Honorable mentions that don't meet the clustered/HA requirement: |
| 60 | + |
| 61 | +- RustFS, still a very alpha solution |
| 62 | +- VersityGW, could only do HA via RWX |
| 63 | + |
| 64 | +[cols=5] |
| 65 | +|=== |
| 66 | +|Criteria |
| 67 | +|SeaweedFS |
| 68 | +|Garage |
| 69 | +|Rook-Ceph |
| 70 | +|Apache Ozone |
| 71 | + |
| 72 | +|IAM |
| 73 | +|✅ |
| 74 | +|✅ footnote:[It's not standard S3 IAM, they have their own simplified system, but sufficient for our use-cases] |
| 75 | +|✅ |
| 76 | +|⚠️ (beta state) |
| 77 | + |
| 78 | +|S3 comp |
| 79 | +|✅ |
| 80 | +|✅ |
| 81 | +|✅ |
| 82 | +|✅ |
| 83 | + |
| 84 | +|HA |
| 85 | +|✅ (10+4 EC) |
| 86 | +|✅ (no EC) |
| 87 | +|✅ |
| 88 | +|✅ |
| 89 | + |
| 90 | +|Storage |
| 91 | +|✅ |
| 92 | +|✅ |
| 93 | +|✅ |
| 94 | +|✅ |
| 95 | + |
| 96 | +|LifeCycle |
| 97 | +|✅ |
| 98 | +|✅ |
| 99 | +|✅ |
| 100 | +|⚠️ (on the road map) |
| 101 | + |
| 102 | +|K8s readiness |
| 103 | +|✅ Charts |
| 104 | +|✅ https://github.com/rajsinghtech/garage-operator[Community Operator]/Helm Chart footnote:[The helm chart does not fully provision a working instance. Manual steps are required https://garagehq.deuxfleurs.fr/documentation/quick-start/#creating-a-cluster-layout[after applying the chart.]] |
| 105 | +|✅ Rook is an Operator |
| 106 | +|✅ Chart, but rudimentary |
| 107 | + |
| 108 | +|Complexity |
| 109 | +|13 pods |
| 110 | +|4 pods |
| 111 | +|12 pods (no HA) |
| 112 | +|12 pods |
| 113 | + |
| 114 | +|AppCat integration |
| 115 | +|Provider |
| 116 | +|Composition thanks to operator |
| 117 | +|Composition thanks to operator |
| 118 | +|Provider |
| 119 | + |
| 120 | +|=== |
| 121 | + |
| 122 | +=== Performance Benchmarks |
| 123 | + |
| 124 | +For completeness sake the benchmarks were also done with a Minio 4 node cluster. |
| 125 | + |
| 126 | +All these benchmarks were done on an M2 Macbook Pro with kind. |
| 127 | +Except for Rook Ceph, as it needs dedicated block storage, so minikube was used. |
| 128 | + |
| 129 | +==== Mixed |
| 130 | +Ran default `minio-warp mixed` against each cluster. |
| 131 | +The table contains the averages of each individual test. |
| 132 | + |
| 133 | +[cols=6] |
| 134 | +|=== |
| 135 | +|Solution |
| 136 | +|Delete |
| 137 | +|Get |
| 138 | +|Put |
| 139 | +|Stat |
| 140 | +|Total |
| 141 | + |
| 142 | +|Seaweedfs |
| 143 | +|6.70 obj/s |
| 144 | +|301.72 MiB/s |
| 145 | +|100.88 MiB/s |
| 146 | +|20.12 obj/s |
| 147 | +|402.60 MiB/s, 67.08 obj/s |
| 148 | + |
| 149 | +|Garage |
| 150 | +|11.95 obj/s |
| 151 | +|538.23 MiB/s |
| 152 | +|179.31 MiB/s |
| 153 | +|35.89 obj/s |
| 154 | +|717.55 MiB/s, 119.60 obj/s |
| 155 | + |
| 156 | +|Rook footnoteref:[rook, During the benchmark the OSD crashed and restarted. It still came back up. But it's evident that it would require more tweaking.] |
| 157 | +|0.15 obj/s |
| 158 | +|6.82 MiB/s |
| 159 | +|5.92 MiB/s |
| 160 | +|0.45 obj/s |
| 161 | +|9.10 MiB/s, 1.51 obj/s |
| 162 | + |
| 163 | +|Ozone footnoteref:[ozone, The cluster crashed unrecoverably] |
| 164 | +|Cluster crashed |
| 165 | +|Cluster crashed |
| 166 | +|Cluster crashed |
| 167 | +|Cluster crashed |
| 168 | +|Cluster crashed |
| 169 | + |
| 170 | +|Minio footnote:[Minio aggressively stores temp data to the point that it was the only solution that completely filled up the disk during the mixed test, using over 100Gb of storage] |
| 171 | +|10.26 obj/s |
| 172 | +|459.90 MiB/s |
| 173 | +|153.44 MiB/s |
| 174 | +|30.70 obj/s |
| 175 | +|613.34 MiB/s |
| 176 | + |
| 177 | +|=== |
| 178 | + |
| 179 | +==== List |
| 180 | + |
| 181 | +This test was to see if the solutions can handle a large amount of objects without failing. |
| 182 | +The test first creates 1 million small objects and then it will list them all. |
| 183 | + |
| 184 | +The command used was: `warp list --obj.size="1Ki" --objects=1000000 --concurrent=16` |
| 185 | + |
| 186 | +[cols=3] |
| 187 | +|=== |
| 188 | +|Solution |
| 189 | +|Creation AVG |
| 190 | +|List AVG |
| 191 | + |
| 192 | +|Seaweedfs |
| 193 | +|4572 obj/s |
| 194 | +|224930.38 obj/s |
| 195 | + |
| 196 | +|Garage |
| 197 | +|2877 obj/s |
| 198 | +|27694.61 obj/s |
| 199 | + |
| 200 | +|Rook footnoteref:[rook] |
| 201 | +|Did not run |
| 202 | +|Did not run |
| 203 | + |
| 204 | +|Ozone footnoteref:[ozone] |
| 205 | +|Did not run |
| 206 | +|Did not run |
| 207 | + |
| 208 | +|Minio |
| 209 | +|498 obj/s |
| 210 | +|4573 obj/s |
| 211 | + |
| 212 | +|=== |
| 213 | + |
| 214 | +While for mixed operations Garage and Seaweedfs provide solid performance, Garage is a clear winner overtaking Seaweedfs. |
| 215 | + |
| 216 | +Seaweedfs really shines with a lot of small objects. |
| 217 | +There it takes the crown by surpassing Garage almost 2 times for put and 10 times for list. |
| 218 | + |
| 219 | +==== Resource Usage |
| 220 | + |
| 221 | +While no in-depth analysis of the resource usage was made during the benchmarks, here are a few observations: |
| 222 | + |
| 223 | +- Generally all solutions ate all the CPU they could get during the benchmark stress testing |
| 224 | +- Garage was by far the least memory hungry with less than 200Mb usage during the stress test, idling at less than 20Mb |
| 225 | +- SeaweedFS and Rook ceph were somewhat on par with around 500Mb memory usage. Although Rook was not deployed with an HA cluster config |
| 226 | +- Ozone takes the last place with over 2Gb memory usage before crashing footnote:[The crashes don't seem memory related, there were stacktraces about missing objects.] |
| 227 | + |
| 228 | +== Decision |
| 229 | +Garage with the community operator. |
| 230 | + |
| 231 | +Garage's performance is overall pretty good and it can handle 1 million files. |
| 232 | +It's the least complex solution and offers good integration into AppCat via their operator. |
| 233 | + |
| 234 | + |
| 235 | +== Consequences |
| 236 | +A new composition for Garage needs to be implemented. |
| 237 | + |
| 238 | +As with Minio, it can't be a self-service product if integration with AppCat is required. This is due to needing specific `ObjectBucket` compositions for each instance. |
0 commit comments