Skip to content

Commit 44a1c12

Browse files
MagicloudMagicloud
committed
minio + sccache
1 parent 1d11ae6 commit 44a1c12

File tree

1 file changed

+173
-0
lines changed

1 file changed

+173
-0
lines changed

‎_posts/2025-12-27-host-minio-for-sccache.markdown‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
---
2+
layout: post
3+
title: "Host Minio for Sccache"
4+
date: 2025-12-27 13:48:37 +0800
5+
categories:
6+
---
7+
They say using [Sccache](https://github.com/mozilla/sccache) speeds up Rust project building. The tool caches compiling results, surely makes the building next time faster. Although the slowest step, linking, does not benefit from this.
8+
9+
While the tool is server-client style, it does not support starts the server once, running the clients every where. Giving sometimes I build in dev host, sometimes I build in containers, further usage of the tool, storage backend, is in order.
10+
11+
Sccache supports a few storage backends, to me, fake S3, AKA Minio, seems alright. Sccache supports both virtual host and sub path style of Minio bucket accessing. But with sub path, when some configurations are wrong, Sccache won't complaint but won't cache, either. So I chose virtual host style.
12+
13+
## Minio
14+
15+
First of all, setup Minio. I followed its [Github Readme](https://github.com/minio/operator/) to install the operator. Its website, seems messed up due to financial crisis.
16+
17+
Then generate a basic Minio setup by `kubectl kustomize github.com/minio/operator/examples/kustomization/base`. I modified a few things.
18+
19+
- Rewrite all `Secret`s to `SealedSecret`s, obviously.
20+
21+
- `metadata` of `Tenant` object, like names, labels, etc.
22+
23+
- `spec.env` of `Tenant` object.
24+
25+
Set `MINIO_DOMAIN` to Minio URL about to be used. Per doc, this should be set, but seems not necessary.
26+
27+
- `spec.pools` of `Tenant` object.
28+
29+
Since this will be run on K3S with single node, I cleared the `affinity` part, and set `servers` to 1. The `volumesPerServer` must be no less than 4.
30+
31+
- A `Ingress`
32+
33+
My Minio tenant name is `any`. TLS is not necessary. Hostname about `*.minio.magicloud.lan` is not necessary as well, at least with DNSMasq nameserver.
34+
35+
The first rule is for console WebUI. The second rule is for virtual host bucket accessing. The third rule is for API endpoint.
36+
37+
```yaml
38+
apiVersion: networking.k8s.io/v1
39+
kind: Ingress
40+
metadata:
41+
name: minio-any-console
42+
namespace: minio-tenant
43+
annotations:
44+
external-dns.alpha.kubernetes.io/hostname: "minio.magicloud.lan,minio-console.magicloud.lan.minio.magicloud.lan"
45+
cert-manager.io/issuer: step-issuer
46+
cert-manager.io/issuer-kind: StepClusterIssuer
47+
cert-manager.io/issuer-group: certmanager.step.sm
48+
spec:
49+
tls:
50+
- secretName: minio-tls
51+
hosts:
52+
- minio.magicloud.lan
53+
- minio-console.magicloud.lan
54+
- "*.minio.magicloud.lan"
55+
rules:
56+
- host: minio-console.magicloud.lan
57+
http:
58+
paths:
59+
- path: /
60+
pathType: Prefix
61+
backend:
62+
service:
63+
name: any-console
64+
port:
65+
number: 9090
66+
- host: "*.minio.magicloud.lan"
67+
http:
68+
paths:
69+
- path: /
70+
pathType: Prefix
71+
backend:
72+
service:
73+
name: any-hl
74+
port:
75+
number: 9000
76+
- host: "minio.magicloud.lan"
77+
http:
78+
paths:
79+
- path: /
80+
pathType: Prefix
81+
backend:
82+
service:
83+
name: any-hl
84+
port:
85+
number: 9000
86+
```
87+
88+
With Minio ready, run following Terraform code to create a bucket named `sccache`, a user named `sccache`, with password `sccache123`, and grant full-access to the bucket. Remember to replace minio/minio123 with the credential from the `Secret` when setup the tenant.
89+
90+
```hcl
91+
terraform {
92+
required_providers {
93+
minio = {
94+
source = "aminueza/minio"
95+
version = "3.12.0"
96+
}
97+
}
98+
}
99+
100+
provider "minio" {
101+
minio_server = "minio.magicloud.lan:443"
102+
minio_user = "minio"
103+
minio_password = "minio123"
104+
minio_ssl = true
105+
}
106+
107+
resource "minio_iam_user" "sccache" {
108+
name = "sccache"
109+
secret = "sccache123"
110+
}
111+
112+
resource "minio_s3_bucket" "sccache" {
113+
bucket = "sccache"
114+
}
115+
116+
resource "minio_iam_policy" "read-write-sccache" {
117+
name = "read-write-sccache"
118+
policy = data.minio_iam_policy_document.sccache.json
119+
}
120+
121+
resource "minio_iam_user_policy_attachment" "sccache" {
122+
user_name = minio_iam_user.sccache.id
123+
policy_name = minio_iam_policy.read-write-sccache.id
124+
}
125+
126+
data "minio_iam_policy_document" "sccache" {
127+
statement {
128+
effect = "Allow"
129+
actions = [
130+
"s3:GetBucketLocation",
131+
"s3:ListBucket",
132+
]
133+
resources = [
134+
"arn:aws:s3:::sccache",
135+
]
136+
}
137+
138+
statement {
139+
effect = "Allow"
140+
actions = [
141+
"s3:DeleteObject",
142+
"s3:GetObject",
143+
"s3:PutObject",
144+
]
145+
resources = [
146+
"arn:aws:s3:::sccache/*",
147+
]
148+
}
149+
}
150+
```
151+
152+
## Sccache
153+
154+
There are always some DNS issues with Musl, hence the static linked Sccache, which led to a few DNS/hostname setup above that I am not sure is necessary.
155+
156+
And to make this actually work, Sccache **must** be build to GNU target.
157+
158+
Then export following environments, and Sccache is ready to run. Confirm it by seeing data appears in the bucket, and no more data appears in local (`~/.cache/sccache/` by default).
159+
160+
```bash
161+
SCCACHE_BUCKET="sccache"
162+
SCCACHE_REGION="auto"
163+
SCCACHE_ENDPOINT="minio.magicloud.lan:443"
164+
SCCACHE_S3_ENABLE_VIRTUAL_HOST_STYLE="true"
165+
SCCACHE_S3_USE_SSL="true"
166+
SCCACHE_S3_SERVER_SIDE_ENCRYPTION="false"
167+
AWS_ACCESS_KEY_ID="sccache"
168+
AWS_SECRET_ACCESS_KEY="sccache123"
169+
```
170+
171+
## PS
172+
173+
This is generally how it was done. And of course, one may want to use Vault or similiar tools to completely hide the passwords passing around here.

0 commit comments

Comments
 (0)