Skip to content

Commit 57cf105

Browse files
thomas-tacquetthomas-tacquet
committed
feat(jobs): s3 snapshot utility
1 parent d957ca7 commit 57cf105

File tree

4 files changed

+282
-12
lines changed

4 files changed

+282
-12
lines changed

‎jobs/snapshot-s3-archiver/README.md‎

Lines changed: 133 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,133 @@
1-
# Serverless Jobs to move snapshots to S3
1+
# Scaleway Instance Snapshot Backup to S3
2+
3+
This project exports available Scaleway Instance snapshots to an S3-compatible bucket (e.g., Scaleway Object Storage), and optionally deletes the snapshot afterward if it's already backed up. It's designed to run as a **Scaleway Serverless Job**, making it ideal for automated, scheduled backups.
4+
5+
---
6+
7+
## 📦 Features
8+
9+
- Lists all available block storage snapshots in a project.
10+
- Checks if a snapshot with the same name already exists in the target bucket.
11+
- Exports missing snapshots to the bucket in `.qcow2` format.
12+
- Deletes local snapshot after successful export (if not already in bucket).
13+
- Uses environment variables for full configuration.
14+
- Built to run in a container on [Scaleway Serverless Jobs](https://www.scaleway.com/en/serverless-jobs/).
15+
16+
---
17+
18+
## ⚙️ Environment Variables
19+
20+
You must set the following environment variables when deploying the job:
21+
22+
| Variable | Description |
23+
|--------|-------------|
24+
| `SCW_DEFAULT_ORGANIZATION_ID` | Your Scaleway Organization ID (legacy; prefer project ID). |
25+
| `SCW_DEFAULT_PROJECT_ID` | Your Scaleway Project ID (preferred way to group resources). |
26+
| `SCW_ACCESS_KEY` | API access key (from IAM). |
27+
| `SCW_SECRET_KEY` | API secret key (from IAM). |
28+
| `SCW_ZONE` | Zone where your snapshots are located (e.g., `fr-par-1`). |
29+
| `SCW_BUCKET_NAME` | Name of the S3 bucket to store exported snapshots. |
30+
| `SCW_BUCKET_ENDPOINT` | S3 endpoint (e.g., `s3.fr-par.scw.cloud`). |
31+
32+
> 🔐 **Security Tip**: Use IAM API keys with minimal required permissions.
33+
34+
---
35+
36+
## 🛠️ Build & Deploy to Scaleway Serverless Jobs
37+
38+
### 1. Build the Docker Image
39+
40+
```bash
41+
docker build -t snapshot-s3-backup .
42+
```
43+
44+
### 2. Tag and Push to Scaleway Container Registry (or any registry)
45+
46+
```bash
47+
# Example using Scaleway CR
48+
docker tag snapshot-s3-backup fr-par.scw.cloud/your-registry/snapshot-s3-backup:v1
49+
docker push fr-par.scw.cloud/your-registry/snapshot-s3-backup:v1
50+
```
51+
52+
> Replace `your-registry` with your actual container registry name.
53+
54+
### 3. Create the Serverless Job
55+
56+
Use the Scaleway CLI or Console:
57+
58+
#### Using `scw` CLI:
59+
60+
```bash
61+
scw job create \
62+
name=backup-snapshots \
63+
image=fr-par.scw.cloud/your-registry/snapshot-s3-backup:v1 \
64+
memory-limit=512Mi \
65+
cpu-limit=500m \
66+
environment='{
67+
"SCW_DEFAULT_PROJECT_ID": "your-project-id",
68+
"SCW_ACCESS_KEY": "your-access-key",
69+
"SCW_SECRET_KEY": "your-secret-key",
70+
"SCW_ZONE": "fr-par-1",
71+
"SCW_BUCKET_NAME": "my-backup-bucket",
72+
"SCW_BUCKET_ENDPOINT": "s3.fr-par.scw.cloud"
73+
}'
74+
```
75+
76+
### 4. (Optional) Schedule the Job
77+
78+
Schedule it to run daily using a cron trigger:
79+
80+
```bash
81+
scw scheduler trigger create-cron \
82+
job-id=your-job-id \
83+
schedule="0 2 * * *" \
84+
name=daily-snapshot-backup
85+
```
86+
87+
This runs the job every day at 2 AM.
88+
89+
---
90+
91+
## 📁 Output Format
92+
93+
Each snapshot is exported as:
94+
```
95+
<snapshot-name>.qcow2
96+
```
97+
98+
Example:
99+
```
100+
my-server-disk-2025-04-05.qcow2
101+
```
102+
103+
---
104+
105+
## ✅ Example Use Case
106+
107+
Run nightly to:
108+
1. Export new snapshots to object storage.
109+
2. Clean up old snapshots once safely backed up.
110+
3. Reduce storage costs and improve disaster recovery.
111+
112+
---
113+
114+
## 🧪 Local Testing (Optional)
115+
116+
Set environment variables:
117+
118+
```bash
119+
export SCW_DEFAULT_PROJECT_ID=...
120+
export SCW_ACCESS_KEY=...
121+
export SCW_SECRET_KEY=...
122+
export SCW_ZONE=fr-par-1
123+
export SCW_BUCKET_NAME=my-backup-bucket
124+
export SCW_BUCKET_ENDPOINT=s3.fr-par.scw.cloud
125+
```
126+
127+
Run:
128+
129+
```bash
130+
go run main.go
131+
```
132+
133+
---

‎jobs/snapshot-s3-archiver/go.mod‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,31 @@
1-
module github.com/scaleway/serverless-examples/jobs/instances-snapshot
1+
module github.com/scaleway/serverless-examples/jobs/snapshot-s3-archiver
22

33
go 1.25
44

5-
require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.34
5+
require (
6+
github.com/minio/minio-go/v7 v7.0.95
7+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.35
8+
)
69

710
require (
11+
github.com/dustin/go-humanize v1.0.1 // indirect
12+
github.com/go-ini/ini v1.67.0 // indirect
13+
github.com/goccy/go-json v0.10.5 // indirect
14+
github.com/google/uuid v1.6.0 // indirect
15+
github.com/klauspost/compress v1.18.0 // indirect
16+
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
817
github.com/kr/pretty v0.3.1 // indirect
18+
github.com/minio/crc64nvme v1.1.1 // indirect
19+
github.com/minio/md5-simd v1.1.2 // indirect
20+
github.com/philhofer/fwd v1.2.0 // indirect
921
github.com/rogpeppe/go-internal v1.14.1 // indirect
22+
github.com/rs/xid v1.6.0 // indirect
23+
github.com/stretchr/testify v1.11.1 // indirect
24+
github.com/tinylib/msgp v1.4.0 // indirect
25+
golang.org/x/crypto v0.42.0 // indirect
26+
golang.org/x/net v0.44.0 // indirect
27+
golang.org/x/sys v0.36.0 // indirect
28+
golang.org/x/text v0.29.0 // indirect
1029
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
1130
gopkg.in/yaml.v2 v2.4.0 // indirect
1231
)

‎jobs/snapshot-s3-archiver/go.sum‎

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,62 @@
11
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
2+
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
3+
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
4+
github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
5+
github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
6+
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
7+
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
8+
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
9+
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
10+
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
11+
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
12+
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
13+
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
14+
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
15+
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
16+
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
17+
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
18+
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
219
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
320
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
421
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
522
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
623
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
724
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
825
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
26+
github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
27+
github.com/minio/crc64nvme v1.1.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
28+
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
29+
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
30+
github.com/minio/minio-go/v7 v7.0.95 h1:ywOUPg+PebTMTzn9VDsoFJy32ZuARN9zhB+K3IYEvYU=
31+
github.com/minio/minio-go/v7 v7.0.95/go.mod h1:wOOX3uxS334vImCNRVyIDdXX9OsXDm89ToynKgqUKlo=
32+
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
33+
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
934
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
35+
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
36+
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
1037
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
1138
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
1239
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
13-
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.34 h1:48+VFHsyVcAHIN2v1Ao9v1/RkjJS5AwctFucBrfYNIA=
14-
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.34/go.mod h1:zFWiHphneiey3s8HOtAEnGrRlWivNaxW5T6d5Xfco7g=
40+
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
41+
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
42+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.35 h1:8xfn1RzeI9yoCUuEwDy08F+No6PcKZGEDOQ6hrRyLts=
43+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.35/go.mod h1:47B1d/YXmSAxlJxUJxClzHR6b3T4M1WyCvwENPQNBWc=
44+
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
45+
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
46+
github.com/tinylib/msgp v1.4.0 h1:SYOeDRiydzOw9kSiwdYp9UcBgPFtLU2WDHaJXyHruf8=
47+
github.com/tinylib/msgp v1.4.0/go.mod h1:cvjFkb4RiC8qSBOPMGPSzSAx47nAsfhLVTCZZNuHv5o=
48+
golang.org/x/crypto v0.42.0 h1:chiH31gIWm57EkTXpwnqf8qeuMUi0yekh6mT2AvFlqI=
49+
golang.org/x/crypto v0.42.0/go.mod h1:4+rDnOTJhQCx2q7/j6rAN5XDw8kPjeaXEUR2eL94ix8=
50+
golang.org/x/net v0.44.0 h1:evd8IRDyfNBMBTTY5XRF1vaZlD+EmWx6x8PkhR04H/I=
51+
golang.org/x/net v0.44.0/go.mod h1:ECOoLqd5U3Lhyeyo/QDCEVQ4sNgYsqvCZ722XogGieY=
52+
golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k=
53+
golang.org/x/sys v0.36.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
54+
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
55+
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
1556
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
1657
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
1758
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
1859
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
1960
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
61+
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
62+
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

‎jobs/snapshot-s3-archiver/main.go‎

Lines changed: 83 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,16 @@
11
package main
22

33
import (
4+
"context"
45
"fmt"
56
"os"
7+
"slices"
68

79
"github.com/scaleway/scaleway-sdk-go/api/instance/v1"
810
"github.com/scaleway/scaleway-sdk-go/scw"
11+
12+
"github.com/minio/minio-go/v7"
13+
"github.com/minio/minio-go/v7/pkg/credentials"
914
)
1015

1116
// Environment variable constants used to configure the Scaleway API client.
@@ -29,11 +34,11 @@ const (
2934
// envBucket is a custom environment variable for specifying the name of an S3-compatible bucket.
3035
// This is not a standard Scaleway variable and is application-specific.
3136
envBucket = "SCW_BUCKET_NAME"
37+
38+
envBucketEndpoint = "SCW_BUCKET_ENDPOINT"
3239
)
3340

3441
func main() {
35-
fmt.Println("moving snapshots to s3...")
36-
3742
// Create a Scaleway client with credentials from environment variables.
3843
client, err := scw.NewClient(
3944
// Get your organization ID at https://console.scaleway.com/organization/settings
@@ -52,25 +57,50 @@ func main() {
5257
panic(err)
5358
}
5459

60+
fmt.Println("Initializing instance API...")
61+
5562
instanceAPI := instance.NewAPI(client)
5663

64+
fmt.Println("Reading all snapshots for the project...")
65+
5766
snapList, err := instanceAPI.ListSnapshots(&instance.ListSnapshotsRequest{}, scw.WithAllPages())
5867
if err != nil {
5968
panic(err)
6069
}
6170

62-
fmt.Printf("number of snapshots: %d\n", snapList.TotalCount)
71+
fmt.Println("Reading all snapshots already in the bucket...")
72+
73+
filesInBucket, err := listBucketFiles()
74+
if err != nil {
75+
panic(err)
76+
}
77+
78+
const snapshotExtension = ".qcow2"
6379

6480
for _, snapshot := range snapList.Snapshots {
65-
fmt.Printf("snap %s\n", snapshot.Name)
81+
fmt.Printf("Checking for snapshot %s\n", snapshot.Name)
6682

6783
if snapshot.State == instance.SnapshotStateAvailable {
68-
fmt.Printf("Exporting snapshot %s (ID: %s) to bucket %s...\n", snapshot.Name, snapshot.ID, os.Getenv(envBucket))
84+
// Check if file already exists in bucket
85+
if slices.Contains(filesInBucket, snapshot.Name+snapshotExtension) {
86+
fmt.Printf("File %s already exists in bucket, can delete the snapshot and skip it\n", snapshot.Name+snapshotExtension)
87+
88+
err = instanceAPI.DeleteSnapshot(&instance.DeleteSnapshotRequest{
89+
SnapshotID: snapshot.ID,
90+
})
91+
if err != nil {
92+
panic(err)
93+
}
94+
95+
continue
96+
}
97+
98+
fmt.Printf("File %s not present in the bucket, expording it to the bucket...\n", snapshot.Name+".qcow2")
6999

70100
snap, err := instanceAPI.ExportSnapshot(&instance.ExportSnapshotRequest{
71101
SnapshotID: snapshot.ID,
72102
Bucket: os.Getenv(envBucket),
73-
Key: snapshot.Name + ".qcow2",
103+
Key: snapshot.Name + snapshotExtension,
74104
})
75105
if err != nil {
76106
fmt.Printf("Failed to export snapshot %s: %v\n", snapshot.Name, err)
@@ -87,11 +117,57 @@ func main() {
87117

88118
// Check for mandatory variables before starting to work.
89119
func init() {
90-
mandatoryVariables := [...]string{envOrgID, envAccessKey, envSecretKey, envZone, envProjectID, envBucket}
120+
mandatoryVariables := [...]string{
121+
envOrgID,
122+
envAccessKey,
123+
envSecretKey,
124+
envZone,
125+
envProjectID,
126+
envBucket,
127+
envBucketEndpoint,
128+
}
91129

92130
for idx := range mandatoryVariables {
93131
if os.Getenv(mandatoryVariables[idx]) == "" {
94132
panic("missing environment variable " + mandatoryVariables[idx])
95133
}
96134
}
97135
}
136+
137+
func listBucketFiles() ([]string, error) {
138+
// Retrieve S3-compatible endpoint and credentials from environment
139+
endpoint := os.Getenv(envBucketEndpoint)
140+
accessKeyID := os.Getenv(envAccessKey)
141+
secretAccessKey := os.Getenv(envSecretKey)
142+
143+
// Create new MinIO client
144+
minioClient, err := minio.New(endpoint, &minio.Options{
145+
Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
146+
Secure: true,
147+
})
148+
if err != nil {
149+
return nil, err
150+
}
151+
152+
// Set up context and result slice
153+
ctx := context.Background()
154+
var files []string
155+
156+
// Channel to signal listing completion
157+
doneCh := make(chan struct{})
158+
defer close(doneCh)
159+
160+
// List all objects in the bucket
161+
for object := range minioClient.ListObjects(ctx, os.Getenv(envBucket), minio.ListObjectsOptions{
162+
Recursive: false,
163+
WithMetadata: true,
164+
}) {
165+
if object.Err != nil {
166+
return nil, object.Err
167+
}
168+
169+
files = append(files, object.Key)
170+
}
171+
172+
return files, nil
173+
}

0 commit comments

Comments
 (0)