Skip to content

Commit a81e79e

Browse files
thomas-tacquetthomas-tacquet
committed
feat(jobs):snapshot cleaner
1 parent 87c9569 commit a81e79e

File tree

5 files changed

+219
-0
lines changed

5 files changed

+219
-0
lines changed

‎jobs/instances-snapshot-cleaner/Dockerfile‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Using apline/golang image
2+
FROM golang:1.23-alpine
3+
4+
# Set destination for COPY
5+
WORKDIR /app
6+
7+
# Copy required files
8+
COPY go.mod ./
9+
COPY go.sum ./
10+
COPY *.go ./
11+
12+
# Build the executable
13+
RUN go build -o /jobs-snapshot-cleaner
14+
15+
# Run the executable
16+
CMD [ "/jobs-snapshot-cleaner" ]

‎jobs/instances-snapshot-cleaner/README.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Serverless Jobs for cleaning old snapshots
2+
3+
This project shows how it's possible to automate tasks using Serverless Jobs.
4+
5+
This simple example shows how to clean up snapshots after X days, it's useful to avoid a growing list of snapshots.
6+
7+
# Set-up
8+
9+
## Requirements
10+
11+
- Scaleway Account
12+
- Docker daemon running to build the image
13+
- Container registry namespace created, for this example we assume that your namespace name is `jobs-snapshot-cleaner`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/)
14+
15+
## Step 1 : Build and push to Container registry
16+
17+
Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works
18+
with containers. So first, use your terminal reach this folder and run the following commands:
19+
20+
```shell
21+
# First command is to login to container registry, you can find it in Scaleway console
22+
docker login rg.fr-par.scw.cloud/jobs-snapshot-cleaner -u nologin --password-stdin <<< "$SCW_SECRET_KEY"
23+
24+
# Here we build the image to push
25+
docker build -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 .
26+
27+
## TIP: for Apple Silicon or other ARM processors, please use the following command as Serverless Jobs supports amd64 architecture
28+
# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 .
29+
30+
# Push the image online to be used on Serverless Jobs
31+
docker push rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1
32+
```
33+
> [!TIP]
34+
> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case.
35+
36+
To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry.
37+
38+
## Step 2: Creating the Job Definition
39+
40+
On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par
41+
42+
1. On Container image, select the image you created in the step before.
43+
1. You can set the image name to something clear like `jobs-snapshot-cleaner` too.
44+
1. For the region you can select the one you prefer :)
45+
1. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory.
46+
1. To schedule your job for example every two days at 2am, you can set the cron to `0 2 */2 * *`.
47+
1. Important: advanced option, you need to set the following environment variables:
48+
49+
> [!TIP]
50+
> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/).
51+
52+
- `SCW_DELETE_AFTER_DAYS`: number of days after the snapshots will be deleted
53+
- `SCW_PROJECT_ID`: project you want to clean up
54+
- `SCW_ZONE`: you need to give the ZONE of your snapshot you want to clean, like `fr-par-2`
55+
- `SCW_ACCESS_KEY`: your access key
56+
- `SCW_SECRET_KEY`: your secret key
57+
- `SCW_DEFAULT_ORGANIZATION_ID`: your organzation ID
58+
59+
* Then click "create job"
60+
61+
## Step 3: Run the job
62+
63+
On your created Job Definition, just click the button "Run Job" and within seconds it should be successful.
64+
65+
## Troubleshooting
66+
67+
If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error.
68+
69+
# Possible improvements
70+
71+
You can exercice by adding the following features:
72+
73+
- Add tags to exclude
74+
- Add alerts if a Job goes in error
75+
- Use secret manager
76+
77+
# Additional content
78+
79+
- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/)
80+
- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/)
81+
- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/)
82+
- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/)

‎jobs/instances-snapshot-cleaner/go.mod‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
module github.com/scaleway/serverless-examples/jobs/instances-snapshot
2+
3+
go 1.23
4+
5+
require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.31
6+
7+
require gopkg.in/yaml.v2 v2.4.0 // indirect

‎jobs/instances-snapshot-cleaner/go.sum‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
2+
github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
3+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.30 h1:yoKAVkEVwAqbGbR8n87rHQ1dulL25rKloGadb3vm770=
4+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.30/go.mod h1:sH0u6fq6x4R5M7WxkoQFY/o7UaiItec0o1LinLCJNq8=
5+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.31 h1:Fj7jPyu9TQjqfXcLylINK5PANSzOWXIX4QtGmfp67AY=
6+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.31/go.mod h1:kzh+BSAvpoyHHdHBCDhmSWtBc1NbLMZ2lWHqnBoxFks=
7+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
8+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
9+
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
10+
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=

‎jobs/instances-snapshot-cleaner/main.go‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"os"
6+
"strconv"
7+
"time"
8+
9+
"github.com/scaleway/scaleway-sdk-go/api/instance/v1"
10+
"github.com/scaleway/scaleway-sdk-go/scw"
11+
)
12+
13+
const (
14+
envOrgID = "SCW_DEFAULT_ORGANIZATION_ID"
15+
envAccessKey = "SCW_ACCESS_KEY"
16+
envSecretKey = "SCW_SECRET_KEY"
17+
envProjectID = "SCW_PROJECT_ID"
18+
envZone = "SCW_ZONE"
19+
20+
// envDeleteAfter name of env variable to deleter older images.
21+
envDeleteAfter = "SCW_DELETE_AFTER_DAYS"
22+
23+
// defaultDaysDeleteAfter is the default days value for older images to be deleted.
24+
defaultDaysDeleteAfter = int(90)
25+
)
26+
27+
func main() {
28+
fmt.Println("cleaning instances snapshots...")
29+
30+
// Create a Scaleway client with credentials from environment variables.
31+
client, err := scw.NewClient(
32+
// Get your organization ID at https://console.scaleway.com/organization/settings
33+
scw.WithDefaultOrganizationID(os.Getenv(envOrgID)),
34+
35+
// Get your credentials at https://console.scaleway.com/iam/api-keys
36+
scw.WithAuth(os.Getenv(envAccessKey), os.Getenv(envSecretKey)),
37+
38+
// Get more about our availability
39+
// zones at https://www.scaleway.com/en/docs/console/my-account/reference-content/products-availability/
40+
scw.WithDefaultRegion(scw.RegionFrPar),
41+
)
42+
if err != nil {
43+
panic(err)
44+
}
45+
46+
// Create SDK objects for Scaleway Instance product
47+
instanceAPI := instance.NewAPI(client)
48+
49+
deleteAfterDays := defaultDaysDeleteAfter
50+
51+
deleteAfterDaysVar := os.Getenv(envDeleteAfter)
52+
53+
if deleteAfterDaysVar != "" {
54+
deleteAfterDays, err = strconv.Atoi(deleteAfterDaysVar)
55+
if err != nil {
56+
panic(err)
57+
}
58+
}
59+
60+
if err := cleanSnapshots(deleteAfterDays, instanceAPI); err != nil {
61+
panic(err)
62+
}
63+
}
64+
65+
func cleanSnapshots(days int, instanceAPI *instance.API) error {
66+
snapshotsList, err := instanceAPI.ListSnapshots(&instance.ListSnapshotsRequest{
67+
Zone: scw.Zone(os.Getenv(envZone)),
68+
Project: scw.StringPtr(os.Getenv(envProjectID)),
69+
},
70+
scw.WithAllPages())
71+
if err != nil {
72+
return fmt.Errorf("error while listing snapshots %w", err)
73+
}
74+
75+
const hoursPerDay = 24
76+
77+
currentTime := time.Now()
78+
79+
for _, snapshot := range snapshotsList.Snapshots {
80+
if snapshot.State == instance.SnapshotStateAvailable && (currentTime.Sub(*snapshot.CreationDate).Hours()/hoursPerDay) > float64(days) {
81+
fmt.Printf("\nDeleting snapshot <%s>:%s created at: %s\n", snapshot.ID, snapshot.Name, snapshot.CreationDate.Format(time.RFC3339))
82+
83+
err := instanceAPI.DeleteSnapshot(&instance.DeleteSnapshotRequest{
84+
SnapshotID: snapshot.ID,
85+
Zone: snapshot.Zone,
86+
})
87+
if err != nil {
88+
return fmt.Errorf("error while deleting snapshot: %w", err)
89+
}
90+
}
91+
}
92+
93+
return nil
94+
}
95+
96+
func init() {
97+
mandatoryVariables := [...]string{envOrgID, envAccessKey, envSecretKey, envZone, envProjectID}
98+
99+
for idx := range mandatoryVariables {
100+
if os.Getenv(mandatoryVariables[idx]) == "" {
101+
panic("missing environment variable " + mandatoryVariables[idx])
102+
}
103+
}
104+
}

0 commit comments

Comments
 (0)