Skip to content

Commit ada415e

Browse files
Merge branch 'main' into jobs-backup-database-example
2 parents 35720c2 + 55dd9e3 commit ada415e

File tree

25 files changed

+925
-60
lines changed

25 files changed

+925
-60
lines changed

README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,9 @@ Table of Contents:
8383
| **[Serverless Jobs Hello World](jobs/terraform-hello-world/README.md)** <br/> An example of building a container image and running it as a Serverless Job using Terraform. | N/A | [Terraform]-[Console] |
8484
| **[Serverless MLOps](jobs/ml-ops/README.md)** <br/> An example of running a Serverless Machine Leaning workflow. | Python | [Terraform]-[Console]-[CLI] |
8585
| **[Auto Snapshot Instances](jobs/instances-snapshot/README.md)** <br/> Use Serverless Jobs to create snapshots of your instances | Go | [Console] |
86+
| **[Instance Snapshot Cleaner](jobs/instances-snapshot-cleaner/README.md)** <br/> Use Serverless Jobs to clean old instances snapshots | Go | [Console] |
87+
| **[Registry Tag Cleaner](jobs/registry-version-based-retention/README.md)** <br/> Use Serverless Jobs to keep a desired amount of tags for each image | Go | [Console] |
88+
| **[Registry Empty Image Cleaner](jobs/registry-empty-ressource-cleaner/README.md)** <br/> Use Serverless Jobs to clean container registry empty namespaces and images | Go | [Console] |
8689

8790
### 💬 Messaging and Queueing
8891

functions/cors-rust/src/lib.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ use axum::{body::Body, extract::Request, response::Response};
22
use http::StatusCode;
33

44
// See: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#the_http_response_headers
5-
pub fn with_permissive_cors(r: response::Builder) -> response::Builder {
5+
pub fn with_permissive_cors(r: http::response::Builder) -> http::response::Builder {
66
r.header("Access-Control-Allow-Headers", "*")
77
.header("Access-Control-Allow-Methods", "*")
88
.header("Access-Control-Allow-Origin", "*")
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Using apline/golang image
2+
FROM golang:1.24-alpine
3+
4+
# Set destination for COPY
5+
WORKDIR /app
6+
7+
# Copy required files
8+
COPY go.mod ./
9+
COPY go.sum ./
10+
COPY *.go ./
11+
12+
# Build the executable
13+
RUN go build -o /jobs-snapshot-cleaner
14+
15+
# Run the executable
16+
ENTRYPOINT [ "/jobs-snapshot-cleaner" ]
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Serverless Jobs for cleaning old snapshots
2+
3+
This project shows how it's possible to automate tasks using Serverless Jobs.
4+
5+
This simple example shows how to clean up snapshots after X days, it's useful to avoid a growing list of snapshots.
6+
7+
# Set-up
8+
9+
## Requirements
10+
11+
- Scaleway Account
12+
- Docker daemon running to build the image
13+
- Container registry namespace created, for this example we assume that your namespace name is `jobs-snapshot-cleaner`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/)
14+
- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/)
15+
16+
## Step 1 : Build and push to Container registry
17+
18+
Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works
19+
with containers. So first, use your terminal reach this folder and run the following commands:
20+
21+
```shell
22+
# First command is to login to container registry, you can find it in Scaleway console
23+
docker login rg.fr-par.scw.cloud/jobs-snapshot-cleaner -u nologin --password-stdin <<< "$SCW_SECRET_KEY"
24+
25+
# Here we build the image to push
26+
docker build -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 .
27+
28+
## TIP: for Apple Silicon or other ARM processors, please use the following command as Serverless Jobs supports amd64 architecture
29+
# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 .
30+
31+
# Push the image online to be used on Serverless Jobs
32+
docker push rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1
33+
```
34+
> [!TIP]
35+
> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case.
36+
37+
To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry.
38+
39+
## Step 2: Creating the Job Definition
40+
41+
On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par
42+
43+
1. On Container image, select the image you created in the step before.
44+
1. You can set the image name to something clear like `jobs-snapshot-cleaner` too.
45+
1. For the region you can select the one you prefer :)
46+
1. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory.
47+
1. To schedule your job for example every two days at 2am, you can set the cron to `0 2 */2 * *`.
48+
1. Important: advanced option, you need to set the following environment variables:
49+
50+
> [!TIP]
51+
> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/).
52+
53+
- `SCW_DELETE_AFTER_DAYS`: number of days after the snapshots will be deleted
54+
- `SCW_PROJECT_ID`: project you want to clean up
55+
- `SCW_ZONE`: you need to give the ZONE of your snapshot you want to clean, like `fr-par-2`
56+
- `SCW_ACCESS_KEY`: your access key
57+
- `SCW_SECRET_KEY`: your secret key
58+
- `SCW_DEFAULT_ORGANIZATION_ID`: your organzation ID
59+
60+
* Then click "create job"
61+
62+
## Step 3: Run the job
63+
64+
On your created Job Definition, just click the button "Run Job" and within seconds it should be successful.
65+
66+
## Troubleshooting
67+
68+
If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error.
69+
70+
# Possible improvements
71+
72+
You can exercice by adding the following features:
73+
74+
- Add tags to exclude
75+
- Add alerts if a Job goes in error
76+
- Use Secret Manager instead of job environment variables
77+
- Support multiple zones dans projects
78+
79+
# Additional content
80+
81+
- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/)
82+
- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/)
83+
- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/)
84+
- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/)
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
module github.com/scaleway/serverless-examples/jobs/instances-snapshot
2+
3+
go 1.24
4+
5+
require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32
6+
7+
require (
8+
github.com/kr/pretty v0.3.1 // indirect
9+
github.com/rogpeppe/go-internal v1.13.1 // indirect
10+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
11+
gopkg.in/yaml.v2 v2.4.0 // indirect
12+
)
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
2+
github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
3+
github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
4+
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
5+
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
6+
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
7+
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
8+
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
9+
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
10+
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
11+
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
12+
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
13+
github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
14+
github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
15+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 h1:4+LP7qmsLSGbmc66m1s5dKRMBwztRppfxFKlYqYte/c=
16+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32/go.mod h1:kzh+BSAvpoyHHdHBCDhmSWtBc1NbLMZ2lWHqnBoxFks=
17+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
18+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
19+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
20+
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
21+
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
package main
2+
3+
import (
4+
"encoding/json"
5+
"errors"
6+
"fmt"
7+
"os"
8+
"strconv"
9+
"time"
10+
11+
"github.com/scaleway/scaleway-sdk-go/api/instance/v1"
12+
"github.com/scaleway/scaleway-sdk-go/scw"
13+
)
14+
15+
const (
16+
envOrgID = "SCW_DEFAULT_ORGANIZATION_ID"
17+
envAccessKey = "SCW_ACCESS_KEY"
18+
envSecretKey = "SCW_SECRET_KEY"
19+
envProjectID = "SCW_PROJECT_ID"
20+
envZone = "SCW_ZONE"
21+
22+
// envDeleteAfter name of env variable to deleter older images.
23+
envDeleteAfter = "SCW_DELETE_AFTER_DAYS"
24+
25+
// defaultDaysDeleteAfter is the default days value for older images to be deleted.
26+
defaultDaysDeleteAfter = int(90)
27+
)
28+
29+
func main() {
30+
fmt.Println("cleaning instances snapshots...")
31+
32+
// Create a Scaleway client with credentials from environment variables.
33+
client, err := scw.NewClient(
34+
// Get your organization ID at https://console.scaleway.com/organization/settings
35+
scw.WithDefaultOrganizationID(os.Getenv(envOrgID)),
36+
37+
// Get your credentials at https://console.scaleway.com/iam/api-keys
38+
scw.WithAuth(os.Getenv(envAccessKey), os.Getenv(envSecretKey)),
39+
40+
// Get more about our availability
41+
// zones at https://www.scaleway.com/en/docs/console/my-account/reference-content/products-availability/
42+
scw.WithDefaultRegion(scw.RegionFrPar),
43+
)
44+
if err != nil {
45+
panic(err)
46+
}
47+
48+
// Create SDK objects for Scaleway Instance product
49+
instanceAPI := instance.NewAPI(client)
50+
51+
deleteAfterDays := defaultDaysDeleteAfter
52+
53+
deleteAfterDaysVar := os.Getenv(envDeleteAfter)
54+
55+
if deleteAfterDaysVar != "" {
56+
deleteAfterDays, err = strconv.Atoi(deleteAfterDaysVar)
57+
if err != nil {
58+
panic(err)
59+
}
60+
}
61+
62+
if err := cleanSnapshots(deleteAfterDays, instanceAPI); err != nil {
63+
var precondErr *scw.PreconditionFailedError
64+
65+
if errors.As(err, &precondErr) {
66+
fmt.Println("\nExtracted Error Details:")
67+
fmt.Println("Precondition:", precondErr.Precondition)
68+
fmt.Println("Help Message:", precondErr.HelpMessage)
69+
70+
// Decode RawBody (if available)
71+
if len(precondErr.RawBody) > 0 {
72+
var parsedBody map[string]interface{}
73+
if json.Unmarshal(precondErr.RawBody, &parsedBody) == nil {
74+
fmt.Println("RawBody (Decoded):", parsedBody)
75+
} else {
76+
fmt.Println("RawBody (Raw):", string(precondErr.RawBody))
77+
}
78+
}
79+
}
80+
panic(err)
81+
}
82+
}
83+
84+
// cleanSnapshots when called will clean snapshots in the project (if specified)
85+
// that are older than the number of days.
86+
func cleanSnapshots(days int, instanceAPI *instance.API) error {
87+
// Get the list of all snapshots
88+
snapshotsList, err := instanceAPI.ListSnapshots(&instance.ListSnapshotsRequest{
89+
Zone: scw.Zone(os.Getenv(envZone)),
90+
Project: scw.StringPtr(os.Getenv(envProjectID)),
91+
},
92+
scw.WithAllPages())
93+
if err != nil {
94+
return fmt.Errorf("error while listing snapshots %w", err)
95+
}
96+
97+
const hoursPerDay = 24
98+
99+
currentTime := time.Now()
100+
101+
// For each snapshot, check conditions
102+
for _, snapshot := range snapshotsList.Snapshots {
103+
// Check if snapshot is in ready state and if it's older than the number of days definied.
104+
if snapshot.State == instance.SnapshotStateAvailable && (currentTime.Sub(*snapshot.CreationDate).Hours()/hoursPerDay) > float64(days) {
105+
fmt.Printf("\nDeleting snapshot <%s>:%s created at: %s\n", snapshot.ID, snapshot.Name, snapshot.CreationDate.Format(time.RFC3339))
106+
107+
// Delete snapshot found.
108+
err := instanceAPI.DeleteSnapshot(&instance.DeleteSnapshotRequest{
109+
SnapshotID: snapshot.ID,
110+
Zone: snapshot.Zone,
111+
})
112+
if err != nil {
113+
return fmt.Errorf("error while deleting snapshot: %w", err)
114+
}
115+
}
116+
}
117+
118+
return nil
119+
}
120+
121+
// Check for mandatory variables before starting to work.
122+
func init() {
123+
mandatoryVariables := [...]string{envOrgID, envAccessKey, envSecretKey, envZone, envProjectID}
124+
125+
for idx := range mandatoryVariables {
126+
if os.Getenv(mandatoryVariables[idx]) == "" {
127+
panic("missing environment variable " + mandatoryVariables[idx])
128+
}
129+
}
130+
}

jobs/instances-snapshot/Dockerfile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Using apline/golang image
2-
FROM golang:1.22-alpine
2+
FROM golang:1.24-alpine
33

44
# Set destination for COPY
55
WORKDIR /app
@@ -13,4 +13,4 @@ COPY *.go ./
1313
RUN go build -o /jobs-snapshot
1414

1515
# Run the executable
16-
CMD [ "/jobs-snapshot" ]
16+
ENTRYPOINT [ "/jobs-snapshot" ]

jobs/instances-snapshot/README.md

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,12 @@ This example is very simple, it generates snapshots of your desired Instance.
1111
- Scaleway Account
1212
- Docker daemon running to build the image
1313
- Container registry namespace created, for this example we assume that your namespace name is `jobs-snapshot`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/)
14+
- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/)
1415

1516
## Step 1 : Build and push to Container registry
1617

1718
Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works
18-
with Containers. So first, using your terminal reach this folder and run the following commands:
19+
with containers. So first, use your terminal reach this folder and run the following commands:
1920

2021
```shell
2122
# First command is to login to container registry, you can find it in Scaleway console
@@ -24,14 +25,16 @@ docker login rg.fr-par.scw.cloud/jobs-snapshot -u nologin --password-stdin <<< "
2425
# Here we build the image to push
2526
docker build -t rg.fr-par.scw.cloud/jobs-snapshot/jobs-snapshot:v1 .
2627

28+
## TIP: for Apple Silicon or other ARM processors, please use the following command as Serverless Jobs supports amd64 architecture
29+
# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/jobs-snapshot/jobs-snapshot:v1 .
30+
2731
# Push the image online to be used on Serverless Jobs
2832
docker push rg.fr-par.scw.cloud/jobs-snapshot/jobs-snapshot:v1
2933
```
34+
> [!TIP]
35+
> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case.
3036
31-
Note about Serverless Containers versus Serverless Jobs. As we do not expose a web server and we do not
32-
require features such as auto-scaling, Serverless Jobs are perfect for this use case.
33-
34-
To check if everyting is ok, on the Console you can verify if your tag is present in Container Registry.
37+
To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry.
3538

3639
## Step 2: Creating the Job Definition
3740

@@ -44,25 +47,33 @@ On Scaleway Console on the following link you can create a new Job Definition: h
4447
1. To schedule your job for example every night at 2am, you can set the cron to `0 2 * * *`.
4548
1. Important: advanced option, you need to set the following environment variables:
4649

50+
> [!TIP]
51+
> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/).
52+
4753
- `INSTANCE_ID`: grab the instance ID you want to create snapshots from
4854
- `INSTANCE_ZONE`: you need to give the ZONE of you instance, like `fr-par-2`
4955
- `SCW_ACCESS_KEY`: your access key
5056
- `SCW_SECRET_KEY`: your secret key
5157
- `SCW_DEFAULT_ORGANIZATION_ID`: your organzation ID
5258

53-
1. Click "create job"
59+
* Then click "create job"
5460

5561
## Step 3: Run the job
5662

5763
On your created Job Definition, just click the button "Run Job" and within seconds it should be successful.
5864

65+
## Troubleshooting
66+
67+
If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error.
68+
5969
# Possible improvements
6070

6171
You can exercice by adding the following features:
6272

6373
- Instead of managing a single instance, make it account wide
6474
- Add disk backups
6575
- Add alerts if something goes wrong
76+
- Use secret manager instead of job environment variables
6677

6778
# Additional content
6879

jobs/instances-snapshot/go.mod

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,12 @@
11
module github.com/scaleway/serverless-examples/jobs/instances-snapshot
22

3-
go 1.22.2
3+
go 1.24
44

5-
require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.21
5+
require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32
66

7-
require gopkg.in/yaml.v2 v2.4.0 // indirect
7+
require (
8+
github.com/kr/pretty v0.3.1 // indirect
9+
github.com/rogpeppe/go-internal v1.13.1 // indirect
10+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
11+
gopkg.in/yaml.v2 v2.4.0 // indirect
12+
)

0 commit comments

Comments
 (0)