Skip to content

Commit 8d4b3b2

Browse files
thomas-tacquetthomas-tacquet
committed
add empty namespace cleaner
1 parent c626bcc commit 8d4b3b2

File tree

12 files changed

+403
-69
lines changed

12 files changed

+403
-69
lines changed

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,8 @@ Table of Contents:
8484
| **[Serverless MLOps](jobs/ml-ops/README.md)** <br/> An example of running a Serverless Machine Leaning workflow. | Python | [Terraform]-[Console]-[CLI] |
8585
| **[Auto Snapshot Instances](jobs/instances-snapshot/README.md)** <br/> Use Serverless Jobs to create snapshots of your instances | Go | [Console] |
8686
| **[Instance Snapshot Cleaner](jobs/instances-snapshot-cleaner/README.md)** <br/> Use Serverless Jobs to clean old instances snapshots | Go | [Console] |
87+
| **[Registry Tag Cleaner](jobs/registry-version-based-retention/README.md)** <br/> Use Serverless Jobs to keep a desired amount of tags for each image | Go | [Console] |
88+
| **[Registry Empty Image Cleaner](jobs/registry-empty-ressource-cleaner/README.md)** <br/> Use Serverless Jobs to clean container registry empty namespaces and images | Go | [Console] |
8789

8890
### 💬 Messaging and Queueing
8991

‎jobs/registry-empty-ressource-cleaner/Dockerfile‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Use the alpine version of the golang image as the base image
2+
FROM golang:1.24-alpine
3+
4+
# Set the working directory inside the container to /app
5+
WORKDIR /app
6+
7+
# Copy the go.mod and go.sum files to the working directory
8+
COPY go.mod ./
9+
COPY go.sum ./
10+
11+
# Copy the Go source files to the working directory
12+
COPY *.go ./
13+
14+
# Build the executable named reg-clean from the Go source files
15+
RUN go build -o /reg-namespace-clean
16+
# Set the default command to run the reg-clean executable when the container starts
17+
18+
CMD ["/reg-namespace-clean"]

‎jobs/registry-empty-ressource-cleaner/README.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Scaleway Container Registry Cleaner
2+
3+
This project helps you clean up your Container Registry by deleting namespaces without images inside.
4+
5+
## Requirements
6+
7+
- Scaleway Account
8+
- Docker daemon running to build the image
9+
- Container registry namespace created, for this example we assume that your namespace name is `registry-cleaner`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/)
10+
- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/)
11+
12+
## Step 1 : Build and push to Container registry
13+
14+
Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works
15+
with containers. So first, use your terminal reach this folder and run the following commands:
16+
17+
```shell
18+
# First command is to login to container registry, you can find it in Scaleway console
19+
docker login rg.fr-par.scw.cloud/registry-cleaner -u nologin --password-stdin <<< "$SCW_SECRET_KEY"
20+
21+
# Here we build the image to push
22+
docker build -t rg.fr-par.scw.cloud/registry-cleaner/empty-namespaces:v1 .
23+
24+
## TIP: for Apple Silicon or other ARM processors, please use the following command as Serverless Jobs supports amd64 architecture
25+
# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/registry-cleaner/empty-namespaces:v1 .
26+
27+
# Push the image online to be used on Serverless Jobs
28+
docker push rg.fr-par.scw.cloud/registry-cleaner/empty-namespaces:v1
29+
```
30+
31+
> [!TIP]
32+
> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case.
33+
34+
To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry.
35+
36+
## Step 2: Creating the Job Definition
37+
38+
On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par
39+
40+
1. On Container image, select the image you created in the step before.
41+
2. You can set the image name to something clear like `registry-namespace-cleaner` too.
42+
3. For the region you can select the one you prefer :)
43+
4. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory.
44+
5. To schedule your job for example every night at 2am, you can set the cron to `0 2 * * *`.
45+
6. Important: advanced option, you need to set the following environment variables:
46+
47+
> [!TIP]
48+
> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/).
49+
50+
- **Environment Variables**: Set the required environment variables:
51+
- `SCW_DEFAULT_ORGANIZATION_ID`: Your Scaleway organization ID.
52+
- `SCW_ACCESS_KEY`: Your Scaleway API access key.
53+
- `SCW_SECRET_KEY`: Your Scaleway API secret key.
54+
- `SCW_PROJECT_ID`: Your Scaleway project ID.
55+
- `SCW_NO_DRY_RUN`: Set to `true` to delete namespaces; otherwise, it will perform a dry run.
56+
57+
* Then click "Create Job"
58+
59+
## Step 3: Run the job
60+
61+
On your created Job Definition, just click the button "Run Job" and within seconds it should be successful.
62+
63+
## Troubleshooting
64+
65+
If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error.
66+
67+
# Additional content
68+
69+
- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/)
70+
- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/)
71+
- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/)
72+
- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/)

‎jobs/registry-empty-ressource-cleaner/go.mod‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
module github.com/scaleway/serverless-examples/jobs/registry-empty-ressource-cleaner
2+
3+
go 1.24.0
4+
5+
require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32
6+
7+
require (
8+
github.com/kr/pretty v0.3.1 // indirect
9+
github.com/rogpeppe/go-internal v1.13.1 // indirect
10+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
11+
gopkg.in/yaml.v2 v2.4.0 // indirect
12+
)

‎jobs/registry-empty-ressource-cleaner/go.sum‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
2+
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
3+
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
4+
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
5+
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
6+
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
7+
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
8+
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
9+
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
10+
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
11+
github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
12+
github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
13+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 h1:4+LP7qmsLSGbmc66m1s5dKRMBwztRppfxFKlYqYte/c=
14+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32/go.mod h1:kzh+BSAvpoyHHdHBCDhmSWtBc1NbLMZ2lWHqnBoxFks=
15+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
16+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
17+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
18+
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
19+
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=

‎jobs/registry-empty-ressource-cleaner/main.go‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
package main
2+
3+
import (
4+
"log/slog"
5+
"os"
6+
"strings"
7+
8+
"github.com/scaleway/scaleway-sdk-go/scw"
9+
)
10+
11+
// Constants for environment variable names used to configure the application
12+
const (
13+
envOrgID = "SCW_DEFAULT_ORGANIZATION_ID" // Scaleway organization ID
14+
envAccessKey = "SCW_ACCESS_KEY" // Scaleway API access key
15+
envSecretKey = "SCW_SECRET_KEY" // Scaleway API secret key
16+
envProjectID = "SCW_PROJECT_ID" // Scaleway project ID
17+
18+
// If set to "true", older tags will be deleted.
19+
// Otherwise, only a dry run will be performed
20+
envNoDryRun = "SCW_NO_DRY_RUN"
21+
)
22+
23+
// Check for mandatory variables before starting to work.
24+
func init() {
25+
// Slice of environmental variables that must be set for the application to run
26+
mandatoryVariables := [...]string{envOrgID, envAccessKey, envSecretKey, envProjectID}
27+
28+
// Iterate through the slice and check if any variables are not set
29+
for idx := range mandatoryVariables {
30+
if os.Getenv(mandatoryVariables[idx]) == "" {
31+
panic("missing environment variable " + mandatoryVariables[idx])
32+
}
33+
}
34+
}
35+
36+
func main() {
37+
slog.Info("cleaning container registry tags...")
38+
39+
// Create a Scaleway client with credentials provided via environment variables.
40+
// The client is used to interact with the Scaleway API
41+
client, err := scw.NewClient(
42+
// Get your organization ID at https://console.scaleway.com/organization/settings
43+
scw.WithDefaultOrganizationID(os.Getenv(envOrgID)),
44+
45+
// Get your credentials at https://console.scaleway.com/iam/api-keys
46+
scw.WithAuth(os.Getenv(envAccessKey), os.Getenv(envSecretKey)),
47+
48+
// Get more about our availability
49+
// zones at https://www.scaleway.com/en/docs/console/my-account/reference-content/products-availability/
50+
scw.WithDefaultRegion(scw.RegionFrPar),
51+
)
52+
if err != nil {
53+
panic(err)
54+
}
55+
56+
// Create a new instance of RegistryAPI, passing the Scaleway client and the project ID
57+
// RegistryAPI is a custom interface for interacting with the Scaleway container registry
58+
regAPI := NewRegistryAPI(client, os.Getenv(scw.ScwDefaultProjectIDEnv))
59+
60+
// Determine whether to perform a dry run or delete the tags
61+
// Default behavior is to perform a dry run (no deletion)
62+
dryRun := true
63+
noDryRunEnv := os.Getenv(envNoDryRun)
64+
65+
// If the SCW_NO_DRY_RUN environment variable is set to "true",
66+
// the tags will be deleted; otherwise, only a dry run will be performed
67+
if strings.EqualFold(noDryRunEnv, "true") {
68+
dryRun = false
69+
}
70+
71+
// Delete the tags or perform a dry run, depending on the dryRun flag
72+
if err := regAPI.DeleteEmptyNamespace(dryRun); err != nil {
73+
panic(err)
74+
}
75+
}

‎jobs/registry-empty-ressource-cleaner/registry.go‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"log/slog"
6+
7+
registry "github.com/scaleway/scaleway-sdk-go/api/registry/v1"
8+
"github.com/scaleway/scaleway-sdk-go/scw"
9+
)
10+
11+
// RegistryAPI represents a Scaleway Container Registry accessor and extends
12+
// capabilities to clean images. It allows you to manage container images and tags
13+
// across one or more projects, and provides options to delete image tags safely or
14+
// with caution.
15+
type RegistryAPI struct {
16+
// regClient Scaleway Container Registry accessor.
17+
regClient *registry.API
18+
19+
// projectID specifies a project to be scoped for operations. If this field
20+
// is nil, operations will be performed on all available projects.
21+
projectID *string
22+
}
23+
24+
// NewRegistryAPI creates a new RegistryAPI to manage the Scaleway Container Registry API.
25+
// It initializes the RegistryAPI struct with the provided Scaleway SDK client and a project
26+
// ID. If the projectID is empty, it will not be passed to the Scaleway SDK, allowing operations
27+
// on all projects.
28+
func NewRegistryAPI(client *scw.Client, projectID string) *RegistryAPI {
29+
return &RegistryAPI{
30+
regClient: registry.NewAPI(client),
31+
projectID: scw.StringPtr(projectID),
32+
}
33+
}
34+
35+
func (r *RegistryAPI) DeleteEmptyNamespace(dryRun bool) error {
36+
namespaces, err := r.regClient.ListNamespaces(&registry.ListNamespacesRequest{ProjectID: r.projectID}, scw.WithAllPages())
37+
if err != nil {
38+
return fmt.Errorf("error listing registry namespaces %w", err)
39+
}
40+
slog.Info("DryRun ENABLED")
41+
42+
for _, namespace := range namespaces.Namespaces {
43+
if namespace.Status == registry.NamespaceStatusReady && namespace.ImageCount == 0 {
44+
slog.Info("deleteing namespace", slog.String("name", namespace.Name), slog.String("id", namespace.ID))
45+
if !dryRun {
46+
_, err := r.regClient.DeleteNamespace(&registry.DeleteNamespaceRequest{
47+
NamespaceID: namespace.ID,
48+
})
49+
if err != nil {
50+
return fmt.Errorf("error deleting namesapce %w", err)
51+
}
52+
}
53+
}
54+
}
55+
56+
return nil
57+
}

‎jobs/registry-version-based-retention/README.md‎

Lines changed: 57 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -2,47 +2,72 @@
22

33
This project aims to clean up Scaleway Container Registry tags to keep only the N latest tags for each image. It is useful for managing disk space and keeping the registry organized.
44

5-
## Usage
5+
## Requirements
66

7-
1. **Build the Application:**
8-
```bash
9-
go build -o reg-clean
10-
```
7+
- Scaleway Account
8+
- Docker daemon running to build the image
9+
- Container registry namespace created, for this example we assume that your namespace name is `registry-cleaner`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/)
10+
- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/)
1111

12-
2. **Run the Application:**
13-
```bash
14-
./reg-clean
15-
```
12+
## Step 1 : Build and push to Container registry
1613

17-
## Environment Variables
14+
Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works
15+
with containers. So first, use your terminal reach this folder and run the following commands:
1816

19-
The application requires the following environment variables to be set:
17+
```shell
18+
# First command is to login to container registry, you can find it in Scaleway console
19+
docker login rg.fr-par.scw.cloud/registry-cleaner -u nologin --password-stdin <<< "$SCW_SECRET_KEY"
2020

21-
- `SCW_DEFAULT_ORGANIZATION_ID`: Your Scaleway organization ID.
22-
- `SCW_ACCESS_KEY`: Your Scaleway access key.
23-
- `SCW_SECRET_KEY`: Your Scaleway secret key.
24-
- `SCW_PROJECT_ID`: Your Scaleway project ID.
25-
- `SCW_NUMBER_VERSIONS_TO_KEEP`: The number of latest tags to keep for each image.
26-
- `SCW_NO_DRY_RUN` (optional): Set to `true` to perform actual deletions. If not set, the application will run in dry-run mode, only logging the actions that would be taken.
21+
# Here we build the image to push
22+
docker build -t rg.fr-par.scw.cloud/registry-cleaner/versions-retention:v1 .
2723

28-
## Example
24+
## TIP: for Apple Silicon or other ARM processors, please use the following command as Serverless Jobs supports amd64 architecture
25+
# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/registry-cleaner/versions-retention:v1 .
2926

30-
To run the application in dry-run mode and keep the 5 latest tags for each image, set the following environment variables and run:
27+
# Push the image online to be used on Serverless Jobs
28+
docker push rg.fr-par.scw.cloud/registry-cleaner/versions-retention:v1
29+
```
3130

32-
```bash
33-
export SCW_DEFAULT_ORGANIZATION_ID=your-organization-id
34-
export SCW_ACCESS_KEY=your-access-key
35-
export SCW_SECRET_KEY=your-secret-key
36-
export SCW_PROJECT_ID=your-project-id
37-
export SCW_NUMBER_VERSIONS_TO_KEEP=5
31+
> [!TIP]
32+
> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case.
3833
39-
./reg-clean
40-
```
34+
To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry.
35+
36+
## Step 2: Creating the Job Definition
37+
38+
On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par
39+
40+
1. On Container image, select the image you created in the step before.
41+
2. You can set the image name to something clear like `registry-version-retention` too.
42+
3. For the region you can select the one you prefer :)
43+
4. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory.
44+
5. To schedule your job for example every night at 2am, you can set the cron to `0 2 * * *`.
45+
6. Important: advanced option, you need to set the following environment variables:
46+
47+
> [!TIP]
48+
> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/).
49+
50+
- **Environment Variables**: Set the required environment variables:
51+
- `SCW_DEFAULT_ORGANIZATION_ID`: Your Scaleway organization ID.
52+
- `SCW_ACCESS_KEY`: Your Scaleway API access key.
53+
- `SCW_SECRET_KEY`: Your Scaleway API secret key.
54+
- `SCW_PROJECT_ID`: Your Scaleway project ID.
55+
- `SCW_NUMBER_VERSIONS_TO_KEEP`: The number of latest tags to keep for each image.
56+
- `SCW_NO_DRY_RUN`: Set to `true` to delete namespaces; otherwise, it will perform a dry run.
57+
58+
* Then click "Create Job"
59+
60+
## Step 3: Run the job
61+
62+
On your created Job Definition, just click the button "Run Job" and within seconds it should be successful.
63+
64+
## Troubleshooting
4165

42-
To run the application and actually delete the tags, set `SCW_NO_DRY_RUN` to `true`:
66+
If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error.
4367

44-
```bash
45-
export SCW_NO_DRY_RUN=true
68+
# Additional content
4669

47-
./reg-clean
48-
```
70+
- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/)
71+
- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/)
72+
- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/)
73+
- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/)

‎jobs/registry-version-based-retention/go.mod‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,9 @@ go 1.24.0
44

55
require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32
66

7-
require gopkg.in/yaml.v2 v2.4.0 // indirect
7+
require (
8+
github.com/kr/pretty v0.3.1 // indirect
9+
github.com/rogpeppe/go-internal v1.13.1 // indirect
10+
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
11+
gopkg.in/yaml.v2 v2.4.0 // indirect
12+
)

0 commit comments

Comments
 (0)