From 13069470cc2325fcceb8358352811825b6c7f26b Mon Sep 17 00:00:00 2001 From: SamyOubouaziz Date: Tue, 19 Aug 2025 13:30:52 +0200 Subject: [PATCH 1/8] docs(srv): how to run complex commands with Jobs and Secrets MTA-6354 --- .../how-to/execute-complex-commands.mdx | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 pages/serverless-jobs/how-to/execute-complex-commands.mdx diff --git a/pages/serverless-jobs/how-to/execute-complex-commands.mdx b/pages/serverless-jobs/how-to/execute-complex-commands.mdx new file mode 100644 index 0000000000..239581388d --- /dev/null +++ b/pages/serverless-jobs/how-to/execute-complex-commands.mdx @@ -0,0 +1,41 @@ +--- +title: How to execute complex commands using Serverless Jobs +description: Learn how to run complex commands and scripts using Scaleway Serverless Jobs and Secrets Manager. +tags: run execute start serverless job scaleway script startup secret bash complex command pipe +dates: + validation: 2025-08-19 + posted: 2025-08-19 +--- +import Requirements from '@macros/iam/requirements.mdx' + +Scaleway Serverless Jobs allows you to execute specific startup commands when running a job. Due to technical specifications, complex commands (e.g. piped commands, `xargs` commands) may fail occasionally, preventing jobs from running successfully. + +You can bypass this limitation by passing complex commands and scripts to a Serverless Job via a [secret reference](/serverless-jobs/concepts/#secrets-reference). You can then inject this secret as a file in your job, and call it at startup to execute the commands it contains. + + + +- A Scaleway account logged into the [console](https://console.scaleway.com) +- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization +- created a [Serverless Job](/serverless-jobs/how-to/create-job/) + +## How to create a secret containing your command + +1. Click **Secret Manager** in the **Security & Identity** section of the [Scaleway console](https://console.scaleway.com/) side menu. + +2. In the **Region** drop-down, select the [region](/secret-manager/concepts/#region) in which you want to store your secret. Secrets cannot be moved from one region to another after creation. + +3. Click **+ Create secret**. + +4. Select **Import secret**, choose the **Opaque** secret type, then drag and drop your file to the dedicated area. The maximum file size for your secret is 64 KiB. + +5. Choose the **Scaleway-managed encryption key**, as it requires no configuration on your side. + +6. Choose a [path](/secret-manager/concepts/#path) for your secret. + +7. Enter a name for your secret, and, optionally, add a description and tags. + +8. Click **Create secret** to confirm. + +Your file can now be passed to your Serverless Job as a secret reference. + +## \ No newline at end of file From 1e51b8823d9813ecc1c36838a722364f7325166c Mon Sep 17 00:00:00 2001 From: SamyOubouaziz Date: Wed, 20 Aug 2025 10:47:29 +0200 Subject: [PATCH 2/8] docs(srv): update --- .../how-to/execute-complex-commands.mdx | 79 ++++++++++++++++++- 1 file changed, 76 insertions(+), 3 deletions(-) diff --git a/pages/serverless-jobs/how-to/execute-complex-commands.mdx b/pages/serverless-jobs/how-to/execute-complex-commands.mdx index 239581388d..2c79bc2154 100644 --- a/pages/serverless-jobs/how-to/execute-complex-commands.mdx +++ b/pages/serverless-jobs/how-to/execute-complex-commands.mdx @@ -1,6 +1,6 @@ --- -title: How to execute complex commands using Serverless Jobs -description: Learn how to run complex commands and scripts using Scaleway Serverless Jobs and Secrets Manager. +title: How to execute complex startup commands using Serverless Jobs +description: Learn how to run complex commands at job run start and scripts using Scaleway Serverless Jobs and Secrets Manager. tags: run execute start serverless job scaleway script startup secret bash complex command pipe dates: validation: 2025-08-19 @@ -27,6 +27,9 @@ You can bypass this limitation by passing complex commands and scripts to a Serv 3. Click **+ Create secret**. 4. Select **Import secret**, choose the **Opaque** secret type, then drag and drop your file to the dedicated area. The maximum file size for your secret is 64 KiB. + + We recommend using a shell script (`.sh`) file containing your command. Refer to the [section below](#complex-commands-examples) for example commands. + 5. Choose the **Scaleway-managed encryption key**, as it requires no configuration on your side. @@ -38,4 +41,74 @@ You can bypass this limitation by passing complex commands and scripts to a Serv Your file can now be passed to your Serverless Job as a secret reference. -## \ No newline at end of file +## Create a Serverles Job referencing your command as a secret + +1. Click **Jobs** in the **Serverless** section of the side menu. The jobs page displays. + +2. Click **Create job**. The job creation wizard displays. + +3. Select the **external** container registry. + +4. Enter the following image URL: + ```sh + scaleway/cli:latest + ``` + +5. Enter a name, select the desired region, and choose the smallest **resources** available. + +6. Enter a CRON schedule to run your job periodically. Refer to the [dedicated documentation](/serverless-jobs/reference-content/cron-schedules/) on CRON schedules. + +7. Add your command file as a [secrets reference](/serverless-functions/concepts/#secrets). Refer to the [dedicated documentation] on secrets for more information. + +8. Add the following startup command to call your file: + ```sh + bash /complex_command.sh start + ``` + +9. Click **Create job** to finish. + +You job is now ready to run at the specified schedule. + +## Complex commands examples + +Below are examples of commands that must be passed via a secret referenced as a file in Serverless Jobs. You can find more complex commands examples on the [Scaleway CLI repository](/https://github.com/scaleway/scaleway-cli/blob/master/docs/cookbook.md). + +**Retrieve a specific field from the output using jq** + +```bash +## Retrieve all available instance type names + +# Using jq +scw -o json instance server-type list | jq -r '.[].name' +# Using CLI templates +scw instance server-type list -o template="{{ .Name }}" +``` + +**Filter command output using jq** + +```bash +# Retrieve all available instance type with GPUs +scw -o json instance server-type list | jq '.[] | select (.gpu > 0)'``` +``` + +**Parallelize actions using xargs** + +```bash +## Reboot all listed servers at the same time, 8 max concurrent actions + +# Using jq +scw -o json instance server list | jq -r '.[].id' | xargs -L1 -P8 scw instance server reboot +# Using CLI templates +scw instance server list -o template="{{ .ID }}" | xargs -L1 -P8 scw instance server reboot +``` + +**Create arguments for a CLI command and use it in xargs** + +```bash +## List private-nics of all listed servers + +# Using jq +scw -o json instance server list | jq -r '.[] | "server-id=\(.id)"' | xargs -L1 scw instance private-nic list +# Using CLI templates +scw instance server list -o template="server-id={{ .ID }}" | xargs -L1 scw instance private-nic list +``` From b633b33e7795561d264b18555e2e83659997a93b Mon Sep 17 00:00:00 2001 From: SamyOubouaziz Date: Wed, 20 Aug 2025 11:12:07 +0200 Subject: [PATCH 3/8] docs(srv): update --- pages/serverless-jobs/how-to/execute-complex-commands.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pages/serverless-jobs/how-to/execute-complex-commands.mdx b/pages/serverless-jobs/how-to/execute-complex-commands.mdx index 2c79bc2154..2241010c00 100644 --- a/pages/serverless-jobs/how-to/execute-complex-commands.mdx +++ b/pages/serverless-jobs/how-to/execute-complex-commands.mdx @@ -58,9 +58,9 @@ Your file can now be passed to your Serverless Job as a secret reference. 6. Enter a CRON schedule to run your job periodically. Refer to the [dedicated documentation](/serverless-jobs/reference-content/cron-schedules/) on CRON schedules. -7. Add your command file as a [secrets reference](/serverless-functions/concepts/#secrets). Refer to the [dedicated documentation] on secrets for more information. +7. From the **Data** tab, add your command file as a [secrets reference](/serverless-functions/concepts/#secrets). Refer to the [dedicated documentation] on secrets for more information. -8. Add the following startup command to call your file: +8. From the **Execution** tab, add the following startup command to call your file: ```sh bash /complex_command.sh start ``` From dc0741406cc26f662df635d5a3f9dc08e7d11b72 Mon Sep 17 00:00:00 2001 From: SamyOubouaziz Date: Wed, 20 Aug 2025 11:56:12 +0200 Subject: [PATCH 4/8] docs(srv): update --- .../automate-resources-management.mdx | 4 ++++ .../job-startup-command-fails.mdx | 20 +++++++++++++++++++ 2 files changed, 24 insertions(+) create mode 100644 pages/serverless-jobs/troubleshooting/job-startup-command-fails.mdx diff --git a/macros/serverless-jobs/automate-resources-management.mdx b/macros/serverless-jobs/automate-resources-management.mdx index c3fd56d6bd..58d0fe0171 100644 --- a/macros/serverless-jobs/automate-resources-management.mdx +++ b/macros/serverless-jobs/automate-resources-management.mdx @@ -7,6 +7,10 @@ import Requirements from '@macros/iam/requirements.mdx' [Scaleway Serverless Jobs](/serverless-jobs/quickstart/) allows you to create and automate recurring tasks. This page shows how to create jobs to perform any operation available with the [Scaleway CLI](https://github.com/scaleway/scaleway-cli/blob/master/docs/commands/config.md) to automate the management of your Scaleway resources. + +To automate complex commands, such as piped or `xargs` commands, refer to the [dedicated documentation](/serverless-jobs/how-to/execute-complex-commands/). + + Serverless Jobs are perfectly adapted for these autonomous tasks, as we do not need autoscaling or exposure via a web server. Refer to the [documentation on differences between jobs, containers, and functions](/serverless-jobs/reference-content/difference-jobs-functions-containers/) for more information. diff --git a/pages/serverless-jobs/troubleshooting/job-startup-command-fails.mdx b/pages/serverless-jobs/troubleshooting/job-startup-command-fails.mdx new file mode 100644 index 0000000000..16a5aa8085 --- /dev/null +++ b/pages/serverless-jobs/troubleshooting/job-startup-command-fails.mdx @@ -0,0 +1,20 @@ +--- +title: My job run fails when executing the startup command +description: Troubleshoot job run failures due to startup commands for your Scaleway Serverless Jobs. +tags: serverless jobs troubleshooting issue error state message fail execution log faliure does not run command +dates: + validation: 2025-08-20 + posted: 2025-08-20 +--- + +## Problem + +My job run fails due to its startup command. + +## Cause + +Complex commands, such as piped or `xargs` commands may lead to failures due to techical limitations of Serverless Jobs. + +## Possible solutions + +You can bypass this issue by using [Secret Manager](/secret-manager/) to pass complex commands in a shell script file via a [secret reference](/serverless-jobs/concepts/secrets-references). Refer to the [dedicated documentation](/serverless-jobs/how-to/execute-complex-commands/) for more information. \ No newline at end of file From 787ad90f7c1c139c1ab5d4cff488c6085ba7e6c9 Mon Sep 17 00:00:00 2001 From: SamyOubouaziz Date: Wed, 20 Aug 2025 12:18:30 +0200 Subject: [PATCH 5/8] docs(srv): update --- .../troubleshooting/job-startup-command-fails.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/serverless-jobs/troubleshooting/job-startup-command-fails.mdx b/pages/serverless-jobs/troubleshooting/job-startup-command-fails.mdx index 16a5aa8085..270d0f2ef4 100644 --- a/pages/serverless-jobs/troubleshooting/job-startup-command-fails.mdx +++ b/pages/serverless-jobs/troubleshooting/job-startup-command-fails.mdx @@ -17,4 +17,4 @@ Complex commands, such as piped or `xargs` commands may lead to failures due to ## Possible solutions -You can bypass this issue by using [Secret Manager](/secret-manager/) to pass complex commands in a shell script file via a [secret reference](/serverless-jobs/concepts/secrets-references). Refer to the [dedicated documentation](/serverless-jobs/how-to/execute-complex-commands/) for more information. \ No newline at end of file +You can bypass this issue by using [Secret Manager](/secret-manager/) to pass complex commands in a shell script file via a [secret reference](/serverless-jobs/concepts/secrets-references). Refer to the [dedicated documentation](/serverless-jobs/how-to/execute-complex-commands/) for more information \ No newline at end of file From d8aa8df52a78d1a5e3babf2716ab99e057050844 Mon Sep 17 00:00:00 2001 From: SamyOubouaziz Date: Wed, 20 Aug 2025 14:26:37 +0200 Subject: [PATCH 6/8] docs(srv): update --- menu/navigation.json | 4 ++++ pages/serverless-jobs/troubleshooting/index.mdx | 1 + 2 files changed, 5 insertions(+) diff --git a/menu/navigation.json b/menu/navigation.json index e52dc4edcf..732fe9eea0 100644 --- a/menu/navigation.json +++ b/menu/navigation.json @@ -4871,6 +4871,10 @@ "label": "Manage the scheduling of a job", "slug": "manage-job-schedule" }, + { + "label": "Execute complex startup commands", + "slug": "execute-complex-commands" + }, { "label": "Reference secrets in a job", "slug": "reference-secret-in-job" diff --git a/pages/serverless-jobs/troubleshooting/index.mdx b/pages/serverless-jobs/troubleshooting/index.mdx index b13732ee4b..9237cdb180 100644 --- a/pages/serverless-jobs/troubleshooting/index.mdx +++ b/pages/serverless-jobs/troubleshooting/index.mdx @@ -52,3 +52,4 @@ productIcon: ServerlessJobsProductIcon - [My Job run is in an error state](/serverless-jobs/troubleshooting/job-in-error-state) - [Missing metrics for my Job Run](/serverless-jobs/troubleshooting/missing-metrics) +- [Job startup command fails](/serverless-jobs/troubleshooting/job-startup-command-fails/) \ No newline at end of file From 0a5d93d0691bc322c202c38d30303d7b2bca01fa Mon Sep 17 00:00:00 2001 From: SamyOubouaziz Date: Thu, 21 Aug 2025 08:46:25 +0200 Subject: [PATCH 7/8] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Néda <87707325+nerda-codes@users.noreply.github.com> --- pages/serverless-jobs/how-to/execute-complex-commands.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pages/serverless-jobs/how-to/execute-complex-commands.mdx b/pages/serverless-jobs/how-to/execute-complex-commands.mdx index 2241010c00..5be8b773ab 100644 --- a/pages/serverless-jobs/how-to/execute-complex-commands.mdx +++ b/pages/serverless-jobs/how-to/execute-complex-commands.mdx @@ -16,7 +16,7 @@ You can bypass this limitation by passing complex commands and scripts to a Serv - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- created a [Serverless Job](/serverless-jobs/how-to/create-job/) +- Created a [Serverless Job](/serverless-jobs/how-to/create-job/) ## How to create a secret containing your command @@ -26,7 +26,7 @@ You can bypass this limitation by passing complex commands and scripts to a Serv 3. Click **+ Create secret**. -4. Select **Import secret**, choose the **Opaque** secret type, then drag and drop your file to the dedicated area. The maximum file size for your secret is 64 KiB. +4. Select **Import secret**, choose the **Opaque** secret type, then drag and drop your file to the dedicated area. The maximum file size for your secret is 64 KiB. We recommend using a shell script (`.sh`) file containing your command. Refer to the [section below](#complex-commands-examples) for example commands. @@ -41,7 +41,7 @@ You can bypass this limitation by passing complex commands and scripts to a Serv Your file can now be passed to your Serverless Job as a secret reference. -## Create a Serverles Job referencing your command as a secret +## Create a Serverless Job referencing your command as a secret 1. Click **Jobs** in the **Serverless** section of the side menu. The jobs page displays. @@ -88,7 +88,7 @@ scw instance server-type list -o template="{{ .Name }}" ```bash # Retrieve all available instance type with GPUs -scw -o json instance server-type list | jq '.[] | select (.gpu > 0)'``` +scw -o json instance server-type list | jq '.[] | select (.gpu > 0)' ``` **Parallelize actions using xargs** From 11733b8ff4a436434ecba242d52a21fbedfa3782 Mon Sep 17 00:00:00 2001 From: SamyOubouaziz Date: Fri, 22 Aug 2025 10:41:16 +0200 Subject: [PATCH 8/8] docs(srv): update --- .../how-to/execute-complex-commands.mdx | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/pages/serverless-jobs/how-to/execute-complex-commands.mdx b/pages/serverless-jobs/how-to/execute-complex-commands.mdx index 5be8b773ab..a06ccb8160 100644 --- a/pages/serverless-jobs/how-to/execute-complex-commands.mdx +++ b/pages/serverless-jobs/how-to/execute-complex-commands.mdx @@ -56,22 +56,20 @@ Your file can now be passed to your Serverless Job as a secret reference. 5. Enter a name, select the desired region, and choose the smallest **resources** available. -6. Enter a CRON schedule to run your job periodically. Refer to the [dedicated documentation](/serverless-jobs/reference-content/cron-schedules/) on CRON schedules. +6. From the **Data** tab, add your command file as a [secrets reference](/serverless-functions/concepts/#secrets). Refer to the [dedicated documentation](/serverless-jobs/how-to/reference-secret-in-job/) on secrets for more information. -7. From the **Data** tab, add your command file as a [secrets reference](/serverless-functions/concepts/#secrets). Refer to the [dedicated documentation] on secrets for more information. - -8. From the **Execution** tab, add the following startup command to call your file: +7. From the **Execution** tab, add the following startup command to call your file: ```sh bash /complex_command.sh start ``` -9. Click **Create job** to finish. +8. Click **Create job** to finish. -You job is now ready to run at the specified schedule. +You job is now ready to run. ## Complex commands examples -Below are examples of commands that must be passed via a secret referenced as a file in Serverless Jobs. You can find more complex commands examples on the [Scaleway CLI repository](/https://github.com/scaleway/scaleway-cli/blob/master/docs/cookbook.md). +Below are examples of commands that must be passed via a secret referenced as a file in Serverless Jobs. You can find more complex command examples on the [Scaleway CLI repository](/https://github.com/scaleway/scaleway-cli/blob/master/docs/cookbook.md). **Retrieve a specific field from the output using jq**