How to pass a github secret to a shell script that is called by the script module ?

[kinggrowler]

Sorry if this is explained elsewhere but I can't figure it out.

During the build process, I want to download some protected assets from a webserver, and this command needs to be in a script referenced by the script module in my recipe file. The curl command should look something like this:

curl -s -L -u "$CURL_CREDS" "${base_url}/${FILE}" | tar -xvzf - -C /usr/share/

Basically, this works fine right now without the "-u" option; e.g. from a website with unprotected files. I would like to put these assets behind apache auth, at least. However I cannot figure out how to add a GitHub secret (containing username:password info) and make this visible to my shell script as an ENV variable. (Or, realistically, whatever method works, env variable or otherwise.)

Here is the pipeline in a rough outline:

• github/workflows/build.yml
• recipe.yml with script module
• script.sh with the curl line from above

I would like to change the curl command to use a github secret with my username/password.

Is this possible?

Here is the relevant code from my build.yml file:

  build-images:
    name: Build all images
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write
    strategy:
      fail-fast: false # stop GH from cancelling all matrix builds if one fails
      matrix:
        recipe:
          - recipe.yml
    steps:
      # the build is fully handled by the reusable github action
      - name: Build Custom Image
        uses: blue-build/github-action@v1.10
        with:
          recipe: ${{ matrix.recipe }}
          cosign_private_key: ${{ secrets.SIGNING_SECRET }}
          registry_token: ${{ github.token }}
          pr_event_number: ${{ github.event.number }}

          # enabled by default, disable if your image is small and you want faster builds
          maximize_build_space: true

Let me know how to achieve this. Thank you !

[gmpinder]

First, you need to know that anything you save in your image will be publicly available if you're publishing on GHCR, so your assets could be picked up by anyone with access to the image. DO NOT store any secrets like passwords or tokens in any files inside your image.

That being said, if you need to use some credentials during a module execution like a script, you can make use of the secrets property. Unfortunately our docs are very lacking on that feature at the moment. You can take a look at the typespec for it and an example of how we use it for signing the kernel in our base images. We have an issue to update the docs on this feature.

[kinggrowler]

Thank you. Currently, I do NOT have any secrets in either my image or in my github repo, but it's always good to reinterate this caveat. This is why I would like to use a github secret, which will ideally get passed from build.yml to recipe.yml to script. I understand that this flow should be secure.

I will look at the examples provided, but based on experience level, I may not understand. I would describe myself as an "enthusiastic amateur." :-)

[gmpinder]

So you should be able to do something like

modules:
  - type: script
    secrets:
      - type: env
        name: CURL_CREDS
        mount:
          type: env
          name: CURL_CREDS
    scripts:
      - script.sh

This would take the environment variable CURL_CREDS from the host and mount the secret as an environment variable CURL_CREDS in the build. This uses docker and podman's built-in secrets handling.

[kinggrowler]

Thank you, I will try this tomorrow and update this discussion with the results.

[kinggrowler]

I'm doing something wrong here because I can't get it to work.

Let's say that everything works perfectly if I do NOT protect my curl'd assets behind apache auth. curl does its thing, my script works, the build succeeds. Huzzah. I then protect my assets with a username:password combination. This works as expected on the command line if I manually try "curl -u ${CREDS} --fail -L ". So I know that the curl portion works when supplied with an ENV variable in a local script or cli.

So here's where it fails. I have created a Github "Environment" for my repo. Let's call the Environment "MAGIC_CURL_STUFF". This Environment has an "Environment Secret" defined. Let's call the secret "SUPER_SECRET_PASSWORD". This secret is simply a "username:password" string.

My recipe.yml uses a from-file format. This works just great without the curl credentials part. Based on the example given in a previous comment, from-file yaml looks like so:

type: script
secrets:
  - type: env
    name: SUPER_SECRET_PASSWORD
    mount:
      type: env
      name: SUPER_SECRET_PASSWORD
scripts:
  - curl_stuff.sh

Here is the relevant portion of curl_stuff.sh:

curl -u "$SUPER_SECRET_PASSWORD" --fail -s -L "${base_url}/${FILE}" | tar -xvzf - -C /usr/share/

But it falls apart in the build because curl still gets a null SUPER_SECRET_PASSWORD. Here is a snippet from the build log:

=> Running script: /tmp/files/scripts/curl_stuff.sh
=> + echo -e '===== Install wallpapers and themes =====\n'
=> + base_url=https://www.example.com/dir
=> + packs=("asset_wallpapers.tgz" "asset_themes.tgz")
=> + for FILE in ${packs[@]}
=> + curl -u '' --fail -s -L https://www.example.com/dir/asset_wallpapers.tgz
=> + tar -xvzf - -C /usr/share/
=> Enter host password for user '':
=>
=> gzip: stdin: unexpected end of file
=> tar: Child returned status 1
=> tar: Error is not recoverable: exiting now
=> + for FILE in ${packs[@]}
=> + curl -u '' --fail -s -L https://www.example.com/dir/asset_themes.tgz
=> + tar -xvzf - -C /usr/share/
=> Enter host password for user '':
=>
=> gzip: stdin: unexpected end of file
=> tar: Child returned status 1
=> tar: Error is not recoverable: exiting now
=> + echo -e '===== Install Scrivener AppImage =====\n'
=> + scriv=https://www.example.com/dir/Scrivener-1.9.0.1beta-x86_64_english_only-20211229.AppImage
=> + curl -u '' --fail -L -o /usr/bin/Scrivener.AppImage
https://www.example.com/dir/Scrivener-1.9.0.1beta-x86_64_english_only-20211229.AppImage
=> Enter host password for user '':
=>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
=>                                  Dload  Upload   Total   Spent    Left  Speed
=>
--:--:-- --:--:-- --:--:--     0
--:--:-- --:--:-- --:--:--     0
=> curl: (22) The requested URL returned error: 401
=> Error: nu::shell::eval_block_with_input
=>
=>   × Eval block failed with pipeline input
=>    ╭─[/tmp/modules/script/script.nu:5:7]
=>  4 │   let config = $config
=>  5 │     | from json
=>    ·       ────┬────
=>    ·           ╰── source value
=>  6 │     | default [] scripts
=>    ╰────
=>
=> Error: nu::shell::non_zero_exit_code
=>
=>   × External command had a non-zero exit code
=>     ╭─[/tmp/modules/script/script.nu:16:8]
=>  15 │       print -e $'(ansi green)Running script: (ansi cyan)($script)(ansi reset)'
=>  16 │       ^$script
=>     ·        ───┬───
=>     ·           ╰── exited with code 22
=>  17 │     }
=>     ╰────
=>
=> ============================ Failed 'script' Module ============================

I'm at a loss.

Questions:

1. Do I need to define/export this environment secret at the start of the workflow in the build.yml file ? If so, how / where? IS it enough to just define it in the recipe file?
2. Does my above stanza need "SUPER_SECRET_PASSWORD" in BOTH places, or do I need to define which Environment, as well? I.e., is "MAGIC_CURL_STUFF" defined anywhere?

Sorry for the lengthy reply. My ignorance regarding both Github secrets and the blue-build process is creating a slew of failed builds. And now I'm obsessed with getting this to work. :-)

[gmpinder]

You'll have to set the secret in the env: block of the step in your Github actions file.