feat: add decryption capability in userdata with Jinja template

[acolombier]

This aims to fix #4417 by implementing the plan proposed in #6655. It uses GPG to pass sensitive information in an encrypted way.

• I have signed the CLA: https://ubuntu.com/legal/contributors
• I have included a comprehensive commit message using the guide below
• I have added unit tests to cover the new behavior under tests/unittests/
  • Test files should map to source files i.e. a source file cloudinit/example.py should be tested by tests/unittests/test_example.py
  • Run unit tests with tox -e py3
• I have kept the change small, avoiding unnecessary whitespace or non-functional changes.
• I have added a reference to issues that this PR relates to in the PR message
• I have updated the documentation with the changed behavior.
  • If the change doesn't change the user interface and is trivial, this step may be skipped.
  • Cloud-config documentation is generated from the jsonschema.
  • Generate docs with tox -e doc.

Proposed Commit Message

feat(userdata): add decryption capability in userdata

When using Jinja template, this commit add a helper function 
which can be used to decrypt an encrypted message at 
runtime, using a GPG private key stored in the cloudinit's 
user keyring, or the keyring pointed by the envvar 
`GNUPGHOME`

Fixes GH-4417

Additional Context

1. This currently requires to add a dependencies to python-gnupg, which might turn out to be controversial. Note that due to the simple use of GPG (--decrypt), it should be possible to also directly wrap GPG invocation by a subprocess Now use the command directly via std lib.
2. I have added documentation, by I expect some user manual text might need updating too. Since I couldn't find where this would be the most relevant, I am keeping this as a TODO, and will follow your feedback as of where to document this.

Test Steps

Unit test can be use to confirm it works. To test on a real setup, make sure to install the GPG key with "encrypt" usage on the cloudinit's user keyring (root by default) or to any keyring pointed by the GNUPGHOME envvar (e.g using systemd).
Encrypt sensitive value and pass it to the userdata in base64 format (see doc/userdata.txt for further details)

Merge type

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

[holmanb]

Thanks for the proposal.

Test failures aside, this isn't packaged in Ubuntu main - I expect other distros would need packaging work too - so this approach is a non-starter.

On another note, this only addresses one security concern (secrets in meta-data), while other significant ones remain: platform trust, platform confidentiality, etc. I don't think that this is the path that we would prefer to take - a more comprehensive TPM-based solution would be preferred.

[acolombier]

Thank you for your initial input.

this isn't packaged in Ubuntu main

That's a fair point. Would it make sense to explore this as an optional feature, perhaps enabled conditionally, whether GPG is available or not? (note that I have suggested dropping the added Python deps and use subprocess instead)

this only addresses one security concern (secrets in meta-data), while other significant ones remain

I understand your perspective. To clarify, I actually submitted a separate feature request (#6655) to address userdata sensitive info separately to platform trust, as I see these as distinct but complementary security layers. For context, TPM-based solutions might not always be practical in PaaS environments, where TPMs are often virtualised and don’t provide the level of trust as in bare-metal/IaaS scenarios.
In those cases, the security guarantees the current PR offers - such as verifying the VM image contains the correct GPG key to allow decryption - can still be valuable for protecting sensitive userdata, and is sufficient in most case for PaaS on public cloud.
That said, I’m open to discussion if there are other attack vectors I might be missing. Would it be possible to revisit #6655 and consider this PR as a focused solution for userdata encryption in public cloud contexts? My immediate need is specifically around securing sensitive information in userdata, and it seems that TPM support is a separate effort which I am not equip to implement. (No IaaS available for development) .

[acolombier]

Hi @holmanb, I have pushed a fixup that brings the two mentioned changes above:

• Drop the introduced python-gnupg dependencies: no need to change packages or distribution requirement, everything now relies on std lib (except for testing, tho I could change that too)
• Make this feature optional. If gpg is not installed on the machine, this function will just skip process. This way, no need for gpg to be a dependency.

This certainly would require more tests and likely more explicit documentation, but keen to hear if this is now aiming to something that could be accepted, before I spent more time on it.

[github-actions]

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging blackboxsw, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag blackboxsw to reopen it.)

[acolombier]

Hi @blackboxsw, this is waiting for feedback

[holmanb]

I don't think that this proposal fits the current scope of this project.

The core functionality proposed here is already possible. A user that is willing to modify an image to deploy GPG keys can already craft a runcmd / bootscript / etc to access encrypted secrets embedded in user-data.

Rather than implementing a new UI for a preexisting capability, a proposal that documents how to do this would be welcome. Or, like I mentioned before, it would be more interesting to see a new solution that utilizes cloud capabilities to provide stronger guarantees than are currently possible with cloud-init.

[acolombier]

The core functionality proposed here is already possible. A user that is willing to modify an image to deploy GPG keys can already craft a runcmd / bootscript / etc to access encrypted secrets embedded in user-data.

I don't think this is quite accurate. Of course, you can run gpg command directly to decrypt files you would have passed first, but this implies that:

• The whole file is encrypted, creating unnecessary obfusaction on configurations or elements that could be read by non-privilege user. This is the main driver for this feature as organisation relying on GitOps will often have multiples personas editing user-data templates.
• This doesn't work for non-file (e.g users:password, which is particularly useful if you need a two-way option, as opposite to a password hash)

Furthermore, if we follow your thinking, then TPM is also already supported and can be used to decrypt files using clevis decrypt tpm2 $OPT < /path/to/file in runcmds.

Here is some cloud-init userdata example to highlight the value

users:
  - name: machine
    passwd: |-
      {{gpgdecrypt("bG9yZW0gaXBzdW0K")}}
write_files:
- content: |
    api_key: {{gpgdecrypt("bG9yZW0gaXBzdW0K")}}
    site: datadog.eu

    tags:
    - "foo:yes"
    - "bar:no"

    cloud_provider_metadata:
      - "aws"
      - "azure"
      - "gcp"

    logs_enabled: true
    logs_config:
      auto_multi_line_detection: true

   # Truncating other log ingestion specific config which would benefit being visible and 
   # editable by a larger audience

As you can see with this datadog config file, only api_key is worth obfuscating. The rest is relevant to keep readable and modifiable so other contributors can (in this special case) add new tags or change log ingestion config. I peaked DataDog as example, but there is many other usecases.

For full disclosure, here is how we previously workaround this lack of feature:

# No simple workaround for users
write_files:
- path: /etc/datadog-agent/datadog.yaml
  content: |
    api_key: $DD_API_KEY
    site: datadog.eu

    tags:
    - "foo:yes"
    - "bar:no"

    cloud_provider_metadata:
      - "aws"
      - "azure"
      - "gcp"

    logs_enabled: true
    logs_config:
      auto_multi_line_detection: true
- path: /var/run/secrets/dd_api_key.gpg
  content: bG9yZW0gaXBzdW0K
runcmds:
- sed -i 's/$DD_API_KEY/'`base64 -d /var/run/secrets/dd_api_key.gpg | gpg --decrypt`'/' /etc/datadog-agent/datadog.yaml

Finally, just to close the argument of opposing this feature with TPM, here is how the same can be accomplished with TPM encryption:

write_files:
- path: /etc/datadog-agent/datadog.yaml
  content: |
    api_key: $DD_API_KEY
    site: datadog.eu

    tags:
    - "foo:yes"
    - "bar:no"

    cloud_provider_metadata:
      - "aws"
      - "azure"
      - "gcp"

    logs_enabled: true
    logs_config:
      auto_multi_line_detection: true
- path: /var/run/secrets/dd_api_key.secret
  content: |-
    eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIiwiY2xldmlzIjp7InBpbiI6InRwbTIiLCJ0cG0yIjp7Imhhc2giOiJzaGEyNTYiLCJqd2tfcHJpdiI6IkFOMEFJTVFTSnA0ZEtTV0E5WDBGNmlJR0lQOURraV9zTXE0NG1DcGJOVDBrOUtrQ0FCQ2dTSTlZY0VYdUVaY1JWd2pFOFFBRnNEZ1FHaFBHRnpPakpVVmI5WDlUamlfUGo2Vl94MjZ1OUQ3T0RZbVVqTjhEV1A3b3NzOVMwT0NwYVhqOUhzTTd2V3lJOU5XMU90VTdweFVYZ19VWjVMRDltcmhVTVIxdWI0akZNOGt6V3N4SFFxcnN2cjF4b1h6dUtKUzRSazg4VFF1WURNcUY2QVZKcWxJWXJJUUluMzFSTlZqMG5UV0oxRHdDT1M4bDdpWkVzWU5LaGRMYXNUTkhHNjNwNW1LYzBjRnFPY0ZKZ3FMMzhFT1ZibXhYamZvRV9VNjRIV0dEaHciLCJqd2tfcHViIjoiQUM0QUNBQUxBQUFFMGdBQUFCQUFJTzcxWFlMSTJEUEttS28xcnBTMlVacDdvVVMxZ3hSalA5bWFZLVNGRldEdiIsImtleSI6ImVjYyJ9fX0..Kf-C6HWum3FilI9V.MzXraw.13HOCcxhQIJzYuJMwFgnfA
runcmds:
- sed -i 's/$DD_API_KEY/'`clevis decrypt tpm2 '{"pcrs":"7"}' < /var/run/secrets/dd_api_key.secret`'/' /etc/datadog-agent/datadog.yaml

[holmanb]

here is how we previously workaround this lack of feature:

The requirements were met with existing cloud-init features. This is similar to the documentation example previously suggested.

The primary objection to this proposal is that there is limited usefulness and user-base for a cloud-init feature that cannot be used without building a custom image.

Additionally, this proposal increases the size of the UI without adding new functionality - a burden for upstream to maintain without a benefit to the majority of users.

but this implies that:

The whole file is encrypted...

The configuration could be stored as a template and rendered during a continuous deployment step.

Alternatively, cloud-init has boothook scripts, user-data scripts, MIME, and cloud-config archive as possible formats to encrypt only part of the file.

Furthermore, if we follow your thinking, then TPM is also already supported and can be used to decrypt files using clevis decrypt tpm2 $OPT < /path/to/file in runcmds.

A preferred solution would use a (v)TPM and not require the user to modify the image. Besides encrypting the user-data, this would be transparent to the user.

[acolombier]

this proposal increases the size of the UI without adding new functionality

"having workaround to accomplish a functionality" != "having the functionality"

Closing as we have wasted enough time already.