Configuration hardening

[cocoon]

As this topic came from other services for me, I was thinking about how to avoid secrets in environment variables to avoid them to show up / leak in ps, logs, errors.

As rauthy currently fully handles config in environment vars, even the config file is completely loaded as env vars, everything will be shown there.

Don't know if this will be an option, but I collected some ideas here.

I found currently 3 options that could be done.

Option 1 (Bash Process Subsitution)

This idea I found here:
https://smallstep.com/docs/step-ca/certificate-authority-server-production/#avoid-storing-passwords-in-environment-variables

Option 2 (delete config file)
If the config file would only be parsed once for importing, I thought an easy way would be to inject the configfile, set an env var like "delete_config_file_after_load" and delete the file after everything was read.

Option 3 (secrets manager / vault)

This is what I would like to use, and centralize configs.

Starting looking into secrets managers I found RustyVault, an early compatible alternative to HCP Vault:
https://github.com/Tongsuo-Project/RustyVault

In combination with "config-vault-rs":
https://github.com/edugzlez/config-vault-rs

One can then do:
set env var with token
(I wouldlike to have a time limited token or OTP)

fn load_config() -> Result<Config, ConfigError> {
    let vault_source = VaultSource::new(
        "http://127.0.0.1:8200".to_string(),  // Vault address
        "hvs.EXAMPLE_TOKEN".to_string(),      // Vault token
        "secret".to_string(),                 // KV mount name
        "dev".to_string(),        // Secret path
        KvVersion::V2, // KV Version
    );

    // Build configuration incorporating Vault and other sources
    Config::builder()
        .add_source(vault_source)
        // You can add other configuration sources
        // .add_source(config::File::with_name("config/default"))
        // .add_source(config::Environment::with_prefix("APP"))
        .build()
}

All this would require that rauthy would use a central serializable config object that then is either filled by env vars, infos from config file (best only loaded once at the beginning), or remote config from vault.

A quick search in rauthy source for "env::" shows that currently many calls directly ready from env.

[sebadob]

even the config file is completely loaded as env vars

Yes, in-memory. If you dont export globally in your shell upfront, and only read them from the file, you won't see them "everywhere". Where are these in-memory values shown to you?

[sebadob]

Ahh, I see. it does an env::set_var() internally.

I do have the centralized config on my TODO, as I don't like these many LazyLock<_>s right now. It grew like this over time.
The idea is to maybe change this to a toml file and accept env vars for these situations, where it makes sense to be overwriteable in that way.

Edit:

Usually, I don't have issues with these env vars, because everything is hidden inside the container and secrets are even auto-masked, but yes this is not that nice if you run it natively.

[cocoon]

Impressive quick response time! You are just waiting for questions, right? :)

centralized config on my TODO

Ah great to read that it might come, that would be the hard part.

The remote config support (config-vault-rs) is not big, just some lines of code.
Could either be directly integrated or use the external lib, maybe even with feature flag, but is is not much code.

But as I thought the topics fits to rauthy it might be a nice feature.

Even thought about you could also do secrets management in rauthy but maybe too many features to support, and as long as you are more or less working alone ...

btw: many thanks for the great work, it is so nice to follow your improvements, even if I was thrown back because of a big incident in a datacenter, I needed to stop working on implementing it and will slow me down a bit currently

[sebadob]

The remote config support (config-vault-rs) is not big, just some lines of code.

The change to a local, probably toml file will be the first step. The "bigger" part about it will be finding the best way how to do this, when I still want to be able to overwrite many values via ENV or in some other way, without having too much boilerplate again.

Some things like a connector to a vault could probably just be added in, if people would like this.

Even thought about you could also do secrets management in rauthy but maybe too many features to support, and as long as you are more or less working alone ...

Actually, I already thought about that. I rather implement things myself than rely on some big company's applications that might just change the license out of the blue one day. If it's kept simple, it should not be too much work to implement and all the encryption scaffolding is implemented and working already. The one thing that might be a bit more work, is if I wanted to add the possibility of sealing and unsealing.

I don't like relying on Hashicorp Vault and others because I don't really trust their licensing, like it's the case for a lot of software these days. I always try to stick to things under the umbrella of the CNCF.

[cocoon]

I still want to be able to overwrite many values via ENV or in some other way, without having too much boilerplate again

have you seen the config example?

It looks like it would be a nice fit for it, all in one:

• load from remote
• load from env
• load from file

 Config::builder()
        .add_source(vault_source)
        // You can add other configuration sources
        // .add_source(config::File::with_name("config/default"))
        // .add_source(config::Environment::with_prefix("APP"))
        .build()

Actually, I already thought about that. I rather implement things myself

😄 OK I can understand that, you want to have all under your control at best, but things can grow so quickly ..

I don't like relying on Hashicorp Vault and others because I don't really trust their licensing

That was why I was searching for alternatives, and there is:

OpenBao (Go)
https://github.com/openbao/openbao

Vaultwarden (Rust)
https://github.com/dani-garcia/vaultwarden/

OIDCVarden (Vaultwarden fork working nicely with Rauthy, I mentioned it there, plan to use it) (Rust)
https://github.com/Timshel/OIDCWarden

and new one:

RustyVault (Rust)
https://github.com/Tongsuo-Project/RustyVault

So maybe just work together with RustyVault or fork it.
To have a comatiple API to HCP Vault might be nice to have.

[sebadob]

have you seen the config example?

Yes I have, thanks. If I implement any external connectors, they should be fully opt-in and I don't want to rely on them the build my whole config in the first place, that's why I would create a separation solution for this.

I would probably look at something like RustyVault first, if it has a compatible API, yes. Vaultwarten has nothing to do with HCP vault, it is a fork of Bitwarden. Have not heard of OIDCVarden yet, but it's nice that they implement the SSO part for vaultwarden, which was a reasons for me to not use it so far.

OpenBao on the other hand is part of the Linux Foundation, which is a big plus for me. So yeah, probably no real need to add this functionality to Rauthy.

[cocoon]

I was thinking about the auto renewal, and yes this would need some custom renewal into the vault and restart the application that way anyways.
It would just be the easiest first implementation.

I was searching and it really seems that the popular acme tools lack the ability to update certs in vaults.
I found some requests, like here, and there referring to 2 more requests:
cert-manager/cert-manager#6012

The recommendation there is to use cert-manager-sync, so for Kubernetes there exists at least something like it.

Traefik Enterprise has an agent that can use vault:
https://doc.traefik.io/traefik-enterprise/tls/distributed-acme/

polling in intervals for new ones it seems:

vault.syncInterval

Optional, Default="1m"

Defines the interval at which the agent checks if an ACME certificates was manually deleted in Vault.

The local filesystem can be observed for changes and certs can be reloaded pretty easily (even though Rauthy does not do this yet)

So currently the only way of auto renewal for rauthy would be the same, replace cert files, manually or scripted shutdown and restart rauthy.

If you plan to add a filesystem watcher for new certificates to reload them dynamically, then adding a REST API or something to trigger a dynamic reload for vault stored certs ones could use the same method, just needs to be triggered by an agent or an internal timer.

For keeping it simple it would make sense to only support one vault, not multiple, for example to store different certs in different vaults.

Then only one vault source config could be added and then only path to the cert in a vault would be needed, something like

#tls_raft_key_vault_path = "tls/tls.key"
#tls_raft_cert_vault_path = "tls/tls.crt"

#tls_api_key_vault_path = "tls/tls.key"
#tls_api_cert_vault_path = "tls/tls.crt"

#cert_path_vault_path = 'tls/cert-chain.pem'
#key_path_vault_path = 'tls/key.pem'

I will experiment now for a while and implement something like this.

[sebadob]

Traefik Enterprise has an agent that can use vault

Yeah but when you use Traefik to terminate your TLS, you probably don't want to run Rauthy with TLS as well.

So currently the only way of auto renewal for rauthy would be the same, replace cert files, manually or scripted shutdown and restart rauthy.

Correct. File-watchers will probably come with the next release.

Then only one vault source config could be added

The thing with the vault config is, that even if you would store it inside some vault, you still need to pass the config vars to Rauthy how to connect to the Vault in the first place. But, when you do this, you could also just pass the config itself.

[cocoon]

Yeah but when you use Traefik to terminate your TLS, you probably don't want to run Rauthy with TLS as well.

That was just an example, where they use a vault, not that I want to use Traefik.

Correct. File-watchers will probably come with the next release.

nice :)

you still need to pass the config vars to Rauthy

I have it already working as POC, config is no problem, works like a charm.
I only need some env vars and if VAULT_TOKEN is set, I go the route to load the config from the vault.

set VAULT_ADDR=http://127.0.0.1:8201
set VAULT_TOKEN=hvs....
set VAULT_MOUNT=secret
set VAULT_PATH=rauthy
set VAULT_VERSION=2

Additionaly I thought about an extra file if it exists: config.vault
that would then contain the vars for the vault, if someone prefers to inject it as file and delete it afterwards or so ...

Currently trying (fighting) to load certs from strings ...

It is all a bit ugly currently, needs to be made nice, but it works with just a few changes.

I extended the RauthyConfig by a HashMap that contains all secrets from the Vault:

WIP ...

[cocoon]

Oh I think it works ... but will have to verify further ...

as I said, just very early testing ...

[cocoon]

Loading certs, also DER (needs to be base64 encoded) is working fine.

I extended the RauthyConfig by a HashMap that contains all secrets from the Vault

This has some drawbacks.
I thought it would be nice to have it available like all other configs but:

• this would have the whole config file (text) and certs loaded for the whole runtime in memory
• I was thinking about a "write config/certs to vault" method and that would then need to explicitly exclude the additional config

I am now trying to change it to only keep a separate static config holding the vault source information.

[cocoon]

Today when starting to create a new docker container for a ca I had the idea:
"why not serve vault secrets as files with FUSE"

As I had already experience when I tried to program an FS for mtp, I thought, it must not be much of a problem, could then serve it time limited etc. That would make it transparent to programs that read secrets from FS.

And guess what, some other smart people had the same idea many years ago already of course.
So I thought I collect some here.

Vault filesystem (and Docker volume plugin) [GO]
https://github.com/asteris-llc/vaultfs

Experimental FUSE clients for Hashicorp REST APIs (ConsulFS,VaultFS,NomadFS,TFEFS,KubernetesFS) [c++]
https://github.com/jboero/hashifuse
Video Talk with nice details: https://www.hashicorp.com/de/resources/hashicorp-apis-via-hashifuse

Hashicorp Vault fuse filesystem (unfinished) [Python]
https://github.com/PsyanticY/vaultfs

hashicorp/vault#110

[wt]

I've been digging the way that k8s handles secrets.