Occasional "access denied" with dynamic database credentials from Vault

[wiebeck]

We have a pretty nice setup that configures dynamic database credentials using Hashicorp Vault for our Quarkus Kubernetes services. Quarkus authenticates against Vault using Kubernetes authentication and we also use Vault as config source. For the dynamic db credentials we pass the following environment variables to Quarkus:

QUARKUS_DATASOURCE_CREDENTIALS_PROVIDER = vault
QUARKUS_DATASOURCE_DB_KIND = mysql
QUARKUS_DATASOURCE_JDBC_URL = jdbc:mysql://.....
QUARKUS_VAULT_CREDENTIALS_PROVIDER_VAULT_DATABASE_CREDENTIALS_ROLE = some_configured_role

(yes, we are aware of the fact that setting QUARKUS_VAULT_CREDENTIALS_PROVIDER_VAULT_DATABASE_CREDENTIALS_ROLE as environment variable only works if we also set quarkus.vault.credentials-provider.vault.database-credentials-role in application.properties.)

When the service starts up everything is working fine, from Vault Kubernetes authentication over reading config from Vault to database connection. But from time to time we get these errors in the logs:

Could not connect to address=(host=*********)(port=3306)(type=master) : (conn=64578) Access denied for user 'v-kubernetes-********'@'*******' (using password: YES)

We are aware of the fact that the credentials are short-lived and need to be re-acquired before they expire. We were under the impression that Quarkus takes care of doing that but it seems to fail on that from time to time.

Is there any config parameter we can use to tweak Quarkus behavior or did we just encounter a simple bug? Maybe it's even an issue of timezones (we're located in UTC+1)?

Thanks for helping out. :)

[sberyozkin]

CC @vsevel @kdubb

[nomaster]

I have discovered that Quarkus has a 'grace period' setting of 1 hour (by default). I guess that with a token lifetime of 1 hour there is a small margin in which the token has actually expired, but the application didn't renew it yet.

[kdubb]

The grace period reduces the TTL of the credentials by the grace period duration. So, it's the opposite. For example, if the credentials TTL is 3hrs and the grace period 1hr then the credentials are renewed after only 2hrs.

[kdubb]

The grace period setting mentioned above should fix any problems like this.

Enable debug logging for io.quarkus.vault.runtime.VaultDynamicCredentialsManager. You should see logs about when the leases are created & renewed. Which should help figure out if that's the issue.

[wiebeck]

I will take a look at the debug logs by setting

quarkus.log.category."io.quarkus.vault.runtime.VaultDynamicCredentialsManager".level=DEBUG

Thanks for the hint, @kdubb !

[wiebeck]

We finally solved our issue. When browsing the Quarkus Vault config options we stumbled up the following:

you need to make sure there will be attempts to fetch secrets within the renewGracePeriod, because that is when the renewals will happen. This is particularly important for db dynamic secrets because if the lease reaches its ttl or max_ttl, the password of the db user will become invalid and it will be not longer possible to log in.

This is exactly what happened to us. We were under the impression that Quarkus will automatically keep the Vault token "fresh" at any time but our service (low traffic) does not constantly communicate with Vault so that the login token expired and with it the dynamic DB credential pair.

We now added a @Scheduled annotated method to one of our beans that performs a simple authenticated Vault operation (e.g. VaultAuthManager#getClientToken()). This will validate and renew the currently cached token if necessary. This keeps the database credentials alive and the exceptions away. ;-)

[vsevel]

btw VaultAuthManager is not part of the public api
see also this related discussion quarkiverse/quarkus-vault#72

[wiebeck]

With quarkus-vault v4.0.0 this problem pops up again, even for access to KV secrets. Check out quarkiverse/quarkus-vault#261 for my findings.