Question: Why is default value of token refresh 30 seconds?

[Visserr2]

Question: Why is default value of token refresh 30 seconds?

Is there a way to override this (I cannot find it) ? In practice always a new token will be created (unless there is a call triggered within the 30 seconds of the expiry time.

[kdubb]

That's configurable in the VaultClient I'm not sure if it's configurable via Quarkus config. I'd need to look it up.

More importantly, that's not a token refresh period. IF a token expires within the next "renew grace period" then the token will be refreshed. So if your token TTL is 5 minutes, and you have the default of 30 seconds grace period, your token will be updated after 4 minutes and 30 seconds.

It's not a configuration you should ever really need, unless your Vault instance is under such load that it takes 20+ seconds to respond to token renewals.

[Visserr2]

Hmm, that is not how I experience it. I tried it with following config (as test):
Vault uses gmt-time and logging has CEST-timezone (+2)

Token Expiry : 2minutes
max-lifetime: 30s
Grace period: 1m
-----------------------------------------------------------------
2024-09-28 00:11:44,654 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:13:10.923708300Z} (expires at 2024-09-27T22:13:10.923708300Z)
2024-09-28 00:12:15,149 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:13:10.923708300Z} (expires at 2024-09-27T22:13:10.923708300Z)
2024-09-28 00:12:45,604 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:13:10.923708300Z} (expires at 2024-09-27T22:13:10.923708300Z)
2024-09-28 00:12:45,653 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:14:45.653515100Z}
2024-09-28 00:13:13,805 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:14:45.653515100Z} (expires at 2024-09-27T22:14:45.653515100Z)
2024-09-28 00:13:44,098 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:14:45.653515100Z} (expires at 2024-09-27T22:14:45.653515100Z)
2024-09-28 00:14:14,796 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:14:45.653515100Z} (expires at 2024-09-27T22:14:45.653515100Z)
2024-09-28 00:14:45,522 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:14:45.653515100Z} (expires at 2024-09-27T22:14:45.653515100Z)
2024-09-28 00:14:45,566 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:16:45.566184900Z}
2024-09-28 00:15:15,867 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:16:45.566184900Z} (expires at 2024-09-27T22:16:45.566184900Z)
2024-09-28 00:15:46,536 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:16:45.566184900Z} (expires at 2024-09-27T22:16:45.566184900Z)
2024-09-28 00:16:17,543 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:16:45.566184900Z} (expires at 2024-09-27T22:16:45.566184900Z)
2024-09-28 00:16:17,588 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:18:17.588061100Z}
2024-09-28 00:17:19,062 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:18:17.588061100Z} (expires at 2024-09-27T22:18:17.588061100Z)
2024-09-28 00:17:49,776 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:18:17.588061100Z} (expires at 2024-09-27T22:18:17.588061100Z)
2024-09-28 00:17:49,820 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:19:49.818513300Z}
2024-09-28 00:18:15,527 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:19:49.818513300Z} (expires at 2024-09-27T22:19:49.818513300Z)
2024-09-28 00:18:46,250 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:19:49.818513300Z} (expires at 2024-09-27T22:19:49.818513300Z)
2024-09-28 00:19:16,694 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:19:49.818513300Z} (expires at 2024-09-27T22:19:49.818513300Z)
2024-09-28 00:19:47,208 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:19:49.818513300Z} (expires at 2024-09-27T22:19:49.818513300Z)
2024-09-28 00:19:47,251 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:21:47.251464800Z}
2024-09-28 00:20:20,656 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:21:47.251464800Z} (expires at 2024-09-27T22:21:47.251464800Z)
2024-09-28 00:21:08,837 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:21:47.251464800Z} (expires at 2024-09-27T22:21:47.251464800Z)
-----------------------------------------------------------------

Token expiry:  2 minutes
max-lifetime: 45s
Grace period: 1m
-----------------------------------------------------------------
2024-09-28 00:23:58,731 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) created new login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:25:58.731462800Z}
2024-09-28 00:23:58,796 DEBUG [io.qua.vau.run.VaultDynamicCredentialsManager] (vert.x-eventloop-thread-2) generated -crm_rw(database/creds) credentials:{leaseId: database/creds/-crm_rw/4J2y2tSLrQFcqnpIdogOfEYG, renewable: true, leaseDuration: 3600s, valid_until: Sat Sep 28 01:23:58 CEST 2024, username: v-approle-RAPDB_schema-crm_rw-itIgJETYuheBfu1dOlqm-1727475838, password:***}
2024-09-28 00:24:48,212 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:25:58.731462800Z} (expires at 2024-09-27T22:25:58.731462800Z)
2024-09-28 00:25:33,691 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:25:58.731462800Z} (expires at 2024-09-27T22:25:58.731462800Z)
2024-09-28 00:25:33,744 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:27:33.744598700Z}
2024-09-28 00:26:19,513 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:27:33.744598700Z} (expires at 2024-09-27T22:27:33.744598700Z)
2024-09-28 00:27:04,671 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:27:33.744598700Z} (expires at 2024-09-27T22:27:33.744598700Z)
2024-09-28 00:27:04,715 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:29:04.715380Z}
2024-09-28 00:27:50,002 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:29:04.715380Z} (expires at 2024-09-27T22:29:04.715380Z)
2024-09-28 00:28:36,024 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:29:04.715380Z} (expires at 2024-09-27T22:29:04.715380Z)
2024-09-28 00:28:36,068 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:30:36.068402300Z}
2024-09-28 00:29:21,905 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:30:36.068402300Z} (expires at 2024-09-27T22:30:36.068402300Z)
2024-09-28 00:30:07,912 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:30:36.068402300Z} (expires at 2024-09-27T22:30:36.068402300Z)
2024-09-28 00:30:07,957 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:32:07.957559700Z}
2024-09-28 00:30:53,648 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:32:07.957559700Z} (expires at 2024-09-27T22:32:07.957559700Z)
2024-09-28 00:31:40,403 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:32:07.957559700Z} (expires at 2024-09-27T22:32:07.957559700Z)
2024-09-28 00:31:40,445 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (vert.x-eventloop-thread-2) extended login token: {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:33:40.445614100Z}
2024-09-28 00:32:25,554 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:33:40.445614100Z} (expires at 2024-09-27T22:33:40.445614100Z)
2024-09-28 00:33:03,172 FINE  [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:33:40.445614100Z} (expires at 2024-09-27T22:33:40.445614100Z) <--- database credentials refreshed later
2024-09-28 00:33:41,050 WARN  [io.agr.pool] (executor-thread-1) Datasource '<default>': The server principal "v-approle--crm_rw-itIgJETYuheBfu1dOlqm-1727475838" is not able to access the database "" under the current security context.
-----------------------------------------------------------------

As you can see in the log the token only gets refresh around 30s or closer to expiry date. It looks like the database lease gets refreshed within grace period when the max-lifetime is over. As you can see in the latest example I triggered a database call around 00:33:03. Around this the the max-lifetime was over but the token wasn't being refreshed because it was not within the 30 seconds. The next refresh is 45 seconds later because thats when the max-lifetime is over. Only when I make a call before the token is expired I get an error because it has not been extended within the 30 window. So the db-lease isn't valid anymore.

Can you spot the error I have?

[Visserr2]

@kdubb Is this something you recognize? Because I having trouble with longer running jobs. The database leases are getting extendend every 30 minutes (max lifetime 30m and grace period 35 min) but the token itself is not. Because my token is valid for 1H the job throws error after one hour:
Datasource '': The server principal "" is not able to access the database "" under the current security context.

I have worked around this problem by making the token valid for longer. But I do not think that should be the fix.

[Visserr2]

@kdubb Sorry to bother you again. But I am still experience this problem. Can you see what is going wrong is logs above?

2024-09-28 00:33:03,172 FINE [io.qua.vau.cli.aut.VaultCachingTokenProvider] (agroal-11) using cached token {clientToken: ***, renewable: true, leaseDuration: PT2M, valid_until: 2024-09-27T22:33:40.445614100Z} (expires at 2024-09-27T22:33:40.445614100Z)

Why is is using a cached token when grace-period is 1m?

[Visserr2]

I tried to create a PR for this repo. But it takes too much time for me right now. I will try to do it later. For now I just overwrite the "default_renew_grace_period" in my app with following code:

This has fixed my problems. The Vault token is now getting refreshed too (at same time as de db-credentials).

[kdubb]

As you can see in the log the token only gets refresh around 30s or closer to expiry date.

This is as designed

It looks like the database lease gets refreshed within grace period when the max-lifetime is over. As you can see in the latest example I triggered a database call around 00:33:03. Around this the the max-lifetime was over but the token wasn't being refreshed because it was not within the 30 seconds.

@Visserr2 Are you saying that the grace period refresh is not based on the max-lifetime?

[kdubb]

The reason I'm asking is that would seem to mean that Vault is returning a lease duration that extends beyond the token's max-lifetime.

[kdubb]

@Visserr2 I've open PR #361 and updated the test for the token caching to verify that the grace period is operating as expected. Specifically, it simulates a token that expires after 1 hour and tests it's usage at 00:45:00, 00:59:20, and then 00:59:35 and it only extends the token at 59:35.

If you can help figure out the issue you are having by helping update, or creating a new test, that demonstrates your issue. That would go a long way to providing a solution.

[Visserr2]

Hello,

I mean the max-lifetime of the connection. The vault login-token is extended correctly within the renew-grace-period (declared in property file). Only the dynamic database credentials is only being extended within those 30 seconds grace-period (default grace period). But if I have a job running that isn't calling the database within those 30 seconds then the token will expire. The next time the job the will access the database it will say that the credentials arent valid anymore.

So the only fixed I could do is:

• set the lifetime of the token to the max length of the running job or
• Put a timer in the job that is calling the database within the 30 seconds of the expiry date or
• Overwrite the default grace period of 30 seconds.

If I call another api then its no problem because another connection will be made (new token). But if you have a running job and the dynamic database credentials isn't extended then the user that is being used doesn't have access anymore. And i'm not sure why the vault login-token will adhere to the renew-grace-period that is being declared in property file. But for the dynamic database credentials it should always be 30 seconds.