Identity Server warmup

[janssensjeroen]

Hi,

we have a constant load of multiple requests per second hitting our IdentityServer. When releasing our IdentityServer or scaling out, we notice that the first calls suffer from an increased latency. We have health probes implemented and traffic only arrives on the new instances once the health probes report healthy. I was wondering what we can do during startup to warmup the instance. Could we warmup the caches for example?

Thanks in advance

Jeroen

[wcabus]

You can perform some tasks to help speed up incoming requests, yes.

Preloading cached data

During the startup phase, after initializing IdentityServer's services and pipeline, you can preload the client and resource data. Note that IClientStore does not offer a way to load (or cache) all client data at once, so I would only focus on the most-requested clients and cache those using their client ID:

var builder = WebApplication.CreateBuilder(args);

var app = builder
    .ConfigureServices()
    .ConfigurePipeline();

// Resolve a scope to retrieve an IClientStore and IResourceStore
await using (var scope = app.Services.CreateAsyncScope())
{
    var clientStore = scope.ServiceProvider.GetRequiredService<IClientStore>();
    string[] clientsToCache = ["client1", "client2"];
    foreach (var clientId in clientsToCache)
    {
        // load and cache the most-requested clients by ID
        await clientStore.FindEnabledClientByIdAsync(clientId); 
    }
    
    // load and cache all identity resources, API scopes and API resources
    var resourceStore = scope.ServiceProvider.GetRequiredService<IResourceStore>();
    await resourceStore.GetAllResourcesAsync();
}

app.Run();

Trigger the /.well-known/openid-configuration endpoint

If you query the discovery document after the startup phase, you will:

• trigger retrieving the signature validation keys which loads and caches the keys;
• potentially immediately renew signature validation keys;
• load and cache all resources (identity resources, API scopes and API resources)

However, there is a caveat: if any of the SigningAlgorithms is configured with UseX509Certificate = true, then you can only trigger this endpoint using an HTTP request to warm up IdentityServer unless IdentityServer is configured with a static issuer URI.

Use a startup probe

If you are running IdentityServer in a Kubernetes cluster, you could use a startup probe to delay traffic to a (new) instance until it's fully ready to handle incoming requests. I wouldn't use this to delay until you've triggered loading the discovery document, but you could delay it until you've preloaded the client and resource data, for example.

[janssensjeroen]

Hi @wcabus,

Thanks for the input. I implemented preloading of resources, clients and keys in combination with a startup probe that reports healthy once the preloading is finished. That drastically improved the latency on the first calls after a release or after scaling out.

To preload the keys I used the IKeyManager. That is the way to go?

Kind regards

Jeroen

[wcabus]

IKeyManager will cache the keys indeed, but in case a key needs to be renewed, you would still potentially have the issue I described earlier (when UseX509Certificate is set to true)

[janssensjeroen]

@wcabus Thanks for the warning. We indeed use the UseX509Certificate on our SigningAlgorithms. The KeyManager uses the IIssuerNameService which uses the HttpContext to retrieve the issuer name, so an HttpContext needs to be available.

Just to be clear: we currently have our identity server exposed on multiple domains. Is there a difference on the keys generated when generation happens from a request for domain A versus domain B? If so, what is the impact?

[wcabus]

Hard to say. From Duende IdentityServer's point of view, there is no impact: the signing key is merely wrapped into a X509 container which happens to need an issuer value, and that issuer value is derived from the incoming HTTP request to fetch the origin unless a static IssuerUri is configured.

When a subsequent request arrives on another issuer domain, for example, when a client authenticates, the existing key material is used and no issuer verification is being performed on the X509 key container. Reason being that the key material was created by Duende IdentityServer and is therefore already trusted.

However, if your solution shares the public certificate with a client and that client does verify the issuer domain, then you may run into a situation where the issuer mismatches.