Token status list retrieval times out

[IngridPuppet]

Requests for retrieving previously published status lists time out and fail:

curl --location 'https://statuslist.eudi-adorsys.com/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c' \
--header 'Accept: application/statuslist+jwt'

This does not occur with non-existent lists:

curl --location 'https://statuslist.eudi-adorsys.com/statuslists/non-existent' --header 'Accept: application/statuslist+jwt'

Here are relevant logs:

2026-01-22T08:45:30.592262Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}: tower_http::trace::on_request: started processing request
2026-01-22T08:45:30.592323Z  INFO request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}: status_list_server::web::handlers::status_list::get_status_list: Cache miss for status list token: a4922b12-cb42-4a28-bbf3-5070d4f9e59c
2026-01-22T08:45:30.601938Z  INFO request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}: sqlx::query: summary="SELECT \"status_lists\".\"list_id\", \"status_lists\".\"issuer\", \"status_lists\".\"status_list\", …" db.statement="\n\nSELECT \"status_lists\".\"list_id\", \"status_lists\".\"issuer\", \"status_lists\".\"status_list\", \"status_lists\".\"sub\" FROM \"status_lists\" WHERE \"status_lists\".\"list_id\" = $1 LIMIT $2\n" rows_affected=1 rows_returned=1 elapsed=8.343088ms elapsed_secs=0.008343088
2026-01-22T08:45:30.602042Z  WARN request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}: status_list_server::utils::cert_manager::storage::aws: Cache error for certs-eudi-adorsys.com-cert_data.json: Redis error: Reconnecting failed: invalid peer certificate: certificate expired: verification time 1769040984 (UNIX), but certificate is not valid after 1767937602 (1103382 seconds ago)
2026-01-22T08:45:30.602111Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:apply_configuration: aws_smithy_runtime::client::orchestrator: timeout settings for this operation: TimeoutConfig { connect_timeout: Set(3.1s), read_timeout: Disabled, operation_timeout: Disabled, operation_attempt_timeout: Disabled }
2026-01-22T08:45:30.602140Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'serialization' phase
2026-01-22T08:45:30.602167Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'before transmit' phase
2026-01-22T08:45:30.602192Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op: aws_smithy_runtime::client::retries::strategy::standard: no client rate limiter configured, so no token is required for the initial request.
2026-01-22T08:45:30.602202Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op: aws_smithy_runtime::client::orchestrator: retry strategy has OKed initial request
2026-01-22T08:45:30.602212Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op: aws_smithy_runtime::client::orchestrator: beginning attempt #1
2026-01-22T08:45:30.602228Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}: aws_sdk_s3::endpoint_auth: resolving endpoint for auth scheme selection endpoint_params=EndpointResolverParams { inner: TypeErasedBox[!Clone]:Params { bucket: Some("status-list-adorsys"), region: Some("eu-central-1"), use_fips: false, use_dual_stack: false, endpoint: None, force_path_style: false, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: Some("certs-eudi-adorsys.com-cert_data.json"), prefix: None, copy_source: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }, property: {} }
2026-01-22T08:45:30.602277Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}: aws_smithy_runtime::expiring_cache: An item existed but it expired. expiry=SystemTime { tv_sec: 1769040903, tv_nsec: 399655455 } delta=Ok(30627.202618191s)
2026-01-22T08:45:30.602303Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}:lazy_load_identity:default_credentials_chain: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=the credential provider was not enabled: environment variable not set (CredentialsNotLoaded(CredentialsNotLoaded { source: Some("environment variable not set") }))
2026-01-22T08:45:30.602327Z  INFO request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}:lazy_load_identity:default_credentials_chain:credentials_provider_chain{provider=Profile}: aws_config::profile::credentials: loaded base credentials creds=Credentials { provider_name: "ProfileFile", access_key_id: "AKIA6JKEXSVAUZ3II25Z", secret_access_key: "** redacted **", expires_after: "never" }
2026-01-22T08:45:30.602351Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}:lazy_load_identity:default_credentials_chain: aws_config::meta::credentials::chain: loaded credentials provider=Profile
2026-01-22T08:45:30.602368Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}:lazy_load_identity: aws_smithy_runtime::client::identity::cache::lazy: identity cache miss occurred; added new identity (took 73.749µs) new_expiration=2026-01-22T09:00:30.602273Z valid_for=899.99990522s partition=IdentityCachePartition(5)
2026-01-22T08:45:30.602386Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}: aws_smithy_runtime::client::identity::cache::lazy: loaded identity
2026-01-22T08:45:30.602396Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}:orchestrate_endpoint: aws_smithy_runtime::client::orchestrator::endpoints: resolving endpoint endpoint_params=EndpointResolverParams { inner: TypeErasedBox[!Clone]:Params { bucket: Some("status-list-adorsys"), region: Some("eu-central-1"), use_fips: false, use_dual_stack: false, endpoint: None, force_path_style: false, accelerate: false, use_global_endpoint: false, use_object_lambda_endpoint: None, key: Some("certs-eudi-adorsys.com-cert_data.json"), prefix: None, copy_source: None, disable_access_points: None, disable_multi_region_access_points: false, use_arn_region: None, use_s3_express_control_endpoint: None, disable_s3_express_session_auth: None }, property: {TypeId(0xccc6d6e5ed18c11598203bdfb9f0a819): TypeErasedBox[!Clone]:Identity { data: Credentials { provider_name: "ProfileFile", access_key_id: "AKIA6JKEXSVAUZ3II25Z", secret_access_key: "** redacted **", expires_after: "never", property_0: TypeErasedBox[Clone]:[CredentialsProfile] }, expiration: None, property_0: TypeErasedBox[!Clone]:FrozenLayer(Layer { name: "IdentityResolutionFeatureIdTracking", items: [TypeErasedBox[!Clone]:Set([CredentialsProfile])] }) }} }
2026-01-22T08:45:30.602429Z DEBUG request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}:S3.GetObject{rpc.service="S3" rpc.method="GetObject" sdk_invocation_id=6815747 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}:orchestrate_endpoint: aws_smithy_runtime::client::orchestrator::endpoints: will apply endpoint Endpoint { url: "https://status-list-adorsys.s3.eu-central-1.amazonaws.com", headers: {}, properties: {"authSchemes": Array([Object({"signingRegion": String("eu-central-1"), "signingName": String("s3"), "disableDoubleEncoding": Bool(true), "name": String("sigv4")})])} } endpoint_prefix=None

An issue with Redis and/or certificate management?

Estimate: 2 days

[Hermann-Core]

Hello @IngridPuppet, thanks for pointing this out. Let’s try to troubleshoot the issue step by step.

First, looking at the following log:

2026-01-22T08:45:30.592323Z  INFO request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}: status_list_server::web::handlers::status_list::get_status_list: Cache miss for status list token: a4922b12-cb42-4a28-bbf3-5070d4f9e59c

This is a normal cold-path scenario. Since the status list is not present in the cache, the service correctly falls back to fetching it directly from the database.

This is confirmed by the next log entry:

2026-01-22T08:45:30.601938Z  INFO request{method=GET uri=/statuslists/a4922b12-cb42-4a28-bbf3-5070d4f9e59c version=HTTP/1.1}: sqlx::query: summary="SELECT \"status_lists\".\"list_id\", \"status_lists\".\"issuer\", \"status_lists\".\"status_list\", …" db.statement="\n\nSELECT \"status_lists\".\"list_id\", \"status_lists\".\"issuer\", \"status_lists\".\"status_list\", \"status_lists\".\"sub\" FROM \"status_lists\" WHERE \"status_lists\".\"list_id\" = $1 LIMIT $2\n" rows_affected=1 rows_returned=1 elapsed=8.343088ms elapsed_secs=0.008343088

Here we can see that the status list was successfully retrieved from the database, so the issue is likely occurring after this point. We’ll need more detailed logs to identify where things go wrong next.

Regarding the Redis error, the Redis server (or proxy) is presenting an expired certificate. As a result, Redis is effectively unusable for this service, which triggers the fallback logic that attempts to retrieve the data from S3 instead.

[IngridPuppet]

Any update on this issue?

[Hermann-Core]

Any update on this issue?

Still trying to replicate the issue locally.

[Awambeng]

I identified the cause of the deployment failure and I’m currently working on a fix.
The root cause is an issue with the connection between the server and the Redis client.