Apache HTTP Client: Connection pool shut down #1058
Description
Describe the Bug
Hi Team,
We have noticed that there is TokenFetch Failure exception during the DestinationAccess using our ProxyDestinationLoader(we are using the proxydestinationloader for loading the destination name in runtime, refer https://sap.stackenterprise.co/questions/40429/40530). Circuit breaker is open and doesn't permit the token fetch call leading to the destination fetch failure. The below is the attached stack trace. Note: We are noticing this error only when the application is running for a long time, it is not reproducible immediately after a fresh deployment.
Steps to Reproduce
- Deploy the application in Cloud foundry
- Access a destination using the ProxyDestination Loader.
- Issue occurs
Expected Behavior
- TokenFetch shouldn't fail.
- Destination should be loaded.
Screenshots
No response
Used Versions
- Java and Maven version via
mvn --version: maven.version: 3.10.3 & java v17 (docker image 'devxci/mbtci-java17-node18:latest) - SAP Cloud SDK version: 5.24.0
- Spring Boot or CAP version: spring boot 3.5.7 & cap java: 3.10.3
Dependency tree via mvn dependency:tree
--- dependency:3.8.1:tree (default-cli) ---
[INFO] +- com.sap.cds:cds-starter-spring-boot:jar:3.10.3:compile
[INFO] | +- com.sap.cds:cds-services-api:jar:3.10.3:compile
[INFO] | | +- com.sap.cds:cds4j-api:jar:3.10.3:compile
[INFO] | | \- com.sap.cloud.environment.servicebinding.api:java-core-api:jar:0.21.0:compile
[INFO] | +- com.sap.cds:cds-services-impl:jar:3.10.3:runtime
[INFO] | | +- com.sap.cds:cds-services-messaging:jar:3.10.3:runtime
[INFO] | | | \- jakarta.jms:jakarta.jms-api:jar:3.1.0:runtime
[INFO] | | +- com.sap.cloud.environment.servicebinding.api:java-access-api:jar:0.21.0:compile
[INFO] | | \- org.apache.commons:commons-csv:jar:1.14.0:runtime
[INFO] | +- com.sap.cds:cds-feature-jdbc:jar:3.10.3:runtime
[INFO] | | +- com.sap.cds:cds4j-runtime:jar:3.10.3:compile
[INFO] | | | \- com.sap.cds:cds4j-jdbc-spi:jar:3.10.3:compile
[INFO] | | \- com.sap.cds:cds4j-tx:jar:3.10.3:runtime
[INFO] | +- com.sap.cds:cds-framework-spring-boot:jar:3.10.3:runtime
[INFO] | +- org.springframework.boot:spring-boot-starter-web:jar:3.5.7:compile
[INFO] | | +- org.springframework.boot:spring-boot-starter-json:jar:3.5.7:compile
[INFO] | | | +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.19.2:compile
[INFO] | | | \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.19.2:compile
[INFO] | | +- org.springframework.boot:spring-boot-starter-tomcat:jar:3.5.7:compile
[INFO] | | | +- org.apache.tomcat.embed:tomcat-embed-core:jar:10.1.48:compile
[INFO] | | | \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:10.1.48:compile
[INFO] | | +- org.springframework:spring-web:jar:6.2.12:compile
[INFO] | | \- org.springframework:spring-webmvc:jar:6.2.12:compile
[INFO] | +- org.springframework.boot:spring-boot-starter-jdbc:jar:3.5.7:compile
[INFO] | | +- com.zaxxer:HikariCP:jar:6.3.3:compile
[INFO] | | \- org.springframework:spring-jdbc:jar:6.2.12:compile
[INFO] | | \- org.springframework:spring-tx:jar:6.2.12:compile
[INFO] | \- org.yaml:snakeyaml:jar:2.4:compile
[INFO] +- com.sap.cds:cds-starter-cloudfoundry:jar:3.10.3:compile
[INFO] | +- com.sap.cds:cds-feature-hana:jar:3.10.3:compile
[INFO] | | \- com.sap.cloud.db.jdbc:ngdbc:jar:2.25.9:compile
[INFO] | +- com.sap.cds:cds-feature-mt:jar:3.10.3:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-databind:jar:2.19.2:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.19.2:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.19.2:compile
[INFO] | +- com.sap.cds:cds-feature-cloudfoundry:jar:3.10.3:compile
[INFO] | | \- com.sap.cloud.environment.servicebinding:java-sap-vcap-services:jar:0.21.0:compile
[INFO] | | \- org.json:json:jar:20250107:compile
[INFO] | +- com.sap.cds:cds-feature-identity:jar:3.10.3:compile
[INFO] | | +- com.sap.cloud.security:java-api:jar:3.6.5:compile
[INFO] | | \- com.sap.cloud.security:java-security:jar:3.6.5:compile
[INFO] | +- com.sap.cloud.security:resourceserver-security-spring-boot-starter:jar:3.6.5:compile
[INFO] | | +- com.sap.cloud.security:spring-security:jar:3.6.5:compile
[INFO] | | | \- io.projectreactor:reactor-core:jar:3.7.12:compile
[INFO] | | | \- org.reactivestreams:reactive-streams:jar:1.0.4:compile
[INFO] | | +- org.springframework.security:spring-security-oauth2-jose:jar:6.5.6:compile
[INFO] | | | \- org.springframework.security:spring-security-oauth2-core:jar:6.5.6:compile
[INFO] | | \- org.springframework.security:spring-security-oauth2-resource-server:jar:6.5.6:compile
[INFO] | \- org.springframework.boot:spring-boot-starter-security:jar:3.5.7:compile
[INFO] | +- org.springframework:spring-aop:jar:6.2.12:compile
[INFO] | \- org.springframework.security:spring-security-config:jar:6.5.6:compile
[INFO] +- com.sap.cds:cds-adapter-odata-v4:jar:3.10.3:compile
[INFO] | +- com.sap.cds:cds-adapter-api:jar:3.10.3:compile
[INFO] | +- com.sap.cds:cds-services-utils:jar:3.10.3:compile
[INFO] | | +- com.sap.cloud.sdk.cloudplatform:connectivity-oauth:jar:5.24.0:compile
[INFO] | | | \- com.sap.cloud.environment.servicebinding.api:java-consumption-api:jar:0.21.0:compile
[INFO] | | +- commons-io:commons-io:jar:2.20.0:compile
[INFO] | | +- io.opentelemetry:opentelemetry-api:jar:1.49.0:compile
[INFO] | | | \- io.opentelemetry:opentelemetry-context:jar:1.49.0:compile
[INFO] | | +- com.sap.cloud.security.xsuaa:token-client:jar:3.6.5:compile
[INFO] | | \- com.sap.cloud.security:env:jar:3.6.5:compile
[INFO] | | \- com.sap.cloud.environment.servicebinding:java-sap-service-operator:jar:0.21.0:compile
[INFO] | +- com.sap.cds:cds4j-core:jar:3.10.3:compile
[INFO] | | \- com.github.ben-manes.caffeine:caffeine:jar:3.2.2:compile
[INFO] | +- com.sap.cds.repackaged:odata-v4-lib:jar:3.10.3:compile
[INFO] | | +- commons-codec:commons-codec:jar:1.18.0:compile
[INFO] | | +- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.19.2:compile
[INFO] | | | \- org.codehaus.woodstox:stax2-api:jar:4.2.2:compile
[INFO] | | \- com.fasterxml:aalto-xml:jar:1.3.3:compile
[INFO] | +- org.slf4j:slf4j-api:jar:2.0.17:compile
[INFO] | \- com.google.guava:guava:jar:33.5.0-jre:compile
[INFO] | +- com.google.guava:failureaccess:jar:1.0.3:compile
[INFO] | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] | +- org.jspecify:jspecify:jar:1.0.0:compile
[INFO] | +- com.google.errorprone:error_prone_annotations:jar:2.43.0:compile
[INFO] | \- com.google.j2objc:j2objc-annotations:jar:3.1:compile
[INFO] +- com.sap.cds:cds-feature-remote-odata:jar:3.10.3:runtime
[INFO] | +- com.sap.cds:cds-integration-cloud-sdk:jar:3.10.3:compile
[INFO] | | +- com.sap.cloud.sdk.cloudplatform:tenant:jar:5.24.0:compile
[INFO] | | +- com.sap.cloud.sdk.cloudplatform:security:jar:5.24.0:compile
[INFO] | | \- com.sap.cloud.sdk.cloudplatform:resilience:jar:5.24.0:compile
[INFO] | | \- com.sap.cloud.sdk.cloudplatform:resilience4j:jar:5.24.0:compile
[INFO] | | +- io.github.resilience4j:resilience4j-core:jar:2.3.0:compile
[INFO] | | | \- org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:1.9.25:compile
[INFO] | | | +- org.jetbrains.kotlin:kotlin-stdlib:jar:1.9.25:compile
[INFO] | | | | \- org.jetbrains:annotations:jar:13.0:compile
[INFO] | | | \- org.jetbrains.kotlin:kotlin-stdlib-jdk7:jar:1.9.25:compile
[INFO] | | +- io.github.resilience4j:resilience4j-circuitbreaker:jar:2.3.0:compile
[INFO] | | +- io.github.resilience4j:resilience4j-bulkhead:jar:2.3.0:compile
[INFO] | | +- io.github.resilience4j:resilience4j-timelimiter:jar:2.3.0:compile
[INFO] | | +- io.github.resilience4j:resilience4j-retry:jar:2.3.0:compile
[INFO] | | +- io.github.resilience4j:resilience4j-ratelimiter:jar:2.3.0:compile
[INFO] | | \- javax.cache:cache-api:jar:1.1.1:compile
[INFO] | \- com.sap.cloud.sdk.cloudplatform:cloudplatform-connectivity:jar:5.24.0:compile
[INFO] | +- com.sap.cloud.sdk.cloudplatform:cloudplatform-core:jar:5.24.0:compile
[INFO] | +- com.sap.cloud.sdk.cloudplatform:resilience-api:jar:5.24.0:compile
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:2.0.17:runtime
[INFO] | +- com.mikesamuel:json-sanitizer:jar:1.2.3:compile
[INFO] | +- com.auth0:java-jwt:jar:4.5.0:compile
[INFO] | +- org.bouncycastle:bcprov-jdk18on:jar:1.82:compile
[INFO] | \- org.bouncycastle:bcpkix-jdk18on:jar:1.82:compile
[INFO] | \- org.bouncycastle:bcutil-jdk18on:jar:1.82:compile
[INFO] +- com.sap.cloud.sdk.cloudplatform:scp-cf:jar:5.24.0:compile
[INFO] | \- com.sap.cloud.sdk:sdk-core:jar:5.24.0:compile
[INFO] | +- com.sap.cloud.sdk.cloudplatform:connectivity-destination-service:jar:5.24.0:compile
[INFO] | +- com.sap.cloud.sdk.cloudplatform:servlet-jakarta:jar:5.24.0:compile
[INFO] | \- com.sap.cloud.sdk.cloudplatform:caching:jar:5.24.0:compile
[INFO] +- org.springframework.boot:spring-boot-starter-actuator:jar:3.5.7:compile
[INFO] | +- org.springframework.boot:spring-boot-starter:jar:3.5.7:compile
[INFO] | | +- org.springframework.boot:spring-boot-starter-logging:jar:3.5.7:compile
[INFO] | | | +- ch.qos.logback:logback-classic:jar:1.5.20:compile
[INFO] | | | | \- ch.qos.logback:logback-core:jar:1.5.20:compile
[INFO] | | | +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.24.3:compile
[INFO] | | | | \- org.apache.logging.log4j:log4j-api:jar:2.24.3:compile
[INFO] | | | \- org.slf4j:jul-to-slf4j:jar:2.0.17:compile
[INFO] | | \- jakarta.annotation:jakarta.annotation-api:jar:2.1.1:compile
[INFO] | +- org.springframework.boot:spring-boot-actuator-autoconfigure:jar:3.5.7:compile
[INFO] | | +- org.springframework.boot:spring-boot-actuator:jar:3.5.7:compile
[INFO] | | \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.19.2:compile
[INFO] | +- io.micrometer:micrometer-observation:jar:1.15.5:compile
[INFO] | | \- io.micrometer:micrometer-commons:jar:1.15.5:compile
[INFO] | \- io.micrometer:micrometer-jakarta9:jar:1.15.5:compile
[INFO] | \- io.micrometer:micrometer-core:jar:1.15.5:compile
[INFO] | +- org.hdrhistogram:HdrHistogram:jar:2.2.2:runtime
[INFO] | \- org.latencyutils:LatencyUtils:jar:2.0.3:runtime
[INFO] +- org.apache.olingo:olingo-odata2-client-core:jar:2.0.13:compile
[INFO] | +- junit:junit:jar:4.13.2:compile
[INFO] | | \- org.hamcrest:hamcrest-core:jar:3.0:compile
[INFO] | +- org.apache.olingo:olingo-odata2-client-api:jar:2.0.13:compile
[INFO] | | \- org.apache.olingo:olingo-odata2-api:jar:2.0.13:compile
[INFO] | +- org.apache.olingo:olingo-odata2-core:jar:2.0.13:compile
[INFO] | | \- javax.ws.rs:javax.ws.rs-api:jar:2.0-m10:compile
[INFO] | \- org.apache.httpcomponents:httpcore:jar:4.4.16:compile
[INFO] +- com.sap.cloud.sdk.datamodel:odata-client:jar:5.24.0:compile
[INFO] | +- com.sap.cloud.sdk.datamodel:fluent-result:jar:5.24.0:compile
[INFO] | +- com.sap.cloud.sdk.cloudplatform:connectivity-apache-httpclient4:jar:5.24.0:compile
[INFO] | +- io.vavr:vavr:jar:0.10.7:compile
[INFO] | | \- io.vavr:vavr-match:jar:0.10.7:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.14:compile
[INFO] | \- com.google.code.gson:gson:jar:2.13.2:compile
[INFO] +- com.sap.xxx.xxx:jar:1.0.29:compile
[INFO] | +- com.sap.cp.security:credential-store-client-java:jar:1.63.0:compile
[INFO] | | +- org.apache.httpcomponents:httpmime:jar:4.5.14:compile
[INFO] | | +- org.bitbucket.b_c:jose4j:jar:0.9.6:compile
[INFO] | | +- commons-fileupload:commons-fileupload:jar:1.6.0:compile
[INFO] | | \- org.tomitribe:tomitribe-http-signatures:jar:1.8:compile
[INFO] | +- org.apache.httpcomponents.client5:httpclient5:jar:5.5.1:compile
[INFO] | | +- org.apache.httpcomponents.core5:httpcore5:jar:5.3.6:compile
[INFO] | | \- org.apache.httpcomponents.core5:httpcore5-h2:jar:5.3.6:compile
[INFO] | +- org.projectlombok:lombok:jar:1.18.42:compile
[INFO] | +- org.apache.commons:commons-text:jar:1.14.0:compile
[INFO] | \- org.springframework.boot:spring-boot-starter-validation:jar:3.5.7:compile
[INFO] | +- org.apache.tomcat.embed:tomcat-embed-el:jar:10.1.48:compile
[INFO] | \- org.hibernate.validator:hibernate-validator:jar:8.0.3.Final:compile
[INFO] | +- jakarta.validation:jakarta.validation-api:jar:3.0.2:compile
[INFO] | +- org.jboss.logging:jboss-logging:jar:3.6.1.Final:compile
[INFO] | \- com.fasterxml:classmate:jar:1.7.1:compile
[INFO] +- com.sap.hcp.cf.logging:cf-java-logging-support-logback:jar:3.8.4:compile
[INFO] | \- com.sap.hcp.cf.logging:cf-java-logging-support-core:jar:3.8.4:compile
[INFO] | \- com.fasterxml.jackson.jr:jackson-jr-objects:jar:2.19.2:compile
[INFO] +- org.springframework.boot:spring-boot-devtools:jar:3.5.7:compile (optional)
[INFO] | +- org.springframework.boot:spring-boot:jar:3.5.7:compile
[INFO] | | \- org.springframework:spring-context:jar:6.2.12:compile
[INFO] | \- org.springframework.boot:spring-boot-autoconfigure:jar:3.5.7:compile
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:3.5.7:test
[INFO] | +- org.springframework.boot:spring-boot-test:jar:3.5.7:test
[INFO] | +- org.springframework.boot:spring-boot-test-autoconfigure:jar:3.5.7:test
[INFO] | +- com.jayway.jsonpath:json-path:jar:2.9.0:test
[INFO] | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:4.0.4:test
[INFO] | | \- jakarta.activation:jakarta.activation-api:jar:2.1.4:test
[INFO] | +- net.minidev:json-smart:jar:2.5.2:test
[INFO] | | \- net.minidev:accessors-smart:jar:2.5.2:test
[INFO] | | \- org.ow2.asm:asm:jar:9.7.1:test
[INFO] | +- org.assertj:assertj-core:jar:3.27.6:test
[INFO] | | \- net.bytebuddy:byte-buddy:jar:1.17.8:test
[INFO] | +- org.awaitility:awaitility:jar:4.2.2:test
[INFO] | +- org.hamcrest:hamcrest:jar:3.0:compile
[INFO] | +- org.junit.jupiter:junit-jupiter:jar:5.12.2:test
[INFO] | | +- org.junit.jupiter:junit-jupiter-api:jar:5.12.2:test
[INFO] | | | +- org.opentest4j:opentest4j:jar:1.3.0:test
[INFO] | | | +- org.junit.platform:junit-platform-commons:jar:1.12.2:test
[INFO] | | | \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] | | +- org.junit.jupiter:junit-jupiter-params:jar:5.12.2:test
[INFO] | | \- org.junit.jupiter:junit-jupiter-engine:jar:5.12.2:test
[INFO] | | \- org.junit.platform:junit-platform-engine:jar:1.12.2:test
[INFO] | +- org.mockito:mockito-core:jar:5.17.0:test
[INFO] | | \- net.bytebuddy:byte-buddy-agent:jar:1.17.8:test
[INFO] | +- org.mockito:mockito-junit-jupiter:jar:5.17.0:test
[INFO] | +- org.skyscreamer:jsonassert:jar:1.5.3:test
[INFO] | | \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
[INFO] | +- org.springframework:spring-core:jar:6.2.12:compile
[INFO] | | \- org.springframework:spring-jcl:jar:6.2.12:compile
[INFO] | +- org.springframework:spring-test:jar:6.2.12:test
[INFO] | \- org.xmlunit:xmlunit-core:jar:2.10.4:test
[INFO] +- org.springframework.security:spring-security-test:jar:6.5.6:test
[INFO] | +- org.springframework.security:spring-security-core:jar:6.5.6:compile
[INFO] | | +- org.springframework.security:spring-security-crypto:jar:6.5.6:compile
[INFO] | | +- org.springframework:spring-beans:jar:6.2.12:compile
[INFO] | | \- org.springframework:spring-expression:jar:6.2.12:compile
[INFO] | \- org.springframework.security:spring-security-web:jar:6.5.6:compile
[INFO] +- org.wiremock:wiremock-standalone:jar:3.12.1:test
[INFO] +- com.github.hazendaz.jmockit:jmockit:jar:1.54.0:test
[INFO] | +- org.objenesis:objenesis:jar:3.4:test
[INFO] | +- org.checkerframework:checker-qual:jar:3.51.1:test
[INFO] | \- com.github.spotbugs:spotbugs-annotations:jar:4.8.6:test
[INFO] +- com.h2database:h2:jar:2.3.232:runtime
[INFO] +- com.nimbusds:nimbus-jose-jwt:jar:10.0.2:compile
[INFO] \- org.apache.commons:commons-lang3:jar:3.18.0:compile
[INFO] ------------------------------------------------------------------------
Code Examples
Refer the answer for this stack question https://sap.stackenterprise.co/questions/40429/40530
Stack Trace
stacktrace:
com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Failed to get destination.
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.resilientCall(DestinationService.java:378)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.lambda$loadAndParseDestination$2(DestinationService.java:138)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationRetrievalStrategyResolver.lambda$prepareSupplier$6(DestinationRetrievalStrategyResolver.java:187)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationRetrieval.get(DestinationRetrieval.java:22)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.loadAndParseDestination(DestinationService.java:146)
at com.sap.cloud.sdk.cloudplatform.connectivity.GetOrComputeSingleDestinationCommand.lambda$prepareCommand$0(GetOrComputeSingleDestinationCommand.java:66)
at io.vavr.control.Try.of(Try.java:74)
at io.vavr.control.Try.ofSupplier(Try.java:91)
at com.sap.cloud.sdk.cloudplatform.connectivity.GetOrComputeSingleDestinationCommand.execute(GetOrComputeSingleDestinationCommand.java:153)
at io.vavr.control.Try.flatMapTry(Try.java:489)
at io.vavr.control.Try.flatMap(Try.java:471)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService$Cache.getOrComputeDestination(DestinationService.java:870)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.tryGetDestination(DestinationService.java:123)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationLoaderChain.tryGetDestination(DestinationLoaderChain.java:85)
at com.sap.xxx.xxx.util.ProxyDestinationLoader.tryGetDestination(ProxyDestinationLoader.java:72)
at com.sap.cds.integration.cloudsdk.destination.RemoteServiceHttpClientProvider.initHttpDestination(RemoteServiceHttpClientProvider.java:102)
at com.sap.cds.integration.cloudsdk.destination.RemoteServiceHttpClientProvider.getHttpClient(RemoteServiceHttpClientProvider.java:51)
at com.sap.cds.services.impl.odata.RemoteODataClient.processEvent(RemoteODataClient.java:65)
at com.sap.cds.services.impl.odata.RemoteODataHandler.read(RemoteODataHandler.java:87)
at com.sap.cds.services.impl.handlerregistry.HandlerRegistryTools$DescribedHandler.process(HandlerRegistryTools.java:174)
at com.sap.cds.services.impl.ServiceImpl.lambda$createOnHandlerChain$4(ServiceImpl.java:282)
at com.sap.cds.services.impl.ServiceImpl.dispatch(ServiceImpl.java:249)
at com.sap.cds.services.impl.ServiceImpl.lambda$dispatchInChangeSetContext$2(ServiceImpl.java:213)
at com.sap.cds.services.impl.runtime.ChangeSetContextRunnerImpl.lambda$run$1(ChangeSetContextRunnerImpl.java:62)
at com.sap.cds.services.impl.runtime.ChangeSetContextRunnerImpl.open(ChangeSetContextRunnerImpl.java:77)
at com.sap.cds.services.impl.runtime.ChangeSetContextRunnerImpl.run(ChangeSetContextRunnerImpl.java:55)
at com.sap.cds.services.impl.runtime.ChangeSetContextRunnerImpl.run(ChangeSetContextRunnerImpl.java:60)
at com.sap.cds.services.impl.ServiceImpl.dispatchInChangeSetContext(ServiceImpl.java:213)
at com.sap.cds.services.impl.ServiceImpl.emit(ServiceImpl.java:193)
at com.sap.cds.services.ServiceDelegator.emit(ServiceDelegator.java:33)
at com.sap.cds.services.utils.services.AbstractCqnService.run(AbstractCqnService.java:52)
at com.sap.cds.services.impl.cds.AbstractCdsDefinedService.lambda$run$0(AbstractCdsDefinedService.java:53)
at com.sap.cds.services.impl.cds.AbstractCdsDefinedService.runProjected(AbstractCdsDefinedService.java:95)
at com.sap.cds.services.impl.cds.AbstractCdsDefinedService.run(AbstractCdsDefinedService.java:53)
at com.sap.cds.services.utils.services.AbstractCqnService.run(AbstractCqnService.java:42)
at jdk.internal.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.sap.cds.services.impl.cds.TypedCqnServiceInvocationHandler.invoke(TypedCqnServiceInvocationHandler.java:76)
at jdk.proxy2/jdk.proxy2.$Proxy47.run(Unknown Source)
at com.sap.xxx.xxx.service.Service1.method1(Service1.java:222)
at com.sap.xxx.xxx.service.Service1.lambda$method2$0(Service1.java:169)
at com.sap.cds.services.impl.runtime.RequestContextRunnerImpl.lambda$run$3(RequestContextRunnerImpl.java:216)
at com.sap.cds.services.impl.runtime.RequestContextRunnerImpl.run(RequestContextRunnerImpl.java:286)
at com.sap.cds.services.impl.runtime.RequestContextRunnerImpl.run(RequestContextRunnerImpl.java:214)
at com.sap.xxx.xxx.service.Service1.method2(Service1.java:166)
at com.sap.xxx.xxx.service.Service1.method3(Service1.java:147)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:360)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:114)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.sap.xxx.xxx.util.MDCHandler.lambda$decorate$0(MDCHandler.java:24)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.sap.cloud.sdk.cloudplatform.resilience.ResilienceRuntimeException: CircuitBreaker 'class com.sap.cloud.sdk.cloudplatform.connectivity.DestinationServicesingleDestResilience' is OPEN and does not permit further calls. Triggered by com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Header provider 'OAuth2HeaderProvider' threw an exception: CircuitBreaker 'xxx.authentication.cert.eu12.hana.ondemand.com-sb-clonexxxxxxxxxxxxxxxxxxxxxxx!bxxxxxx|destination-xsappname!bx' is OPEN and does not permit further calls. Triggered by com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
at com.sap.cloud.sdk.cloudplatform.resilience4j.DefaultCircuitBreakerProvider.lambda$decorateCallable$2(DefaultCircuitBreakerProvider.java:94)
at io.vavr.control.Try.of(Try.java:74)
at io.vavr.control.Try.ofCallable(Try.java:104)
at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$4(Resilience4jDecorationStrategy.java:186)
at io.vavr.control.Try.of(Try.java:74)
at io.vavr.control.Try.ofCallable(Try.java:104)
at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateSupplier$1(Resilience4jDecorationStrategy.java:156)
at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorationStrategy.executeSupplier(ResilienceDecorationStrategy.java:99)
at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorator.executeSupplier(ResilienceDecorator.java:197)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.resilientCall(DestinationService.java:372)
... 59 more
Suppressed: io.github.resilience4j.circuitbreaker.CallNotPermittedException: CircuitBreaker 'class com.sap.cloud.sdk.cloudplatform.connectivity.DestinationServicesingleDestResilience' is HALF_OPEN and does not permit further calls
at io.github.resilience4j.circuitbreaker.CallNotPermittedException.createCallNotPermittedException(CallNotPermittedException.java:48)
at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine$HalfOpenState.acquirePermission(CircuitBreakerStateMachine.java:1098)
at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine.acquirePermission(CircuitBreakerStateMachine.java:184)
at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateCallable$4(CircuitBreaker.java:165)
at com.sap.cloud.sdk.cloudplatform.resilience4j.DefaultCircuitBreakerProvider.lambda$decorateCallable$2(DefaultCircuitBreakerProvider.java:82)
... 68 more
Caused by: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Header provider 'OAuth2HeaderProvider' threw an exception: CircuitBreaker 'xxx.authentication.cert.eu12.hana.ondemand.com-sb-clonexxxxxxxxxxxxxxxxxxxxxxx!bxxxxxx|destination-xsappname!bx' is OPEN and does not permit further calls. Triggered by com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:242)
at com.sap.cloud.sdk.cloudplatform.thread.DefaultThreadContextExecutorService.lambda$decorate$0(DefaultThreadContextExecutorService.java:68)
at com.sap.cds.integration.cloudsdk.decorator.CdsThreadContextDecorator.lambda$decorateCallable$0(CdsThreadContextDecorator.java:37)
at com.sap.cds.services.impl.runtime.RequestContextRunnerImpl.run(RequestContextRunnerImpl.java:286)
at com.sap.cds.integration.cloudsdk.decorator.CdsThreadContextDecorator.lambda$decorateCallable$1(CdsThreadContextDecorator.java:34)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
... 3 more
Caused by: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Header provider 'OAuth2HeaderProvider' threw an exception: CircuitBreaker 'xxx.authentication.cert.eu12.hana.ondemand.com-sb-clonexxxxxxxxxxxxxxxxxxxxxxx!bxxxxxx|destination-xsappname!bx' is OPEN and does not permit further calls. Triggered by com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
at com.sap.cloud.sdk.cloudplatform.connectivity.DefaultHttpDestination.getHeadersFromHeaderProviders(DefaultHttpDestination.java:213)
at com.sap.cloud.sdk.cloudplatform.connectivity.DefaultHttpDestination.getHeaders(DefaultHttpDestination.java:174)
at com.sap.cloud.sdk.cloudplatform.connectivity.HttpClientWrapper.wrapRequest(HttpClientWrapper.java:126)
at com.sap.cloud.sdk.cloudplatform.connectivity.HttpClientWrapper.execute(HttpClientWrapper.java:144)
at com.sap.cloud.sdk.cloudplatform.connectivity.HttpClientWrapper.execute(HttpClientWrapper.java:32)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationServiceAdapter.getConfigurationAsJson(DestinationServiceAdapter.java:133)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.retrieveDestination(DestinationService.java:153)
at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.lambda$loadAndParseDestination$1(DestinationService.java:138)
at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$2(Resilience4jDecorationStrategy.java:175)
at io.github.resilience4j.bulkhead.Bulkhead.lambda$decorateCallable$5(Bulkhead.java:173)
at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.call(ThreadContextExecutor.java:293)
at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:236)
... 8 more
Caused by: com.sap.cloud.sdk.cloudplatform.resilience.ResilienceRuntimeException: CircuitBreaker 'xxx.authentication.cert.eu12.hana.ondemand.com-sb-clonexxxxxxxxxxxxxxxxxxxxxxx!bxxxxxx|destination-xsappname!bx' is OPEN and does not permit further calls. Triggered by com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
at com.sap.cloud.sdk.cloudplatform.resilience4j.DefaultCircuitBreakerProvider.lambda$decorateCallable$2(DefaultCircuitBreakerProvider.java:94)
at io.vavr.control.Try.of(Try.java:74)
at io.vavr.control.Try.ofCallable(Try.java:104)
at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$4(Resilience4jDecorationStrategy.java:186)
at io.vavr.control.Try.of(Try.java:74)
at io.vavr.control.Try.ofCallable(Try.java:104)
at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateSupplier$1(Resilience4jDecorationStrategy.java:156)
at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorationStrategy.executeSupplier(ResilienceDecorationStrategy.java:99)
at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorator.executeSupplier(ResilienceDecorator.java:197)
at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.retrieveAccessToken(OAuth2Service.java:151)
at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2HeaderProvider.getHeaders(OAuth2HeaderProvider.java:30)
at com.sap.cloud.sdk.cloudplatform.connectivity.DefaultHttpDestination.getHeadersFromHeaderProviders(DefaultHttpDestination.java:207)
... 19 more
Suppressed: io.github.resilience4j.circuitbreaker.CallNotPermittedException: CircuitBreaker 'xxx.authentication.cert.eu12.hana.ondemand.com-sb-clonexxxxxxxxxxxxxxxxxxxxxxx!bxxxxxx|destination-xsappname!bx' is OPEN and does not permit further calls
at io.github.resilience4j.circuitbreaker.CallNotPermittedException.createCallNotPermittedException(CallNotPermittedException.java:48)
at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine$OpenState.acquirePermission(CircuitBreakerStateMachine.java:730)
at io.github.resilience4j.circuitbreaker.internal.CircuitBreakerStateMachine.acquirePermission(CircuitBreakerStateMachine.java:184)
at io.github.resilience4j.circuitbreaker.CircuitBreaker.lambda$decorateCallable$4(CircuitBreaker.java:165)
at com.sap.cloud.sdk.cloudplatform.resilience4j.DefaultCircuitBreakerProvider.lambda$decorateCallable$2(DefaultCircuitBreakerProvider.java:82)
... 30 more
Caused by: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:242)
... 8 more
Caused by: com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.buildException(OAuth2Service.java:222)
at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.lambda$executeClientCredentialsFlow$1(OAuth2Service.java:208)
at io.vavr.control.Try.getOrElseThrow(Try.java:747)
at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.executeClientCredentialsFlow(OAuth2Service.java:208)
at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.lambda$retrieveAccessToken$0(OAuth2Service.java:157)
at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$2(Resilience4jDecorationStrategy.java:175)
at io.github.resilience4j.bulkhead.Bulkhead.lambda$decorateCallable$5(Bulkhead.java:173)
at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.call(ThreadContextExecutor.java:293)
at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:236)
... 8 more
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
at com.sap.cloud.security.xsuaa.client.DefaultOAuth2TokenService.executeRequest(DefaultOAuth2TokenService.java:83)
at com.sap.cloud.security.xsuaa.client.DefaultOAuth2TokenService.requestAccessToken(DefaultOAuth2TokenService.java:63)
at com.sap.cloud.security.xsuaa.client.AbstractOAuth2TokenService.getAndCacheToken(AbstractOAuth2TokenService.java:289)
at com.sap.cloud.security.xsuaa.client.AbstractOAuth2TokenService.getOrRequestAccessToken(AbstractOAuth2TokenService.java:248)
at com.sap.cloud.security.xsuaa.client.AbstractOAuth2TokenService.getOAuth2TokenResponse(AbstractOAuth2TokenService.java:231)
at com.sap.cloud.security.xsuaa.client.AbstractOAuth2TokenService.retrieveAccessTokenViaClientCredentialsGrant(AbstractOAuth2TokenService.java:113)
at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.lambda$executeClientCredentialsFlow$1c851743$1(OAuth2Service.java:201)
at io.vavr.control.Try.of(Try.java:74)
at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.executeClientCredentialsFlow(OAuth2Service.java:199)
Log File
Log file
...Affected Development Phase
Production
Impact
Blocked
Timeline
BTP SaaS application with delivery milestone in the Mid of February 2026.