Upload failing randomly with com.microsoftopentechnologies.windowsazurestorage.exceptions.WAStorageException: Fail to upload individual files to blob error #56
Description
Jenkins and plugins versions report
Environment
Jenkins: 2.387.2
OS: Linux - 5.4.0-1105-azure
Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
analysis-model-api:11.1.0
ansicolor:1.0.2
ant:481.v7b_09e538fcca
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
authentication-tokens:1.53.v1c90fd9191a_b_
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-cloudformation:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-codebuild:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-ec2:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-ecr:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-ecs:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-efs:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-elasticbeanstalk:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-iam:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-kinesis:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-logs:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-minimal:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-sns:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-sqs:1.12.447-380.v65b_d0c044a_b_a_
aws-java-sdk-ssm:1.12.447-380.v65b_d0c044a_b_a_
azure-artifact-manager:109.vaf4232a14e2e
azure-container-agents:253.vd2f5cd5c5040
azure-credentials:254.v64da_8176c83a
azure-sdk:118.v43f74dd9ca_dc
azure-vm-agents:825.v470cb_9e7361a_
basic-branch-build-strategies:71.vc1421f89888e
beer:42.v776b_04d96de3
blueocean:1.27.3
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.3
blueocean-commons:1.27.3
blueocean-config:1.27.3
blueocean-core-js:1.27.3
blueocean-dashboard:1.27.3
blueocean-display-url:2.4.1
blueocean-events:1.27.3
blueocean-git-pipeline:1.27.3
blueocean-github-pipeline:1.27.3
blueocean-i18n:1.27.3
blueocean-jira:1.27.3
blueocean-jwt:1.27.3
blueocean-personalization:1.27.3
blueocean-pipeline-api-impl:1.27.3
blueocean-pipeline-editor:1.27.3
blueocean-pipeline-scm-api:1.27.3
blueocean-rest:1.27.3
blueocean-rest-impl:1.27.3
blueocean-web:1.27.3
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.2-2
bouncycastle-api:2.27
branch-api:2.1071.v1a_188a_562481
build-discarder:139.v05696a_7fe240
buildtriggerbadge:251.vdf6ef853f3f5
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:2.0.0
cloud-stats:267.v577e3742c282
cloudbees-bitbucket-branch-source:800.va_b_b_9a_a_5035c1
cloudbees-folder:6.815.v0dd5a_cb_40e0e
code-coverage-api:4.2.2
command-launcher:90.v669d7ccb_7c31
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-36.vc008c8fcda_7b_
conditional-buildstep:1.4.2
config-file-provider:3.11.1
configuration-as-code:1625.v27444588cc3d
copyartifact:686.v6fd37018d7c2
credentials:1224.vc23ca_a_9a_2cb_0
credentials-binding:604.vb_64480b_c56ca_
cvs:2.19.1
dark-theme:315.va_22e7d692ea_a
data-tables-api:1.13.3-3
disable-github-multibranch-status:1.2
display-url-api:2.3.7
docker-commons:419.v8e3cd84ef49c
docker-workflow:563.vd5d2e5c4007f
durable-task:504.vb10d1ae5ba2f
ec2:2.0.7
echarts-api:5.4.0-3
embeddable-build-status:369.vb_a_68a_575a_b_11
extended-read-permission:3.2
external-monitor-job:203.v683c09d993b_9
favorite:2.4.1
font-awesome-api:6.3.0-2
forensics-api:2.1.0
generic-tool:1.0
git:5.0.0
git-client:4.2.0
git-forensics:2.0.0
github:1.37.0
github-api:1.303-417.ve35d9dd78549
github-branch-source:1703.vd5a_2b_29c6cdc
github-checks:1.0.19
groovy:453.vcdb_a_c5c99890
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.31
http_request:1.16
instance-identity:142.v04572ca_5b_265
ionicons-api:45.vf54fca_5d2154
jackson2-api:2.14.2-319.v37853346a_229
jacoco:3.3.3
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:226.v71211feb_e7e9
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:63.v62d2fd4b_4793
jenkins-design-language:1.27.3
jersey2-api:2.39.1-1
jira:3.9
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.16-2
jobConfigHistory:1207.vd28a_54732f92
jquery:1.12.4-1
jquery3-api:3.6.4-1
jsch:0.1.55.61.va_e9ee26616e7
junit:1189.v1b_e593637fa_e
junit-attachments:131.vef2117770555
junit-realtime-test-reporter:102.v3f020960a_646
keyboard-shortcuts-plugin:1.4
kubernetes:3910.ve59cec5e33ea_
kubernetes-client-api:6.4.1-215.v2ed17097a_8e9
kubernetes-credentials:0.10.0
ldap:671.v2a_9192a_7419d
lockable-resources:1141.v7c5f8f31d2ee
mailer:448.v5b_97805e3767
mapdb-api:1.0.9-28.vf251ce40855d
matrix-auth:3.1.6
matrix-project:785.v06b_7f47b_c631
maven-plugin:3.21
metrics:4.2.13-420.vea_2f17932dd6
mina-sshd-api-common:2.9.2-62.v199162f0a_2f8
mina-sshd-api-core:2.9.2-62.v199162f0a_2f8
node-iterator-api:49.v58a_8b_35f8363
okhttp-api:4.10.0-132.v7a_7b_91cef39c
pam-auth:1.10
parallel-test-executor:383.vde60119c849a
parameterized-trigger:2.45
pipeline-build-step:488.v8993df156e8d
pipeline-github:2.8-147.3206e8179b1c
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-graph-view:183.v9e27732d970f
pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2125.vddb_a_44a_d605e
pipeline-model-definition:2.2125.vddb_a_44a_d605e
pipeline-model-extensions:2.2125.vddb_a_44a_d605e
pipeline-rest-api:2.32
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2125.vddb_a_44a_d605e
pipeline-stage-view:2.32
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.2.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
prism-api:1.29.0-4
pubsub-light:1.17
run-condition:1.5
scm-api:631.v9143df5b_e4a_a
script-security:1229.v4880b_b_e905a_6
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
sse-gateway:1.26
ssh-agent:327.v230ecd01f86f
ssh-credentials:305.v8f4381501156
ssh-slaves:2.877.v365f5eb_a_b_eec
sshd:3.275.v9e17c10f2571
structs:324.va_f5d6774f3a_d
subversion:2.17.1
support-core:1266.v6d096c154c90
theme-manager:1.6
timestamper:1.24
token-macro:321.vd7cc1f2a_52c8
translation:1.16
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
warnings-ng:10.1.0
windows-azure-storage:380.va3a027b784f4
workflow-aggregator:596.v8c21c963d92d
workflow-api:1208.v0cc7c6e0da_9e
workflow-basic-steps:1010.vf7a_b_98e847c1
workflow-cps:3653.v07ea_433c90b_4
workflow-durable-task-step:1244.vee71f675dee6
workflow-job:1289.vd1c337fd5354
workflow-multibranch:733.v109046189126
workflow-scm-step:408.v7d5b_135a_b_d49
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
What Operating System are you using (both controller, and any agents involved in the problem)?
- Controller: Docker LTS image 2.387.2 JDK11 (Debian) running on Ubuntu 18.04 Host
- Agents:
- Linux Ubuntu 22.04 base OS (https://github.com/jenkins-infra/packer-images/releases/tag/1.2.1) Docker image
- Kubernetes agents: either on AWS EKS or Digital Ocean managed cluster
Reproduction steps
- Configured the following Azure Storage Account in the controller (+ Verified successfully through the "Verify Account" button):
Click to view the JSON storage account definition
{
"sku": {
"name": "Standard_LRS",
"tier": "Standard"
},
"kind": "StorageV2",
"id": "/subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.Storage/storageAccounts/cijenkinsioartifacts",
"name": "cijenkinsioartifacts",
"type": "Microsoft.Storage/storageAccounts",
"location": "<redacted>",
"properties": {
"dnsEndpointType": "Standard",
"defaultToOAuthAuthentication": false,
"publicNetworkAccess": "Enabled",
"keyCreationTime": {
"key1": "<redacted>",
"key2": "<redacted>"
},
"allowCrossTenantReplication": false,
"privateEndpointConnections": [],
"minimumTlsVersion": "TLS1_2",
"allowBlobPublicAccess": true,
"allowSharedKeyAccess": true,
"networkAcls": {
"bypass": "AzureServices",
"virtualNetworkRules": [],
"ipRules": [],
"defaultAction": "Allow"
},
"supportsHttpsTrafficOnly": true,
"encryption": {
"requireInfrastructureEncryption": false,
"services": {
"file": {
"keyType": "Account",
"enabled": true,
"lastEnabledTime": "<redacted>"
},
"blob": {
"keyType": "Account",
"enabled": true,
"lastEnabledTime": "<redacted>"
}
},
"keySource": "Microsoft.Storage"
},
"accessTier": "Hot",
"provisioningState": "Succeeded",
"creationTime": "<redacted>",
"primaryEndpoints": {
"dfs": "https://cijenkinsioartifacts.dfs.core.windows.net/",
"web": "https://cijenkinsioartifacts.z20.web.core.windows.net/",
"blob": "https://cijenkinsioartifacts.blob.core.windows.net/",
"queue": "https://cijenkinsioartifacts.queue.core.windows.net/",
"table": "https://cijenkinsioartifacts.table.core.windows.net/",
"file": "https://cijenkinsioartifacts.file.core.windows.net/"
},
"primaryLocation": "<redacted>",
"statusOfPrimary": "available"
}
}- Configured the Artifact Management as below:
Expected Results
All of the archiveArtifacts and stash methods should work as expected
Actual Results
- Some jobs are failing with the following error stack (and no message on the controller logs):
io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 10000ms
at io.netty.handler.ssl.SslHandler$7.run(SslHandler.java:2113)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Also: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
Caused: reactor.core.Exceptions$ReactiveException
at reactor.core.Exceptions.propagate(Exceptions.java:392)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
at reactor.core.publisher.Mono.block(Mono.java:1707)
at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:191)
at com.azure.storage.blob.BlobClient.uploadFromFileWithResponse(BlobClient.java:473)
at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadThread.call(UploadService.java:426)
at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadThread.call(UploadService.java:387)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
Caused: java.util.concurrent.ExecutionException
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadOnSlave.invoke(UploadService.java:323)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ec2-3-143-84-83.us-east-2.compute.amazonaws.com/3.143.84.83:33447
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1000)
at hudson.FilePath.act(FilePath.java:1192)
at hudson.FilePath.act(FilePath.java:1181)
at com.microsoftopentechnologies.windowsazurestorage.service.UploadToBlobService.uploadIndividuals(UploadToBlobService.java:133)
at com.microsoftopentechnologies.windowsazurestorage.service.UploadService.execute(UploadService.java:527)
at com.microsoft.jenkins.artifactmanager.AzureArtifactManager.stash(AzureArtifactManager.java:427)
at org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:118)
at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:119)
at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:107)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused: java.io.IOException
at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadOnSlave.invoke(UploadService.java:326)
at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadOnSlave.invoke(UploadService.java:295)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3578)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
at java.base/java.lang.Thread.run(Unknown Source)
Caused: com.microsoftopentechnologies.windowsazurestorage.exceptions.WAStorageException: Fail to upload individual files to blob
at com.microsoftopentechnologies.windowsazurestorage.service.UploadToBlobService.uploadIndividuals(UploadToBlobService.java:138)
at com.microsoftopentechnologies.windowsazurestorage.service.UploadService.execute(UploadService.java:527)
at com.microsoft.jenkins.artifactmanager.AzureArtifactManager.stash(AzureArtifactManager.java:427)
Caused: java.io.IOException
at com.microsoft.jenkins.artifactmanager.AzureArtifactManager.stash(AzureArtifactManager.java:430)
at org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:118)
at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:119)
at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:107)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[2023-04-14T20:20:29.867Z] [GitHub Checks] GitHub check (name: Jenkins, status: completed) has been published.
[2023-04-14T20:20:29.881Z] Finished: FAILURE
- It looks like that any non Azure agent has the same failure. But
- I can't reproduce from a local (to my computer) controller in my own ISP
- The blob endpoint URL is reachable (tested with a
curl -v <url>) from the non Azure agent
Anything else?
-
Public Jenkins Infrastructure issue tracking the tentative on ci.jenkins.io: [ci.jenkins.io] Use Artifact Manager to store archived artifacts (and stashes) jenkins-infra/helpdesk#3496
-
Example of failing build: https://ci.jenkins.io/job/Packaging/job/packaging/job/master/208/
It could be a misconfiguration of course, but I can't find any way to diagnose