Fix folder creation tracker panic when copying large folders with --overwrite=false/ifsourcenewer

[Copilot]

Description

• Feature / Bug Fix: Race condition in jpptFolderTracker causes panic when copying large folder trees with --overwrite=false or --overwrite=ifsourcenewer.

  The folder properties transfer sets SkippedEntityAlreadyExists on the plan entry via jptm.SetStatus() outside the tracker mutex. A concurrent CreateFolder call from a child file transfer (parent dir creation) then reads this status, which wasn't recognized as terminal — falling through to debugCheckState which panics on the mismatch.

  Fix: use ts.StatusLocked() instead of enumerating individual terminal statuses in both CreateFolder (early-return) and debugCheckState (legitimate-mismatch check). This covers SkippedEntityAlreadyExists and is forward-compatible with any future terminal status values.

• Related Links:

  • Issues

Type of Change

• Bug fix
• New feature
• Documentation update required
• Code quality improvement
• Other (describe):

How Has This Been Tested?

Added TestFolderCreationTracker_skippedEntityDoesNotPanic — simulates the exact race: registers a folder, creates it (gets FolderExisted), externally mutates the plan status to SkippedEntityAlreadyExists, then verifies a subsequent CreateFolder returns nil without panic or re-creation. Existing tracker tests pass unchanged.

Thank you for your contribution to AzCopy!

Original prompt

---

This section details on the original issue you should resolve

<issue_title>AzCopy v10.32.0 error when copying large folders</issue_title>
<issue_description>### Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

10.32.0

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

.\azcopy.exe copy --recursive --overwrite=ifsourcenewer ".\dotnet-sdk-8.0.124-win-x64\" "https://storageaccount.file.core.windows.net/Test/?sastoken"

What problem was encountered?

Terminating error "panic: internal folder state didn't match plan state: (internal: 1) (plan: SkippedEntityAlreadyExists)"

Stack trace:

INFO: Trying 4 concurrent connections (initial starting point)
3.3 %, 156 Done, 0 Failed, 5031 Pending, 589 Skipped, 5776 Total, 2-sec Throughput (Mb/s): 62.4179panic: internal folder state didn't match plan state: (internal: 1) (plan: SkippedEntityAlreadyExists)

goroutine 124 [running]:
github.com/Azure/azure-storage-azcopy/v10/ste.(*jpptFolderTracker).debugCheckState(0x22cc540?, {0xc00054b6d8?, 0x22cc540?})
        D:/a/_work/1/s/ste/folderCreationTracker.go:207 +0xee
github.com/Azure/azure-storage-azcopy/v10/ste.(*jpptFolderTracker).CreateFolder(0xc000640c18, {0xc002681ec0, 0x4d}, 0xc004379f20)
        D:/a/_work/1/s/ste/folderCreationTracker.go:175 +0x2ce
github.com/Azure/azure-storage-azcopy/v10/ste.AzureFileParentDirCreator.CreateDirToRoot({}, {0x22ed3a0, 0xc000926fc0}, 0xc0008b0048, 0xc0008b0060, {0x22eb030, 0xc000640c18})
        D:/a/_work/1/s/ste/sender-azureFile.go:464 +0x383
github.com/Azure/azure-storage-azcopy/v10/ste.(*azureFileSenderBase).EnsureFolderExists(0xc0026fa120)
        D:/a/_work/1/s/ste/sender-azureFile.go:321 +0x6d
github.com/Azure/azure-storage-azcopy/v10/ste.anyToRemote_folder({0x2307528, 0xc000919880}, 0xc00094b080, {0x2126f658180, 0xc0004d2fc0}, 0x2126668, 0x21266b0)
        D:/a/_work/1/s/ste/xfer-anyToRemote-folder.go:69 +0x2be
github.com/Azure/azure-storage-azcopy/v10/ste.anyToRemote({0x2307528, 0xc000919880}, {0x2126f658180, 0xc0004d2fc0}, 0x2126668, 0x21266b0)
        D:/a/_work/1/s/ste/xfer-anyToRemote-file.go:198 +0x357
github.com/Azure/azure-storage-azcopy/v10/ste.computeJobXfer.parameterizeSend.func5({0x2307528?, 0xc000919880?}, {0x2126f658180?, 0xc0004d2fc0?})
        D:/a/_work/1/s/ste/xfer.go:76 +0x34
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobPartMgr).StartJobXfer(0x230c4c0?, {0x2307528?, 0xc000919880?})
        D:/a/_work/1/s/ste/mgr-JobPartMgr.go:422 +0x3b
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobPartTransferMgr).StartJobXfer(0xc000919880?)
        D:/a/_work/1/s/ste/mgr-JobPartTransferMgr.go:250 +0x2c
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).transferProcessor.func1({0x2307528, 0xc000919880})
        D:/a/_work/1/s/ste/mgr-JobMgr.go:1043 +0x19c
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).transferProcessor(0xc0000c46c8, 0xa52154daaaaaa521?)
        D:/a/_work/1/s/ste/mgr-JobMgr.go:1055 +0xac
created by github.com/Azure/azure-storage-azcopy/v10/ste.NewJobMgr in goroutine 1
        D:/a/_work/1/s/ste/mgr-JobMgr.go:187 +0xb1d

How can we reproduce the problem in the simplest way?

Run the copy command to upload a folder from the local machine to a fileshare in an Azure storage account. The folder should have several layers of subfolders and files, like the dotnet SDK binaries in the example - the issue is intermittent but seems to happen more often with more files and subfolders. Run the command again after the initial upload using the --overwrite=ifsourcenewer or --overwrite=false flag to skip existing items, it may need to be run a few times to see the error

Have you found a mitigation/solution?

No</issue_description>

Comments on the Issue (you are @copilot in this section)

@gapra-msft @adreed-msft could you take a look?

• Fixes AzCopy v10.32.0 error when copying large folders #3374

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.