Skip to content

User/kylemoore/c2c stage enumerate from root#3393

Open
kylemoore-microsoft wants to merge 12 commits intomover/c2c-stagefrom
user/kylemoore/c2c-stage-enumerateFromRoot
Open

User/kylemoore/c2c stage enumerate from root#3393
kylemoore-microsoft wants to merge 12 commits intomover/c2c-stagefrom
user/kylemoore/c2c-stage-enumerateFromRoot

Conversation

@kylemoore-microsoft
Copy link
Copy Markdown
Collaborator

@kylemoore-microsoft kylemoore-microsoft commented Mar 2, 2026

Description

This PR fixes a bug in syncOrchestrator where nameless virtual (ex: ///b.txt) directories in azure blob storage and S3 were resulting in infinite loops.

When a nameless dir reached processor(), buildChildDir() would incorrectly return the same path as the parent. It would get enqueued again and there would be an infinite loop

The fix is to update syncOrchestrator such that st.dir always has a trailing '/', and when a nameless dir is passed to buildChildPath() another trailing slash is appended. For example, imagine there is a nameless dir at the root. buildChildDir("/", "") is called and it returns "//".

An important change was made to syncOrchestrator to support this behavior. I modified the traverser to always construct paths which have a leading slash and a trailing slash in case of directory. This was done because it made it really easy to handle nameless dirs. If you ever encounter a nameless dir, all you have to do is append another slash to the path. For example at root the path is "/", so you just append another slash for "//" if there is a nameless dir. Similarly if the current path is "/dir1/" you just append a slash for "/dir1//".

Type of Change

  • Bug fix
  • New feature
  • Documentation update required
  • Code quality improvement
  • Other (describe):

How Has This Been Tested?

Test Results Summary

Test location: c2ctest/sync_orchestrator_test.go
Command: go test -tags "netgo,smslidingwindow" -timeout=1h -v -run "TestSync_C2C_" ./c2ctest
Build tag: smslidingwindow (exercises the syncOrchestrator code path)

Results: 14/20 tests passed, 6 failed

Each test case is run for BlobBlob S3Blob BlobFSBlobFS BlobFSBlob BlobBlobFS

Test Result
FreshSync (5 subtests) ✅ PASS
IncrementalSync (5 subtests) ✅ PASS
MirrorFileDelete ❌ FAIL — BlobToBlobFS
MirrorDirDelete ❌ FAIL — BlobToBlobFS
NamelessDirFreshSync (2 subtests) ✅ PASS
NamelessDirIncrementalSync (2 subtests) ✅ PASS
NamelessDirMirrorDelete (2 subtests) ✅ PASS
DeepNamelessDirs (1 subtest) ✅ PASS
BlobNamelessToBlobFS (1 subtest) ✅ PASS
BlobFSToBlob (1 subtest) ✅ PASS
BlobFSToBlobNamelessMerge (1 subtest) ✅ PASS
BlobFSToBlobNamelessMirrorDelete ❌ FAIL — BlobFSToBlob
BlobNamelessToBlobFSMirrorDelete ❌ FAIL — BlobToBlobFS
RootOnlyFiles (1 subtest) ✅ PASS
PrefixCollision (1 subtest) ✅ PASS
MixedNamedNamelessDirs (1 subtest) ✅ PASS
NamelessDirMirrorDeleteOldSource ❌ FAIL — S3ToBlob, BlobToBlob
BlobFSToBlobNamelessMirrorDeleteOldSource ❌ FAIL — BlobFSToBlob
MirrorFileDeleteOldSource ❌ FAIL — BlobToBlobFS
SubpathFreshSync (5 subtests) ✅ PASS
SubpathIncrementalSync (5 subtests) ✅ PASS
SubpathMirrorDelete ❌ FAIL — BlobToBlobFS
SubpathDeepSync (5 subtests) ✅ PASS
SubpathPrefixCollision (5 subtests) ✅ PASS
SubpathUnicodeSync ❌ FAIL — S3ToBlob (panic)
CrossSubpathSync (5 subtests) ✅ PASS
CrossSubpathMirrorDelete (5 subtests) ✅ PASS

Failure Categories (all pre-existing, not regressions)

  1. BlobToBlobFS deletions broken — DFS destination doesn't properly delete files/dirs with --delete-destination=true
  2. Nameless dir mirror deletepath.Join normalizes nameless virtual dir paths (a///ba/b), causing files
    under single-char dirs to be incorrectly deleted
  3. S3 Unicode subpath panicURLWithPlusDecodedInPath panics for S3 URLs with CJK characters in subpath

All failures are identical with and without the smslidingwindow build tag, confirming zero regressions from the
syncOrchestrator.

@kylemoore-microsoft kylemoore-microsoft self-assigned this Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants