Add --follow-symlinks option to 'az storage blob upload-batch'

[anders-nexthop]

Related command

az storage blob upload-batch

Is your feature request related to a problem? Please describe.

The command skips over symlinks to directories. If I'm trying to copy a large directory structure to the cloud, I end up having to write my own directory handling structure to detect and traverse symlinked directories. It seems like a bulk upload command should be able to handle this.

Describe the solution you'd like

The command should detect if a symlink is to a directory and if so, copy the underlying directory. I don't need the original linked structure to be retained, and if a file or directory is referenced more than once it can just be copied to each path (so creating a new directory with the same name as the symlink).

original structure:

dir1
---dir2
------dir3
---dir4 (link to dir3)
---dir5 (link to dir4)

stored structure:

dir1
---dir2
------dir3
---dir4
------dir3
---dir5
------dir4
---------dir3

My use case only requires symlinks to directories to be handled, but something similar could be done with symlinked files, where the content of the linked file is stored under the name of the symlink.

Describe alternatives you've considered

I have implemented my own symlink handler to work around this. I call the command on a directory to upload everything in the base directory, then scan for symlinks which are directories and traverse them one by one. In each directory I do the same steps recursively (call the command again, look for symlink dirs again, etc).

Additional context

A related change was added in Azure/azure-storage-azcopy#1303 for the azcopy command, with the argument --follow-symlinks.

AWS copy command implements this feature (check out --follow-symlinks arg here). I am trying to switch an automated build system from AWS to Azure and was hoping az storage blob upload-batch would be a drop-in replacement for aws s3 copy --recursive (I do a bulk copy of built artifacts to cloud storage after the build completes).

[yonzhan]

Thank you for opening this issue, we will look into it.

[calvinhzy]

Can you give az storage copy a try which allows --follow-symlinks #13589

[anders-nexthop]

I was able to get this to work using az storage copy --follow-symlinks --recursive as suggested by @calvinhzy. Thanks!