TransferUtility fails with request signature we calculated does not match signature provided for certain files

[steve-wark]

I'm in the process of copying a large AWS S3 bucket to a Wasabi S3 bucket using the .net SDK and the TransferUitilty. There are approximately 6 million files and most have copied acrosss with no issue. However, certain files will not copy and conminually throw the "request signature we calculated does not match signature provided" error. Out of the 6 million this error has occurred on less than 1,000 files and when I re-run it fails on these same files continually. For example one of the files key is shown berlow:-

1686188140-fe37915db7b246d6b9c747fa96082dfc/FW_ WO for Pt Adelaide DC Kitchen and Scrub floors (1).zip

I've scanned the keys for non-ascii or illegal characters and while the key is lonmg, other longer keys have worked with no issue.

Does anyone have any advice on how I might be able to debug and see what the issue is?

[normj]

@steve-wark Apologize but we don't have the bandwidth to debug S3 emulators like it looks like Wasabi is. Using the latest version of the SDK I had no problems download an object with that S3 object key using either the service clients GetObjectAsync operation or the TransferUtility. If your code for getting the object for S3 is meaningfully different let me know and I'll try that.

var client = new AmazonS3Client(RegionEndpoint.USWest2);

var bucketName = "<my-bucket>";
var key = "1686188140-fe37915db7b246d6b9c747fa96082dfc/FW_ WO for Pt Adelaide DC Kitchen and Scrub floors (1).zip";

using var getResponse = await client.GetObjectAsync(bucketName, key);
Console.WriteLine(getResponse.HttpStatusCode);

var transfer = new TransferUtility(client);
await transfer.DownloadAsync(@"c:\temp\foo.zip", bucketName, key);

[steve-wark]

Thanks normj, currently my transfers are wrapped up in an overall sync process ie scan AWS / scan Wasabi, work out differences and copy those.

I'll follow your example and try a simple transfer on 2 or 3 of the failures to try and isolate the issue.

[steve-wark]

Since data is coming from AWS S3 bucket and going to Wasabi S3 bucket I never considered the overall length as a issue since it was S3 buck to S3 bucket, but I just read there is a 63 character limit.
Just a thought, the AWS S3 bucket has files starting from 2014 and the keys are significantly longer than 63 characters. When did the 63 character limit come in?

[normj]

S3 has a 63 character limit on bucket names. The object key has a much longer limit, not sure what it is off the top of my head. S3 didn't always have the 63 limit but it has been a decade or so since it was put in place. I don't know the full history but I know it had to deal with getting the bucket name being dns compatible as there is a preference to use host style vs path style addressing of the bucket.