AmazonCryptoException: Failed to decrypt: mac check in GCM failed when downloading exactly 1GB file

[ajdali]

Describe the bug

Description
When downloading a file that is exactly 1GB (1000MB) using the Amazon.Extensions.S3.Encryption library, the decryption process fails with a "mac check in GCM failed" error. This issue appears to be size-specific, as files of other sizes download correctly.
Environment

.NET version:
net6.0-macos
Operating System: macOS (based on file paths), reproducible on Windows as well

Error Details
Copy[17:22:17 ERR] Failed to process restore item. RetryHelper.Value is empty. Failed file bcd074670c64//Users/ajdali/Documents/TestIssue/1000mb.txt
Amazon.Extensions.S3.Encryption.AmazonCryptoException: Failed to decrypt: mac check in GCM failed
---> Org.BouncyCastle.Crypto.InvalidCipherTextException: mac check in GCM failed
at Org.BouncyCastle.Crypto.Modes.GcmBlockCipher.DoFinal(Byte[] output, Int32 outOff) in //crypto/src/crypto/modes/GCMBlockCipher.cs:line 488
at Org.BouncyCastle.Crypto.BufferedAeadBlockCipher.DoFinal(Byte[] output, Int32 outOff) in //crypto/src/crypto/BufferedAeadBlockCipher.cs:line 235
at Org.BouncyCastle.Crypto.BufferedAeadBlockCipher.DoFinal(Byte[] input, Int32 inOff, Int32 inLen) in //crypto/src/crypto/BufferedAeadBlockCipher.cs:line 205
at Org.BouncyCastle.Crypto.IO.CipherStream.ReadAndProcessBlock() in //crypto/src/crypto/io/CipherStream.cs:line 124
at Org.BouncyCastle.Crypto.IO.CipherStream.FillInBuf() in //crypto/src/crypto/io/CipherStream.cs:line 92
at Org.BouncyCastle.Crypto.IO.CipherStream.Read(Byte[] buffer, Int32 offset, Int32 count) in //crypto/src/crypto/io/CipherStream.cs:line 72
at Amazon.Extensions.S3.Encryption.Util.AesGcmDecryptStream.Read(Byte[] buffer, Int32 offset, Int32 count)

The issue only occurs with files that are exactly 1GB (1000MB), but could occur with other files that are a multiple of this.
Files of other sizes (both smaller and larger) download and decrypt correctly
the error occurs in the BouncyCastle/AWS encryption code
This appears to be a boundary condition issue specific to the AES-GCM implementation when handling exactly 1GB of data

While I test the downloading, I found the file are not able to be downloaded are related to multi-part upload.
Impact: Files with any sizes of X times of break point (100_000_000 bytes now) (X > 1)
Those files are uploaded without encryption tag added, it might be library issue or our issue.

Workarounds Attempted

Retrying the download does not resolve the issue
The error is consistently reproducible with 1GB files

Any assistance in resolving this issue would be greatly appreciated, as it's blocking our ability to reliably handle files of this specific size.

Expected Behavior

The 1GB file should download and decrypt successfully, just like files of other sizes.

Current Behavior

the decryption process fails with a "mac check in GCM failed" error. This issue appears to be size-specific, as files of other sizes download correctly.

Error Details
Copy[17:22:17 ERR] Failed to process restore item. RetryHelper.Value is empty. Failed file bcd074670c64//Users/ajdali/Documents/TestIssue/1000mb.txt
Amazon.Extensions.S3.Encryption.AmazonCryptoException: Failed to decrypt: mac check in GCM failed
---> Org.BouncyCastle.Crypto.InvalidCipherTextException: mac check in GCM failed
at Org.BouncyCastle.Crypto.Modes.GcmBlockCipher.DoFinal(Byte[] output, Int32 outOff) in //crypto/src/crypto/modes/GCMBlockCipher.cs:line 488
at Org.BouncyCastle.Crypto.BufferedAeadBlockCipher.DoFinal(Byte[] output, Int32 outOff) in //crypto/src/crypto/BufferedAeadBlockCipher.cs:line 235
at Org.BouncyCastle.Crypto.BufferedAeadBlockCipher.DoFinal(Byte[] input, Int32 inOff, Int32 inLen) in //crypto/src/crypto/BufferedAeadBlockCipher.cs:line 205
at Org.BouncyCastle.Crypto.IO.CipherStream.ReadAndProcessBlock() in //crypto/src/crypto/io/CipherStream.cs:line 124
at Org.BouncyCastle.Crypto.IO.CipherStream.FillInBuf() in //crypto/src/crypto/io/CipherStream.cs:line 92
at Org.BouncyCastle.Crypto.IO.CipherStream.Read(Byte[] buffer, Int32 offset, Int32 count) in //crypto/src/crypto/io/CipherStream.cs:line 72
at Amazon.Extensions.S3.Encryption.Util.AesGcmDecryptStream.Read(Byte[] buffer, Int32 offset, Int32 count)

Reproduction Steps

Steps to Reproduce

Upload a file that is exactly 1GB (1000MB) to S3 with client-side encryption enabled
Attempt to download and decrypt the file using Amazon.Extensions.S3.Encryption
Observe the "mac check in GCM failed" error

AWS .NET SDK and/or Package version used

AWS SDKs for .NET used:
PackageReference Include="AWSSDK.Core" Version="3.7.302"
PackageReference Include="AWSSDK.S3" Version="3.7.305.17"
PackageReference Include="AWSSDK.SecurityToken" Version="3.7.103.14"
PackageReference Include="Amazon.Extensions.S3.Encryption" Version="2.1.0"

Targeted .NET Platform

net6.0-macos

Operating System and version

macOS 15.4

[ashishdhingra]

Not getting the same exception as reported in this issue, but the issue is reproducible using the below steps (on Mac):

• Create a 1GB using command mkfile -n 1g temp_1GB_file:

• Execute the code below using dotnet run command (notice it also configures verbose logging):
  .csproj

  <Project Sdk="Microsoft.NET.Sdk">
  
    <PropertyGroup>
      <OutputType>Exe</OutputType>
      <TargetFramework>net9.0</TargetFramework>
      <ImplicitUsings>enable</ImplicitUsings>
      <Nullable>enable</Nullable>
    </PropertyGroup>
  
    <ItemGroup>
      <PackageReference Include="Amazon.Extensions.S3.Encryption" Version="2.2.1" />
    </ItemGroup>
  
  </Project>

  Program.cs

  using Amazon;
  using Amazon.Extensions.S3.Encryption;
  using Amazon.Extensions.S3.Encryption.Primitives;
  using Amazon.S3;
  using Amazon.S3.Transfer;
  using System.Security.Cryptography;
  
  Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
  Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.Console;
  Amazon.AWSConfigs.LoggingConfig.LogMetrics = true;
  Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());
  
  string bucketName = "<<bucket-name>>";
  string filePath = "/<<path-to-file>>/temp_1GB_file"; // temp_900MB_file
  string keyName = "testkey-transfer-" + (new Random()).Next();
  
  var asymmetricAlgorithm = RSA.Create();
  var encryptionMaterial = new EncryptionMaterialsV2(asymmetricAlgorithm, AsymmetricAlgorithmType.RsaOaepSha1);
  var configuration = new AmazonS3CryptoConfigurationV2(SecurityProfile.V2);
  configuration.RegionEndpoint = RegionEndpoint.USEast2;
  var encryptionClient = new AmazonS3EncryptionClientV2(configuration, encryptionMaterial);
  
  UploadFileAsync(encryptionClient, bucketName, keyName, filePath).Wait();
  DownloadFileAsync(encryptionClient, bucketName, keyName, $"/tmp/{keyName}").Wait();
  
  static async Task UploadFileAsync(AmazonS3EncryptionClientV2 encryptionClient, string bucketName, string keyName, string filePath)
  {
      try
      {
          var transferUtility = new TransferUtility(encryptionClient, new TransferUtilityConfig());
  
          var transferUtilityUploadRequest = new TransferUtilityUploadRequest
          {
              BucketName = bucketName,
              FilePath = filePath,
              Key = keyName
          };
  
          transferUtilityUploadRequest.UploadProgressEvent += (sender, e) =>
          {
              Console.WriteLine("{0}/{1}", e.TransferredBytes, e.TotalBytes);
          };
  
          await transferUtility.UploadAsync(transferUtilityUploadRequest);
          Console.WriteLine("Upload completed");
      }
      catch (AmazonS3Exception e)
      {
          Console.WriteLine("Error encountered on server. Message:'{0}' when writing an object", e.Message);
      }
      catch (Exception e)
      {
          Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message);
      }
  }
  
  static async Task DownloadFileAsync(AmazonS3EncryptionClientV2 encryptionClient, string bucketName, string keyName, string downloadPath)
  {
      try
      {
          var transferUtility = new TransferUtility(encryptionClient, new TransferUtilityConfig());
  
          var transferUtilityDownloadRequestRequest = new TransferUtilityDownloadRequest
          {
              BucketName = bucketName,
              FilePath = downloadPath,
              Key = keyName
          };
  
          await transferUtility.DownloadAsync(transferUtilityDownloadRequestRequest);
          Console.WriteLine($"File downloaded at: {downloadPath}");
      }
      catch (AmazonS3Exception e)
      {
          Console.WriteLine("Error encountered while downloading object. Message:'{0}', Stack Trace: {1}", e.Message, e.StackTrace);
      }
      catch (Exception e)
      {
          Console.WriteLine("Error encountered while downloading object. Message:'{0}', Stack Trace: {1}", e.Message, e.StackTrace);
      }
  }

It gives below exception:

TransferUtility 37|2025-03-24T15:13:15.047Z|ERROR|Encountered a non retryable InvalidCipherTextException, rethrowing exception. --> Org.BouncyCastle.Crypto.InvalidCipherTextException: data wrong
   at Org.BouncyCastle.Crypto.Encodings.OaepEncoding.DecodeBlock(Byte[] inBytes, Int32 inOff, Int32 inLen) in /_/crypto/src/crypto/encodings/OaepEncoding.cs:line 282
   at Org.BouncyCastle.Crypto.Encodings.OaepEncoding.ProcessBlock(Byte[] inBytes, Int32 inOff, Int32 inLen) in /_/crypto/src/crypto/encodings/OaepEncoding.cs:line 132
   at Org.BouncyCastle.Crypto.BufferedAsymmetricBlockCipher.DoFinal() in /_/crypto/src/crypto/BufferedAsymmetricBlockCipher.cs:line 124
   at Org.BouncyCastle.Crypto.BufferedAsymmetricBlockCipher.DoFinal(Byte[] input, Int32 inOff, Int32 length) in /_/crypto/src/crypto/BufferedAsymmetricBlockCipher.cs:line 139
   at Org.BouncyCastle.Crypto.BufferedCipherBase.DoFinal(Byte[] input) in /_/crypto/src/crypto/BufferedCipherBase.cs:line 72
   at Amazon.Extensions.S3.Encryption.EncryptionUtils.DecryptEnvelopeKeyUsingAsymmetricKeyPairV2(AsymmetricAlgorithm asymmetricAlgorithm, Byte[] encryptedEnvelopeKey)
   at Amazon.Extensions.S3.Encryption.EncryptionUtils.DecryptNonKmsEnvelopeKeyV2(Byte[] encryptedEnvelopeKey, EncryptionMaterialsBase materials)
   at Amazon.Extensions.S3.Encryption.EncryptionUtils.BuildInstructionsFromObjectMetadata(GetObjectResponse response, EncryptionMaterialsBase materials, Byte[] decryptedEnvelopeKeyKMS)
   at Amazon.Extensions.S3.Encryption.Internal.SetupDecryptionHandler.DecryptObjectUsingMetadata(GetObjectResponse getObjectResponse, Byte[] decryptedEnvelopeKeyKMS)
   at Amazon.Extensions.S3.Encryption.Internal.SetupDecryptionHandler.DecryptObjectAsync(Byte[] decryptedEnvelopeKeyKMS, GetObjectResponse getObjectResponse)
   at Amazon.Extensions.S3.Encryption.Internal.SetupDecryptionHandler.PostInvokeAsync(IExecutionContext executionContext)
   at Amazon.Extensions.S3.Encryption.Internal.SetupDecryptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Extensions.S3.Encryption.Internal.SetupEncryptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Extensions.S3.Encryption.Internal.SetupEncryptionHandlerV2.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Transfer.Internal.DownloadCommand.ExecuteAsync(CancellationToken cancellationToken)

It works with 900MB file though.

[ashishdhingra]

Need to reproduce it with latest preview version of this library that targets .NET V4 (which is in preview). We updated the project to use BouncyCastle.Cryptography and that might have fixed this issue in current preview version.

[ashishdhingra]

@ajdali Good afternoon. Based on my testing, this appears to be working when using Amazon.Extensions.S3.Encryption version 3.0.0-preview.4. Kindly note that this preview package depends on AWS SDK for .NET V4 (in preview currently). Please test it at your end, may be using copy of your current project.

Just to give you context,

• current GA version Amazon.Extensions.S3.Encryption version 2.2.1 had dependency on Portable.BouncyCastle version 1.8.10. Due to backwards compatibility reasons, we could not bump or remove this dependency.
• In new preview version, we have updated dependency on BouncyCastle.Cryptography version 2.4.0.

Thanks,
Ashish