uploading large file leaves too much tmp files

[antoinetran]

Describe the bug

When uploading a large file (170GiB), using nextcloud and S3 backend, with MultiPart enabled, the aws code uses temp files to uploads. The issue is that it takes the same amount of files twice in disk, once to received the whole file, and once splitted as multipart split. This results in a requirement of 170GiB * 2 as temporary disk in my kubernetes cluster.

Isn't it possible to read and send as-you-go, in streaming mode, and cleaning the split part after having sent the multipart successfully, instead of waiting for a fclose of the input stream?

https://github.com/aws/aws-sdk-php/blob/3.360.1/src/S3/StreamWrapper.php#L32 says

 * Because Amazon S3 requires a Content-Length header, write only streams will
 * maintain a 'php://temp' stream to buffer data written to the stream until
 * the stream is flushed (usually by closing the stream with fclose).

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

1/ The input file is read in a buffer in temp dir and after each read, the read data is discarded
2/ In a multipart upload, as soon as one split part is sent, the temporary file is deleted to free space.

Current Behavior

1/ The input file of 170GiB is stored in /tmp/phpXXXX.
2/ In a multipart upload, all temporary files of split parts stays until the end of the whole upload.

Reproduction Steps

Not an easy test case, to be improved:

1. deploy nextcloud
2. configure an external storage as s3 backend a cloud provider (scaleway, gcs, aws, etc.)
3. use rclone to send to nextcloud using webdav.

Possible Solution

Put unlink carefully at the multiupload code somewhere?

Additional Information/Context

No response

SDK version used

3.349

Environment details (Version of PHP (php -v)? OS name and version, etc.)

8.3.27

[yenfryherrerafeliz]

Hi @antoinetran, sorry about the issues. I would need to investigate further this in order to understand what happens. I did not see any usage of multipart operations in our StreamWrapper, but I will get back to you a more concrete answer on this. I will work on reproducing the behavior first.

Thank you!

[yenfryherrerafeliz]

Hi @antoinetran, I tried to reproduce this issue by using a test 50TB file and using the sample code below, however, I got some other issues unrelated to what is being reported here. For example, I got some issues when processing the request and invoking the checksum calculation middleware due to the size of the stream, and therefore I need more information from you that actually helps me understand better what you are doing. For what I can see, the current StreamWrapper implementation does not do a multipart processing. So please, could you provide more context on this issue?, like how you are actually uploading the file, etc. A sample code will be good here where I can clearly see which parameters you are providing, and which configurations are given to the client, etc. Please see my sample code as reference.

$phpTempStreamCount = 0;
  // Register wrapper
  $s3Client = new S3Client([
      'region' => 'us-east-2',
      'http_handler' => function (RequestInterface $request) 
          use (&$phpTempStreamCount) {
          
          $resources = get_resources('stream');
          foreach ($resources as $resource) {
              if (($resource['uri'] ?? '') === 'php://temp') {
                  $phpTempStreamCount++;
                  echo "\n\rOpened stream count: $phpTempStreamCount \n";
              }
          }
          return new Response();
      }
  ]);
  $s3Client->registerStreamWrapper();
  
  // Prepare source
  $localFilePath = "./s3-large-object.txt";
  $createLargeFileFn = (function () use ($localFilePath) {
      $size = 1024 * 1024 * 1024 * 50;
      $chunkSize = 1024 * 1024 * 1024;
      $remaining = $size;
      while ($remaining > 0) {
          echo "\nwriting $chunkSize bytes...\n";
          file_put_contents(
              $localFilePath,
              str_repeat(
                  '*',
                  $chunkSize
              ),
              FILE_APPEND
          );
          $remaining -= $chunkSize;
      }
  })(...);
  if (!file_exists($localFilePath)) {
      call_user_func($createLargeFileFn);
  }
  
  // Define target
  $s3Target = "s3://sample-bucket/s3-large-object.txt";
  
  try {
      if (copy($localFilePath, $s3Target)) {
          echo "File was copied successfully to S3.\n";
      } else {
          echo "Failed to copy the file to S3.\n";
      }
  } catch (Exception $e) {
      echo "Error: " . $e->getMessage() . "\n";
  }

Thank you!

[antoinetran]

Hi,

So as I wrote above, I did not use the AWS client directly, but with Nextcloud code. It never calls registerStreamWrapper() as you did.

Instead, it calls MultipartUploader (see here) and I assumed it uses StreamWrapper implicitely.

I don't know php a lot, so it will be a bit time-consuming for me to craft a test case. Is-it possible for you to try just a MultipartUploader and see if this creates a big temp file?