GTMSessionFetcher seems to be accumulating data, even though a a download path is specified

[toshGitHubIrv]

We are using the GTMSessionFetcher (1.1.6) in order to download files from Google Drive. We are having memory issues with large files.

Previously we were using GTMFetcher, but we updated our libraries. In using the session fetcher, we are specifying a DownloadProgressBlock, DestinationFileURL and completion handler, like the following:

fetcher = [weakSelf.driveService.fetcherService fetcherWithURLString:url];

[fetcher setDownloadProgressBlock:downloadProgressHandler];
NSURL *myURL = [NSURL fileURLWithPath:fullpath];
[fetcher setDestinationFileURL:myURL];
[fetcher beginFetchWithCompletionHandler:completionHandler];

Everything downloads okay, but the memory increases and, for large files, runs out of memory.

Previously, we had the following, which did not have the memory problem:
fetcher = [weakSelf.driveService.fetcherService fetcherWithURLString:url];

[fetcher setReceivedDataBlock:receivedData]; // setReceivedDataBlock is no longer available in the updated library
[fetcher setDownloadPath:fullpath]; // setDownloadPath is no longer available in the updated library
[fetcher beginFetchWithCompletionHandler:completionHandler];

Is there some additional configuration needed, so that data is not accumulated? I tried [fetcher setAccumlatedDataBlock:nil], but that seemed to have no effect.

Thank you!
-Kevin

[mwyman]

As long as destinationFileURL is set before calling beginFetchWithCompletionHandler:, the fetcher creates an NSURLSessionDownloadTask which handles downloading to a temporary file, and no data is accumulated by the fetcher; the fetcher moves the temporary file to the destinationFileURL after downloading is complete.

accumulatedDataBlock defaults to nil, but that will have no impact if you've setup the destination file URL before starting the fetcher. But if the destinationFileURL is nil when the fetcher is started, not having an accumulatedDataBlock means the fetcher will accumulate the data itself. You could try setting the block, and see if it gets called.

[toshGitHubIrv]

Thank you for the help and quick reply.

I'm definitely setting the destinationFileURL before calling beginFetchWithCompletionHandler: and I verified destinationFileURL is not nil before the beginFetchWithCompletionHandler call. In addition, I tried setting the accumulatedDataBlock and it is not being called. So, as far as I can tell, I'm using it correctly.

However, the memory definitely continuously increases as it downloads the file. So, I'm puzzled currently. I'll continue investigating. Thanks again.

[toshGitHubIrv]

Also, if I don't set the destinationFileURL and I add the accumulatedDataBlock, then it will call the accumulatedDataBlock, as expected, for that case.

[mwyman]

I'd suggest looking in Instruments to see where the memory growth is occurring; I don't see anything in the fetcher itself should be hogging memory as a result of a download-to-file, but of course there could be a bug triggered somehow. Some StackOverflow articles hint at folks seeing memory pressure from NSURLSessionDownloadTask, although it appears some of that was Xcode/debugger related and since fixed. Instruments should show where allocated memory is lurking.

[toshGitHubIrv]

I will look into that to see what I find. Thanks again for the help.

[toshGitHubIrv]

I looked into this using Instruments and, from what I can tell, it does not show a memory issue. No leaks are shown and when looking at Memory Allocation, it stays steady around 60MiB. When setting Generations at about every 5% download (for a 600 MiB file) the Growth is in the KiB range.

I tried switching to using beginFetchWithDelegate:didFinishSelector: and this still showed the issue when debugging. In addition, I also tried adding my own accumulatedBlock handler and writing the NSData on my own within the block, but even with this, the memory issue was still there.

All very bizarre. Continuing investigation, but just wanted to update with the current status.

[mwyman]

When you run with the debugger in Xcode, do you have NSZombie or other memory diagnostics enabled?

[toshGitHubIrv]

No. NSZombie nor are any other memory diagnostics enabled. I'm just using Xcode's built-in memory tool. Nothing special.

I did try using Zombie from Instruments. For comparison:

• Without Zombie: Memory grew from about 18.6 -> 19.9 MiB in 30 seconds after starting the file download.
• With Zombie: Memory grew from 20 MiB -> 80 MiB in 30 seconds after starting the file download.

I thought maybe it was just an issue with the Debugger, but when running the App, without the debugger, it still runs out of memory.

Sometimes while downloading, it will end up timing out instead and return with the following error in the completionHandler:
Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={NSUnderlyingError=0x12e169cc0 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, NSErrorFailingURLStringKey=https://www.googleapis.com/drive/v3/files/?alt=media, NSErrorFailingURLKey=https://www.googleapis.com/drive/v3/files/?alt=media, _kCFStreamErrorDomainKey=4, _kCFStreamErrorCodeKey=-2102, NSLocalizedDescription=The request timed out.}

[toshGitHubIrv]

I downloaded the DriveSample App from https://github.com/google/google-api-objectivec-client/tree/master/Examples to see if the issue replicates outside of our App. Unfortunately the DriveSample only support Mac, but I updated the code to follow the same logic that is being used in our app. With the same code, I found the memory issue does not exist in the Mac Sample App, while it does exist in our App. Code copied below. Next I am going to look into: Updating the sample App to Support iOS to see if the issue occurs

Sample Mac GoogleDrive App (Modiefied to download file instead of thumbnail): No memory issue

(void)updateThumbnailImage {
GTLDriveFile *selectedFile = [self selectedFileListEntry];

if (selectedFile == nil) {
    return;
}

// Makes sure it doesn't download the file multiple times, since the method is called when the UI is updated, in the default sample app behavior
if (currentlyDownloading) {
return;
}
currentlyDownloading = YES;

NSString *url = [NSString
                 stringWithFormat:
                 @"https://www.googleapis.com/drive/v3/files/%@?alt=media",
                 selectedFile.identifier];

// Create the block that will handle updating the progress of the file
// download
void (^downloadProgressHandler)(int64_t bytesWritten,
                                int64_t totalBytesWritten,
                                int64_t totalBytesExpectedToWrite) =
^(int64_t bytesWritten, int64_t totalBytesWritten,
 int64_t totalBytesExpectedToWrite) {
    @autoreleasepool {
        NSLog(@"%lld", totalBytesWritten);
    }
};

  GTMSessionFetcher *fetcher =
      [self.driveService.fetcherService fetcherWithURLString:url];
  fetcher.authorizer = self.driveService.authorizer;

    [fetcher setDownloadProgressBlock:downloadProgressHandler];
    NSString * pathToLocalDestination = NSTemporaryDirectory();
    NSString *fullpath =
    [NSString stringWithFormat:@"%@%@", pathToLocalDestination, selectedFile.name];
    NSURL *myUrl = [NSURL fileURLWithPath:fullpath];
    [fetcher setDestinationFileURL:myUrl];
  [fetcher beginFetchWithCompletionHandler:^(NSData *data, NSError *error) {
      NSLog(@"completed");
      currentlyDownloading = NO;
  }];

}

Our App: Memory issue - Modified to match Sample App logic

NSString *fileName = file.fileTitle;
// Create the path and the destination file
NSString *fullpath =
[NSString stringWithFormat:@"%@%@", pathToLocalDestination, fileName];

// Create the block that will handle updating the progress of the file
// download
void (^downloadProgressHandler)(int64_t bytesWritten,
int64_t totalBytesWritten,
int64_t totalBytesExpectedToWrite) =
^(int64_t bytesWritten, int64_t totalBytesWritten,
int64_t totalBytesExpectedToWrite) {
@autoreleasepool {
NSLog(@"%lld", totalBytesWritten);
}
};
NSString _url =
[NSString stringWithFormat:
@"https://www.googleapis.com/drive/v3/files/%@?alt=media",
_hardcoded-file-id-here*];
GTMSessionFetcher *fetcher2 =
[self.driveService.fetcherService fetcherWithURLString:url];
fetcher2.authorizer = self.driveService.authorizer;

// hook up the progress and completion handlers
[fetcher2 setDownloadProgressBlock:downloadProgressHandler];
NSURL *myUrl = [NSURL fileURLWithPath:fullpath];
[fetcher2 setDestinationFileURL:myUrl];
[fetcher2 beginFetchWithCompletionHandler:^(NSData *data, NSError *error) {
NSLog(@"complete");
}];

[toshGitHubIrv]

We created a new basic iOS App with sign-in and file download, following the sample Mac app implementation. We see the same memory behavior.

Running in Simulator: Does not show a memory issue.
Running on Physical iPhone: Shows memory issue
Profiling on Physical iPhone: Does not show a memory issue

Regardless of how running, large files do not seem to download completely:

• 55 MB file: Will download and save correctly
• 657 MB file: Downloads part of the file (68448903 bytes), freezes and then returns complete, but file is not written out.

Error returned:
NSURLErrorDomain - code: 18446744073709550615
= NSUnderlyingError: domain: kCFErrorDomainCFNetwork - code: 18446744073709550615

• NSLocalizedDescription: @"The request timed out."

Currently out of ideas.

[thomasvl]

Can you post your sample so we can be sure we're comparing the same things?

[toshGitHubIrv]

Hi,
Please find attached the sample.

GDrive Sample.zip

Sorry for the delay. I did some additional investigation and found that the results vary depending on the iOS version on a physical device.

To summarize findings:

• Simulator - No memory issue
• Physical Device:

iPad - 8.4.1 - No Connection Timeout, No Memory Issue
iPhone - 8.4 - No Connection Timeout, No Memory Issue
iPhone - 9.0 - Connection Times out with NSURLErrorDomain error, Memory leak (Memory at time of connection loss 585 MB, bytes downloaded at time of connection loss 69599507 bytes)
iPhone - 9.3.5 - Connection Times out with NSURLErrorDomain error, No Memory Issue
iPad - 9.3.1 - No Connection Timeout, No Memory Issue
iPod Touch - 9.0 - Connections times out with NSURLErrorDomain error, Memory leak (Memory at time of connection loss 378 MB, bytes downloaded at time of connection loss 69628069 bytes)

Sample App Notes:

1. In ViewController.h, you'll need to update the following defines: DRIVE_KEYCHAIN_ITEM_NAME, DRIVE_CLIENT_SECRET, DRIVE_CLIENT_ID
2. Steps for reproducing are:
   a. Launch App
   b. Sign In
   c. Enter the File Identifier for a large file (Note: I didn't not want to add additional logic of selecting a file from the list, as I wanted to keep it minimal, so you'll need to get the identifier via another App or Sample). Please let me know, if you need me to add logic to make this easier.
   d. Set a breakpoint in the beginFetchWithCompletionHandler, so that you can see the error hit when it finishes
   d. Hit Download

Result:
Progress will show the number of bytes being downloaded (I had it as a percentage before, but I've found that the totalBytesWritten sometimes comes back as -1 throwing off the percentage).
For iOS Version 9.0 on a physical device, you'll see the memory increasing. In addition, on other versions of 9.X, you will see it timeout early with a NSURLErrorDomain - code: 18446744073709550615 error.

[toshGitHubIrv]

Any thoughts on this? For now, we are just preventing large file downloads on the affected iOS versions. Thank you