CopyFilesOverSSH very slow

[yuvallb]

copying files from build machine to deployment server using pscp works fine.
copying files using VSTS CopyFilesOverSSH is VERY slow (12 minutes for a single 30MB file).

[madhurig]

@yuvallb: Please attach the debug log. Do you see the slowness in establishing the connection or the transfer of the file itself. Are you using your own Windows build agent?

Thanks,
Madhuri

[yuvallb]

The slowness is in the file transfer itself, not in establishing the initial connection.
This is running on a build agent.
Attached the debug log:

2017-01-24T06:49:02.5140622Z Set workingFolder to default: C:\agent\tasks\CopyFilesOverSSH\0.1.2
2017-01-24T06:49:02.7015635Z ##[debug]agent.workFolder=C:\agent\_work
2017-01-24T06:49:02.7015635Z ##[debug]loading inputs and endpoints
2017-01-24T06:49:02.7015635Z ##[debug]loading ENDPOINT_AUTH_0855B4B3-C590-41C6-9CFC-A1827337A692
2017-01-24T06:49:02.7015635Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_0855B4B3-C590-41C6-9CFC-A1827337A692_PASSWORD
2017-01-24T06:49:02.7015635Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_0855B4B3-C590-41C6-9CFC-A1827337A692_USERNAME
2017-01-24T06:49:02.7015635Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2017-01-24T06:49:02.7015635Z ##[debug]loading ENDPOINT_AUTH_SCHEME_0855B4B3-C590-41C6-9CFC-A1827337A692
2017-01-24T06:49:02.7015635Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2017-01-24T06:49:02.7015635Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2017-01-24T06:49:02.7015635Z ##[debug]loading INPUT_CLEANTARGETFOLDER
2017-01-24T06:49:02.7015635Z ##[debug]loading INPUT_CONTENTS
2017-01-24T06:49:02.7015635Z ##[debug]loading INPUT_FLATTENFOLDERS
2017-01-24T06:49:02.7015635Z ##[debug]loading INPUT_OVERWRITE
2017-01-24T06:49:02.7015635Z ##[debug]loading INPUT_SOURCEFOLDER
2017-01-24T06:49:02.7015635Z ##[debug]loading INPUT_SSHENDPOINT
2017-01-24T06:49:02.7015635Z ##[debug]loading INPUT_TARGETFOLDER
2017-01-24T06:49:02.7015635Z ##[debug]loaded 14
2017-01-24T06:49:02.8422152Z ##[debug]check path : C:\agent\tasks\CopyFilesOverSSH\0.1.2\task.json
2017-01-24T06:49:02.8422152Z ##[debug]set resource file to: C:\agent\tasks\CopyFilesOverSSH\0.1.2\task.json
2017-01-24T06:49:02.8422152Z ##[debug]system.culture=en-US
2017-01-24T06:49:02.8422152Z ##[debug]sshEndpoint=0855b4b3-c590-41c6-9cfc-a1827337a692
2017-01-24T06:49:02.8422152Z ##[debug]0855b4b3-c590-41c6-9cfc-a1827337a692 auth param username = ********
2017-01-24T06:49:02.8422152Z ##[debug]0855b4b3-c590-41c6-9cfc-a1827337a692 auth param password = ********
2017-01-24T06:49:02.8422152Z ##[debug]0855b4b3-c590-41c6-9cfc-a1827337a692 data host = 123.45.67.89
2017-01-24T06:49:02.8422152Z ##[debug]0855b4b3-c590-41c6-9cfc-a1827337a692 data port = 22
2017-01-24T06:49:02.8422152Z ##[debug]Using username and password for ssh connection.
2017-01-24T06:49:02.8422152Z ##[debug]contents=**/*.tar.gz
2017-01-24T06:49:02.8422152Z ##[debug]sourceFolder=C:\agent\_work\51d260ba4\BuildDef\drop
2017-01-24T06:49:02.8422152Z ##[debug]check path : C:\agent\_work\51d260ba4\BuildDef\drop
2017-01-24T06:49:02.8422152Z ##[debug]targetFolder=/home/********/package-project/20170124.1
2017-01-24T06:49:02.8422152Z ##[debug]cleanTargetFolder=false
2017-01-24T06:49:02.8422152Z ##[debug]overwrite=true
2017-01-24T06:49:02.8422152Z ##[debug]flattenFolders=false
2017-01-24T06:49:02.8422152Z Setting up SSH connection to remote host 123.45.67.89.
2017-01-24T06:49:05.7484795Z ##[debug]include content pattern: **/*.tar.gz
2017-01-24T06:49:05.7484795Z ##[debug]find findPath=C:\agent\_work\51d260ba4\BuildDef\drop, options=undefined
2017-01-24T06:49:05.7484795Z ##[debug]  C:\agent\_work\51d260ba4\BuildDef\drop
2017-01-24T06:49:05.7484795Z ##[debug]    is a directory
2017-01-24T06:49:05.7484795Z ##[debug]  C:\agent\_work\51d260ba4\BuildDef\drop\package-name_Build-2103.tar.gz
2017-01-24T06:49:05.7484795Z ##[debug]    is a file
2017-01-24T06:49:05.7484795Z ##[debug]2 results
2017-01-24T06:49:05.7484795Z ##[debug]allFiles contains 1 files
2017-01-24T06:49:05.7484795Z ##[debug]Include matching C:\agent\_work\51d260ba4\BuildDef\drop\**\*.tar.gz
2017-01-24T06:49:05.7484795Z ##[debug]match patterns: C:\agent\_work\51d260ba4\BuildDef\drop\**\*.tar.gz
2017-01-24T06:49:05.7484795Z ##[debug]match options: [object Object]
2017-01-24T06:49:05.7484795Z ##[debug]applying pattern: C:\agent\_work\51d260ba4\BuildDef\drop\**\*.tar.gz
2017-01-24T06:49:05.7484795Z ##[debug]matched 1 items
2017-01-24T06:49:05.7484795Z ##[debug]Include matched 1 files
2017-01-24T06:49:05.7484795Z ##[debug]Number of files to copy = 1
2017-01-24T06:49:05.7484795Z ##[debug]filesToCopy = C:\agent\_work\51d260ba4\BuildDef\drop\package-name_Build-2103.tar.gz
2017-01-24T06:49:05.7484795Z Found 1 files to copy to the remote machine.
2017-01-24T06:49:05.7484795Z ##[debug]fileToCopy = C:\agent\_work\51d260ba4\BuildDef\drop\package-name_Build-2103.tar.gz
2017-01-24T06:49:05.7484795Z ##[debug]relativePath = package-name_Build-2103.tar.gz
2017-01-24T06:49:05.7484795Z Copying file C:\agent\_work\51d260ba4\BuildDef\drop\package-name_Build-2103.tar.gz to /home/********/package-project/20170124.1/package-name_Build-2103.tar.gz on remote machine.
2017-01-24T06:49:05.7484795Z ##[debug]Upload C:\agent\_work\51d260ba4\BuildDef\drop\package-name_Build-2103.tar.gz to /home/********/package-project/20170124.1/package-name_Build-2103.tar.gz on remote machine.
2017-01-24T07:01:18.8264918Z Completed copying 1 files to the remote machine.
2017-01-24T07:01:18.8264918Z ##[debug]Closing the client connection
2017-01-24T07:01:18.8264918Z ##[debug]Failed to close SFTP client.
 

[danports]

I was experimenting with VSTS today and was excited to see that it had native support for SFTP transfers...but then when I tried it, I soon concluded that this task transferred files so slowly it was basically unusable. I tried uploading files from my own agent via this task and via WinSCP; this task used only ~1Mbps of the agent's available upstream bandwidth, whereas the WinSCP transfer maxed out the agent's upstream bandwidth (~5Mbps). Transfer performance on a hosted agent seemed to be even worse - I had to kill the task before it completed because it was taking so long. There are certainly workarounds (pscp, WinSCP, etc.), but it would be nice to use this built-in task without resorting to one of those.

[yuvallb]

I replaced the copyOverSsh task with a command line task running pscp, using a preconfigured putty session with a public key authentication. It works perfectly.
I agree it would be nice to have the copyOverSsh task work, as my workaround requires too much configuration and moving parts.

[madhurig]

@yuvallb: Sorry for the issues you are facing and thanks for providing the workaround you are using. Our tests with files and directories in the size range you mentioned didn't show performance issues (completed in < minute). Are you using the hosted build VMs or your own windows build servers? What types of files are you transferring?

@danports:
We are using a cross platform native node library to do the transfer. I can imagine tools built specifically for Windows like WinSCP might be faster. We will investigate the best approach to speed up the transfers in the task.

Thanks,
Madhuri

[yuvallb]

I am using my own windows build server (VM on Azure), and transferring a zip file to a linux server (~35MB).

[late-in-autumn]

I noticed that in the debug log, there is a line:
2017-01-24T07:01:18.8264918Z ##[debug]Failed to close SFTP client.

This indicates that the agent is using SFTP, and in comparision @danports uses SCP. In http://www.jscape.com/blog/scp-vs-sftp it mentioned:

1. Speed
   When comparing SCP vs SFTP in terms of speed, i.e., in transferring files, SCP is generally much faster. This is due to the way it confirms received packets. Traditionally, SFTP has to ACK (acknowledge) every tiny packet, while SCP does not. That's why the disparity becomes more evident in high latency networks.

Advantage: SCP

So my guess that the agent is using SFTP is another reason. It will be informational to see the performance of vsts-tasks agent, pscp (which uses SCP) and psftp (which uses SFTP).