SFTP file read very slow, length is ignored #68
Description
Even though SFTPFile methods read(from offset:length:) and readAll() request specific length of bytes to be read, the underlying NIO channel ignores that and seems to discard all data past its default chunk size value which is effectively:
NonBlockingFileIO.defaultChunkSize = 128 * 1024 // 131072 bytes or 128kbThis results in very slow download times as the same portions of file are being multiple times but NIO stripping (or so it seems) all data past 131072 bytes for each request. Effectively the file is being read many times over again and again.
The only way to somewhat remedy the problem is to use the length of 131072 but the download speed is still too slow due to many requests needed with 128k chunks.
The file's attributes are known and its attributes.size contains correct number of bytes. The readAll() attempts to read the file at once using its attributes.size value.
Code to reproduce:
let buffer = try await file.readAll()Client:
- OS: macOS Sonoma
- Client: Citadel
Server:
- OS: linux
- Server: remote SFTPD
Additional context
The log of reading a 5Mb file where readAll() is requesting full file download but length is being ignored and replaced by default NIO value.
[ DEBUG ] SFTP starting chunked read operation on file 0100 (Citadel/SFTPFile.swift:102)
[ TRACE ] SFTP OUT: read{7}(0100, 5207962 bytes from 0) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ TRACE ] SFTP OUT: read{8}(0100, 5076890 bytes from 131072) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ TRACE ] SFTP OUT: read{9}(0100, 4945818 bytes from 262144) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ TRACE ] SFTP OUT: read{10}(0100, 4814746 bytes from 393216) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ TRACE ] SFTP OUT: read{11}(0100, 4683674 bytes from 524288) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
...
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ TRACE ] SFTP OUT: read{42}(0100, 620442 bytes from 4587520) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ TRACE ] SFTP OUT: read{43}(0100, 489370 bytes from 4718592) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ TRACE ] SFTP OUT: read{44}(0100, 358298 bytes from 4849664) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ TRACE ] SFTP OUT: read{45}(0100, 227226 bytes from 4980736) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 131072 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ TRACE ] SFTP OUT: read{46}(0100, 96154 bytes from 5111808) (Citadel/SFTPClient.swift:93)
[ DEBUG ] SFTP read 96154 bytes from file 0100 (Citadel/SFTPFile.swift:81)
[ DEBUG ] SFTP completed chunked read operation on file 0100 (Citadel/SFTPFile.swift:128)