[source-sftp] File Size Limitation

[dyoochest]

I am trying to use SFTP(in airbyte) to transfer big files like more than 2GB. The SFTP connector doesn't recognize big files and throws the error "Internal Server Error: Cannot invoke "io.airbyte.api.model.generated.AirbyteCatalog.getStreams()" because "discovered" is null". Is it expected behavior? Can't we use the SFTP connector with big files? Did I miss anything?

[marcosmarxm]

The SFTP connector is a community-level graded connector and it is expected to have some limitations.
How many columns the file you're trying to ingest has? Because the error (I think) can be related to schema size and not the file itself.

[dyoochest]

Hi, Thanks for looking into this.

…

On Mon, Dec 11, 2023 at 7:06 AM Marcos Marx ***@***.***> wrote: The SFTP connector is a community-level graded connector and it is expected to have some limitations. How many columns the file you're trying to ingest has? Because the error (I think) can be related to schema size and not the file itself. — Reply to this email directly, view it on GitHub <#31303 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDGQC75FABWX32MLRB2ZEVTYI4AOLAVCNFSM6AAAAAA54MXV3GVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMJZGU3TC> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- *Dongchul Yoo, PhD* Senior Data Engineer American College of Chest Physicians 2595 Patriot Boulevard Glenview, Illinois 60026 USA ***@***.*** T: 224/521-9521 F: 224/521-9801 chestnet.org

[dyoochest]

Hi, Thanks for looking into this. The number of columns for the file to import was 330. You can get the a sample from http://download.cms.gov/nppes/NPI_Files.html I think that it is OK if I upload fews lines of the file.

…

On Mon, Dec 11, 2023 at 7:06 AM Marcos Marx ***@***.***> wrote: The SFTP connector is a community-level graded connector and it is expected to have some limitations. How many columns the file you're trying to ingest has? Because the error (I think) can be related to schema size and not the file itself. — Reply to this email directly, view it on GitHub <#31303 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDGQC75FABWX32MLRB2ZEVTYI4AOLAVCNFSM6AAAAAA54MXV3GVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMJZGU3TC> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- *Dongchul Yoo, PhD* Senior Data Engineer American College of Chest Physicians 2595 Patriot Boulevard Glenview, Illinois 60026 USA ***@***.*** T: 224/521-9521 F: 224/521-9801 chestnet.org