Problem configuring an SFTP Bulk source #39123
-
|
Hey guys! I have a problem configuring an SFTP Bulk source. On my server there are two types of files together at the root, but I would like to ingest them separately into two streams. Could anyone help me with this? |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 2 replies
-
|
I am having this same issue |
Beta Was this translation helpful? Give feedback.
-
|
I'm sorry you're having the same problem. Unfortunately in my case I had to deal with data separation during refinement, as this glob filter did not work during ingestion. It was the only way I found to deliver the data to the client :/ |
Beta Was this translation helpful? Give feedback.
-
|
I am also in this same boat. I have briefly gone through the source code of the connector, although it was not all that insightful at the depth I went. I intend to dig deeper and see what is going on soon. |
Beta Was this translation helpful? Give feedback.
-
|
Found the issue! The SFTP Bulk code doesn't check whether the provided Folder Path already ends in a slash (which is the case by default), and may double it up. All you have to do is add that double slash to the glob where appropriate. E.g. in @isisgoliveira's case if you have not set a folder path and your files are at the root you will need a glob pattern like '//my_prefix_*.csv' Also note if your files are in a folder, you need to include that folder in your glob too. E.g. for files in a folder 'my_folder', your glob will need to be 'my_folder/my_prefix_*.csv' even when your folder path is 'my_folder' (if you don't set a folder path, it'll need to be '//my_folder/my_prefix_*.csv'!!). I tried this for my use-case and it seems to be working fine now! |
Beta Was this translation helpful? Give feedback.
-
|
This helped me immensely as I was stuck with this issue for a while. Thank you so much! |
Beta Was this translation helpful? Give feedback.
-
|
do wildcards work if the file i want is in a variety of folders? e.g. /myprefix_.csv |
Beta Was this translation helpful? Give feedback.
Found the issue! The SFTP Bulk code doesn't check whether the provided Folder Path already ends in a slash (which is the case by default), and may double it up. All you have to do is add that double slash to the glob where appropriate.
E.g. in @isisgoliveira's case if you have not set a folder path and your files are at the root you will need a glob pattern like '//my_prefix_*.csv'
Also note if your files are in a folder, you need to include that folder in your glob too. E.g. for files in a folder 'my_folder', your glob will need to be 'my_folder/my_prefix_*.csv' even when your folder path is 'my_folder' (if you don't set a folder path, it'll need to be '//my_folder/my_prefix_*.csv'!!).
I…