[source-slack] Pull all public channels with messages and Threads is taking more then 5 hours to execute for 1947 records

[SidduHussain]

Dear Team,

I am trying to use Slack --> Snowflake pipeline and it is not working as expected. it is taking 5 hours to load 1947 records.

Above are the settings I am using and we have 31 channels with approximately 991 messages. I have tried the same in Fivetran and it completed the full sync in 7 minutes.

I am wondering if anyone has similar timelines to load data. I feel 5 hours for loading 1947 records is not an ideal scenario.

Thanks
Siddu Hussain V

[marcosmarxm]

Can you share the sync logs to identify what is causing the delay or taking more time?

[SidduHussain]

@marcosmarxm - Do need a specific Section in log, log has user data when back off is tried so I will not be able to extract the complete log

[marcosmarxm]

Can you share the sync summary?

[SidduHussain]

you can ignore the run after 19th Jan as this is only user and channel data and more of full refresh

but if you see the run on 19th Jan , you can see that it is very high

I have tried it in both your cloud and my OSS instance and have similar output .

Let me know if you want to connect to discuss further.

Thanks
Siddu Hussain V

[marcosmarxm]

Can you share the sync sumary or a the complete log file?

[SidduHussain]

@marcosmarxm ,

unfortunately it has PII in backoff tries and I will not be able to share it, but here is another observation.
getting Threads is taking too much time so I removed Threads and it went down to 42 secs
We have huge volumes of data around 300K per day in production. I was wondering if this is expected behavior for threads.

Thanks
Siddu Hussain V