[source-zendesk-support] HeartbeatTimeoutException Heartbeat has stopped. Heartbeat freshness threshold

[pranasas-karutis]

Hi,

I am trying to make my Zendesk Support connector work. Initial sync goes through fine; however, later on it starts getting one same "Heartbeat has stopped" and usually after multiple attempts it fails. Sometimes it goes through after several attempts.

Logs are rather uninformative, as the message doesn't make too much sense to me. One other thing - it usually says X records extracted, 0 records loaded where X is somewhat expected number of records for a normal sync (e.g., Last attempt: 660.21 MB |
116,832 records extracted | no records loaded | Job id: 63805 | 3h 35m 21s. ) Does it mean the API response fetch is successful and that it fails later on, trying to get the records into Bigquery?

My setup: Airbyte Open Source on VM on GCP, destination is Bigquery. Currently running the latest version of Airbyte and of the Zendesk Support connector.

Any ideas or suggestions appreciated!

Update: attaching a relevant excerpt from one failed attempt
failed_sync_log.txt

[sajarin]

It seems like you're experiencing a similar issue to what has been reported in the past where the Zendesk Support connector freezes or fails after some hours of syncing source.

The "Heartbeat has stopped" message usually indicates that the process is taking longer than expected. The fact that it says X records extracted, 0 records loaded could indeed mean that the API response fetch is successful but it fails when trying to load the records into Bigquery.

There have been instances where the error comes from a non-JSON response sent by the Zendesk API. The connector tries to deserialize JSON payload but fails because the response does not contain JSON. This is probably because you hit the rate limit of the Zendesk API and the connector does not manage the error response well source.

You might want to check if you're hitting the rate limit of the Zendesk API. The connector is restricted by normal Zendesk requests limitation source.