MySQL -> ClickHouse Incremental Sync Issues

[mattdavenport]

I have about 300GB I am trying to sync from MySQL. The sync process has yet to complete fully, as it crashes midway through. The main error message I am seeing is:

Additional Failure Information: scheduledEventId=84, startedEventId=85, activityType='GetSyncWorkflowInputWithAttemptNumber', activityId='b6e3a915-2ac4-3c68-8faf-7f33b5e5379e', identity='1@8871b39ce0ca', retryState=RETRY_STATE_NON_RETRYABLE_FAILURE

The last logfile entry is: INFO i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to set attempt sync config and it does not get past that step. At this stage, multiple tables have been read and processed. I've checked resource usage on the machine and that seems to be fine. Looking further into the initial sync attempt, I also noticed a bunch of these lines before the failure:

2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))
2023-12-21 16:23:13 ^[[42mnormalization^[[0m >
2023-12-21 16:23:13 ^[[42mnormalization^[[0m > Code: 373. DB::Exception: Session dbt::743cea8d-03f3-455c-af77-b117c51b8af9 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 23.11.2.11 (official build))

Is there some way I can resolve this? It seems like the process almost finishes, then due to lock contention it fails. Additional sync retries yield the same results.

Airbyte Version: 0.44.3
Mysql Connector: 3.3.0
Clickhouse Connector: 0.2.5

Thanks!

[davinchia]

Hi, please give the latest Clickhouse version a shot. We've vetted it on larger datasets and are confident it'll work well for larger syncs. Thanks!

[mattdavenport]

Thanks for coming back to this one @davinchia! I'll give it a shot.