-
Notifications
You must be signed in to change notification settings - Fork 71
Description
Summary
When running an incremental sync from MySQL 8.0.42 to Postgres 17 using PyAirbyte on Python 3.10.19 (Ubuntu), the sync starts normally:
Source MySQL connector works
Destination Postgres connector creates all schemas & tables
Metadata and raw tables are created successfully
But when writing actual records, the sync hangs indefinitely during:
progress_tracker.tally_confirmed_writes
Only when I manually interrupt the process (Ctrl+C), a KeyboardInterrupt occurs and the following stack trace appears.
for destination_message in progress_tracker.tally_confirmed_writes(
File ".../airbyte/progress.py", line 359, in tally_confirmed_writes
for message in messages:
File ".../airbyte/_connector_base.py", line 477, in _execute
for line in self.executor.execute(args, stdin=stdin, suppress_stderr=suppress_stderr):
File ".../airbyte/_executors/base.py", line 218, in execute
with _stream_from_subprocess(
File ".../contextlib.py", line 135, in enter
return next(self.gen)
File ".../airbyte/_executors/base.py", line 94, in _stream_from_subprocess
input_thread.join()
File ".../threading.py", line 1096, in join
self._wait_for_tstate_lock()
File ".../threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
Environment
Python: 3.10.19
OS: Ubuntu Server 22.04
Conda env
Databases
MySQL source: 8.0.42
PostgreSQL destination: 17.0
PyAirbyte version
airbyte : 0.33.1
airbyte-api : 0.53.0
source-mysql: 3.50.9
destination-postgres: 2.4.7
Connector: source-mysql ( → destination-postgres
Both MySQL and Postgres users have superuser/admin permissions
Request
Could the maintainers please investigate why:
progress_tracker.tally_confirmed_writes()
or the underlying destination executor stream
hangs indefinitely during sync?