Skip to content

Incremental MySQL → Postgres sync hangs during write phase (PyAirbyte stuck in tally_confirmed_writes) #871

@HARIHARANVB

Description

@HARIHARANVB

Summary

When running an incremental sync from MySQL 8.0.42 to Postgres 17 using PyAirbyte on Python 3.10.19 (Ubuntu), the sync starts normally:

Source MySQL connector works
Destination Postgres connector creates all schemas & tables
Metadata and raw tables are created successfully

But when writing actual records, the sync hangs indefinitely during:
progress_tracker.tally_confirmed_writes

Only when I manually interrupt the process (Ctrl+C), a KeyboardInterrupt occurs and the following stack trace appears.
for destination_message in progress_tracker.tally_confirmed_writes(
File ".../airbyte/progress.py", line 359, in tally_confirmed_writes
for message in messages:
File ".../airbyte/_connector_base.py", line 477, in _execute
for line in self.executor.execute(args, stdin=stdin, suppress_stderr=suppress_stderr):
File ".../airbyte/_executors/base.py", line 218, in execute
with _stream_from_subprocess(
File ".../contextlib.py", line 135, in enter
return next(self.gen)
File ".../airbyte/_executors/base.py", line 94, in _stream_from_subprocess
input_thread.join()
File ".../threading.py", line 1096, in join
self._wait_for_tstate_lock()
File ".../threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt

Environment

Python: 3.10.19
OS: Ubuntu Server 22.04
Conda env

Databases
MySQL source: 8.0.42
PostgreSQL destination: 17.0

PyAirbyte version
airbyte : 0.33.1
airbyte-api : 0.53.0
source-mysql: 3.50.9
destination-postgres: 2.4.7

Connector: source-mysql ( → destination-postgres
Both MySQL and Postgres users have superuser/admin permissions

Image Image

Request
Could the maintainers please investigate why:
progress_tracker.tally_confirmed_writes()
or the underlying destination executor stream
hangs indefinitely during sync?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions