Skip to content

refactor: Parallel PG -> Qdrant migration for improved throughput #198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Anush008 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

refactor: Parallel PG -> Qdrant migration for improved throughput #198

Anush008 wants to merge 4 commits into from
+367 −30

Conversation

@Anush008
Copy link
Member

@Anush008 Anush008 commented Jan 6, 2026 •
edited
Loading

Description

Adds support for migrating from PG to Qdrant with parallel batches.

Plus, #199 was merged into this.

How?

Methodology same as #168.

Samples random sorted keys, creates non-overlapping ranges, then workers migrate the ranges independently. Each range tracks its own offset for resumability.

By default --migration.num-workers is set to the number of CPU cores.

If --migration.num-workers is set to 1, the current sequential migration is used.

Results

BEFORE - Migrating 500k points with a batch size of 250.

┌─────────────────────────────────────────────────────────────────────┐
| From → To: | embeddings_table@postgres  →  target-collection@qdrant |
└─────────────────────────────────────────────────────────────────────┘

INFO  Starting from the beginning                                                  
[500000/500000] ██████████████████████████████████████████████████ 100% | 4m10s
SUCCESS  Data migration finished successfully
INFO  Target collection has 500000 points

THIS PR - Migrating 500k points with a batch size of 250 and 8 workers.

┌─────────────────────────────────────────────────────────────────────┐
| From → To: | embeddings_table@postgres  →  target-collection@qdrant |
└─────────────────────────────────────────────────────────────────────┘

INFO  Using parallel migration with 8 workers
INFO  Starting from the beginning                                                  
[500000/500000] ████████████████████████████████████████████████████ 100% | 14s
SUCCESS  Data migration finished successfully
INFO  Target collection has 500000 points

About 17x faster.

@Anush008 Anush008 requested a review from a team as a code owner January 6, 2026 13:22
@Anush008 Anush008 marked this pull request as draft January 6, 2026 13:25
@Anush008 Anush008 marked this pull request as ready for review January 7, 2026 07:21
@Anush008
chore: Misc updates
f6669ce 
Signed-off-by: Anush008 <[email protected]>
@Anush008
refactor: PG UUID columns handling (#199)
1810c92 
* refactor: PG UUID handling

Signed-off-by: Anush008 <[email protected]>

* chore: Misc updates

Signed-off-by: Anush008 <[email protected]>

---------

Signed-off-by: Anush008 <[email protected]>
@Anush008 Anush008 enabled auto-merge (squash) January 9, 2026 18:12
@Anush008
Copy link
Member Author

Anush008 commented Jan 9, 2026

Merged #199 into here.

@Anush008 Anush008 requested a review from bashofmann January 9, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bashofmann bashofmann Awaiting requested review from bashofmann

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Anush008