|
1 | 1 | # qdrant-migration |
2 | 2 |
|
3 | | -This tool helps to migrate data to Qdrant from other sources. |
| 3 | +This tool helps to migrate data to Qdrant from other sources. It will stream all vectors from a collection in the source Qdrant instance to the target Qdrant instance. |
| 4 | + |
| 5 | +The target collection can have a different replication or sharding configuration. |
4 | 6 |
|
5 | 7 | Supported sources: |
6 | 8 |
|
7 | 9 | * Other Qdrant instances |
8 | 10 |
|
9 | 11 | ## Installation |
10 | 12 |
|
| 13 | +The easiest way to run the qdrant-migration tool is as a container. You can run it any machine where you have connectivity to both the source and the target Qdrant databases. Direct connectivity between both databases is not required. For optimal performance, you should run the tool on a machine with a fast network connection and minimum latency to both databases. |
| 14 | + |
| 15 | +To pull the latest image run: |
| 16 | + |
| 17 | +```bash |
| 18 | +$ docker pull registry.cloud.qdrant.io/library/qdrant-migration |
| 19 | +``` |
| 20 | + |
| 21 | +In addtion, every release providies precompiled binaries for all major OS and CPU architectures. You can download the latest release from the [releases page](https://github.com/qdrant/migration/releases). |
| 22 | + |
11 | 23 | ## Usage |
| 24 | + |
| 25 | +To migrate from one Qdrant instance to another, you can provide the following parameters: |
| 26 | + |
| 27 | +```bash |
| 28 | +$ docker run --rm -it registry.cloud.qdrant.io/library/qdrant-migration qdrant --help |
| 29 | +Usage: migration qdrant --source-url=STRING --source-collection=STRING --target-url=STRING --target-collection=STRING [flags] |
| 30 | + |
| 31 | +Migrate data from a Qdrant database to Qdrant. |
| 32 | + |
| 33 | +Flags: |
| 34 | + -h, --help Show context-sensitive help. |
| 35 | + --debug Enable debug mode. |
| 36 | + --trace Enable trace mode. |
| 37 | + --skip-tls-verification Skip TLS verification. |
| 38 | + --version Print version information and quit |
| 39 | + |
| 40 | + --source-url=STRING Source GRPC URL, e.g. https://your-qdrant-hostname:6334 |
| 41 | + --source-collection=STRING Source collection |
| 42 | + --source-api-key=STRING Source API key ($SOURCE_API_KEY) |
| 43 | + --target-url=STRING Target GRPC URL, e.g. https://your-qdrant-hostname:6334 |
| 44 | + --target-collection=STRING Target collection |
| 45 | + --target-api-key=STRING Target API key ($TARGET_API_KEY) |
| 46 | + -b, --batch-size=50 Batch size |
| 47 | + -c, --create-target-collection Create the target collection if it does not exist |
| 48 | + -m, --migration-marker=STRING Migration marker to resume the migration |
| 49 | +``` |
| 50 | +
|
| 51 | +Example: |
| 52 | +
|
| 53 | +```bash |
| 54 | +$ docker run --rm -it registry.cloud.qdrant.io/library/qdrant-migration qdrant \ |
| 55 | + --source-url 'https://source-qdrant-hostname:6334' \ |
| 56 | + --source-collection 'source-collection' \ |
| 57 | + --target-url 'https://target-qdrant-hostname:6334' \ |
| 58 | + --target-collection 'target-collection' |
| 59 | +``` |
| 60 | +
|
| 61 | +You can provide the API keys either as command line arguments or as environment variables: |
| 62 | +
|
| 63 | +```bash |
| 64 | +$ docker run --rm -it \ |
| 65 | + -e SOURCE_API_KEY='xyz' \ |
| 66 | + registry.cloud.qdrant.io/library/qdrant-migration qdrant \ |
| 67 | + --source-url 'https://source-qdrant-hostname:6334' \ |
| 68 | + --source-collection 'source-collection' \ |
| 69 | + --target-url 'https://target-qdrant-hostname:6334' \ |
| 70 | + --target-collection 'target-collection' \ |
| 71 | + --target-api-key 'abc' |
| 72 | +``` |
| 73 | +
|
| 74 | +If you want to resume a cancelled migration, or if you want to migrate vectors that may have been added after the last migration run, you can pass the migration marker as a flag, which is printed out at the beginning of the migration process: |
| 75 | +
|
| 76 | +```bash |
| 77 | +$ docker run --rm -it registry.cloud.qdrant.io/library/qdrant-migration qdrant \ |
| 78 | + --source-url 'https://source-qdrant-hostname:6334' \ |
| 79 | + --source-collection 'source-collection' \ |
| 80 | + --target-url 'https://target-qdrant-hostname:6334' \ |
| 81 | + --target-collection 'target-collection' \ |
| 82 | + --migration-marker 'migration-2025-02-14T19:18:30+01:00' |
| 83 | +``` |
| 84 | +
|
| 85 | +## Development |
| 86 | +
|
| 87 | +### Running tests |
| 88 | +
|
| 89 | +The migration tool has two kind of tests, Golang unit tests and integration tests written with [bats](https://bats-core.readthedocs.io/). |
| 90 | +
|
| 91 | +To run the Golang tests, execute: |
| 92 | +
|
| 93 | +```bash |
| 94 | +$ make test_unit |
| 95 | +``` |
| 96 | +
|
| 97 | +To run the integration tests, execute: |
| 98 | +
|
| 99 | +```bash |
| 100 | +$ make test_integration |
| 101 | +``` |
| 102 | +
|
| 103 | +To run all tests, execute: |
| 104 | +
|
| 105 | +```bash |
| 106 | +$ make test |
| 107 | +``` |
| 108 | +
|
| 109 | +### Linting |
| 110 | +
|
| 111 | +This project uses [golangci-lint](https://golangci-lint.run/) to lint the code. To run the linter, execute: |
| 112 | +
|
| 113 | +```bash |
| 114 | +$ make lint |
| 115 | +``` |
| 116 | +
|
| 117 | +Code formatting is ensured with [gofmt](https://pkg.go.dev/cmd/gofmt). To format the code, execute: |
| 118 | +
|
| 119 | +```bash |
| 120 | +$ make fmt |
| 121 | +``` |
| 122 | +
|
| 123 | +### Pre-commit hooks |
| 124 | +
|
| 125 | +This project uses [pre-commit](https://pre-commit.com/) to run the linter and the formatter before every commit. To install the pre-commit hooks, execute: |
| 126 | +
|
| 127 | +```bash |
| 128 | +$ pre-commit install-hooks |
| 129 | +``` |
| 130 | +
|
| 131 | +## Releasing a new version |
| 132 | +
|
| 133 | +To release a new version create and push a release branch that follows the pattern `release/vX.Y.Z`. The release branch should be created from the `main` branch, or from the latest release branch in case of a hot fix. |
| 134 | +
|
| 135 | +A GitHub Action will then create a new release, build the binaries, push the Docker image to the registry, and create a Git tag. |
0 commit comments