Skip to content

Commit d82fb66

Browse files
authored
Add readme (#9)
1 parent b83993e commit d82fb66

File tree

1 file changed

+125
-1
lines changed

1 file changed

+125
-1
lines changed

README.md

Lines changed: 125 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,135 @@
11
# qdrant-migration
22

3-
This tool helps to migrate data to Qdrant from other sources.
3+
This tool helps to migrate data to Qdrant from other sources. It will stream all vectors from a collection in the source Qdrant instance to the target Qdrant instance.
4+
5+
The target collection can have a different replication or sharding configuration.
46

57
Supported sources:
68

79
* Other Qdrant instances
810

911
## Installation
1012

13+
The easiest way to run the qdrant-migration tool is as a container. You can run it any machine where you have connectivity to both the source and the target Qdrant databases. Direct connectivity between both databases is not required. For optimal performance, you should run the tool on a machine with a fast network connection and minimum latency to both databases.
14+
15+
To pull the latest image run:
16+
17+
```bash
18+
$ docker pull registry.cloud.qdrant.io/library/qdrant-migration
19+
```
20+
21+
In addtion, every release providies precompiled binaries for all major OS and CPU architectures. You can download the latest release from the [releases page](https://github.com/qdrant/migration/releases).
22+
1123
## Usage
24+
25+
To migrate from one Qdrant instance to another, you can provide the following parameters:
26+
27+
```bash
28+
$ docker run --rm -it registry.cloud.qdrant.io/library/qdrant-migration qdrant --help
29+
Usage: migration qdrant --source-url=STRING --source-collection=STRING --target-url=STRING --target-collection=STRING [flags]
30+
31+
Migrate data from a Qdrant database to Qdrant.
32+
33+
Flags:
34+
-h, --help Show context-sensitive help.
35+
--debug Enable debug mode.
36+
--trace Enable trace mode.
37+
--skip-tls-verification Skip TLS verification.
38+
--version Print version information and quit
39+
40+
--source-url=STRING Source GRPC URL, e.g. https://your-qdrant-hostname:6334
41+
--source-collection=STRING Source collection
42+
--source-api-key=STRING Source API key ($SOURCE_API_KEY)
43+
--target-url=STRING Target GRPC URL, e.g. https://your-qdrant-hostname:6334
44+
--target-collection=STRING Target collection
45+
--target-api-key=STRING Target API key ($TARGET_API_KEY)
46+
-b, --batch-size=50 Batch size
47+
-c, --create-target-collection Create the target collection if it does not exist
48+
-m, --migration-marker=STRING Migration marker to resume the migration
49+
```
50+
51+
Example:
52+
53+
```bash
54+
$ docker run --rm -it registry.cloud.qdrant.io/library/qdrant-migration qdrant \
55+
--source-url 'https://source-qdrant-hostname:6334' \
56+
--source-collection 'source-collection' \
57+
--target-url 'https://target-qdrant-hostname:6334' \
58+
--target-collection 'target-collection'
59+
```
60+
61+
You can provide the API keys either as command line arguments or as environment variables:
62+
63+
```bash
64+
$ docker run --rm -it \
65+
-e SOURCE_API_KEY='xyz' \
66+
registry.cloud.qdrant.io/library/qdrant-migration qdrant \
67+
--source-url 'https://source-qdrant-hostname:6334' \
68+
--source-collection 'source-collection' \
69+
--target-url 'https://target-qdrant-hostname:6334' \
70+
--target-collection 'target-collection' \
71+
--target-api-key 'abc'
72+
```
73+
74+
If you want to resume a cancelled migration, or if you want to migrate vectors that may have been added after the last migration run, you can pass the migration marker as a flag, which is printed out at the beginning of the migration process:
75+
76+
```bash
77+
$ docker run --rm -it registry.cloud.qdrant.io/library/qdrant-migration qdrant \
78+
--source-url 'https://source-qdrant-hostname:6334' \
79+
--source-collection 'source-collection' \
80+
--target-url 'https://target-qdrant-hostname:6334' \
81+
--target-collection 'target-collection' \
82+
--migration-marker 'migration-2025-02-14T19:18:30+01:00'
83+
```
84+
85+
## Development
86+
87+
### Running tests
88+
89+
The migration tool has two kind of tests, Golang unit tests and integration tests written with [bats](https://bats-core.readthedocs.io/).
90+
91+
To run the Golang tests, execute:
92+
93+
```bash
94+
$ make test_unit
95+
```
96+
97+
To run the integration tests, execute:
98+
99+
```bash
100+
$ make test_integration
101+
```
102+
103+
To run all tests, execute:
104+
105+
```bash
106+
$ make test
107+
```
108+
109+
### Linting
110+
111+
This project uses [golangci-lint](https://golangci-lint.run/) to lint the code. To run the linter, execute:
112+
113+
```bash
114+
$ make lint
115+
```
116+
117+
Code formatting is ensured with [gofmt](https://pkg.go.dev/cmd/gofmt). To format the code, execute:
118+
119+
```bash
120+
$ make fmt
121+
```
122+
123+
### Pre-commit hooks
124+
125+
This project uses [pre-commit](https://pre-commit.com/) to run the linter and the formatter before every commit. To install the pre-commit hooks, execute:
126+
127+
```bash
128+
$ pre-commit install-hooks
129+
```
130+
131+
## Releasing a new version
132+
133+
To release a new version create and push a release branch that follows the pattern `release/vX.Y.Z`. The release branch should be created from the `main` branch, or from the latest release branch in case of a hot fix.
134+
135+
A GitHub Action will then create a new release, build the binaries, push the Docker image to the registry, and create a Git tag.

0 commit comments

Comments
 (0)