1
1
# esdump-rs
2
2
3
- Dump elasticsearch indexes to blob storage, really-really fast :rocket :
3
+ Dump Elasticsearch or OpenSearch indexes to blob storage, really-really fast :rocket :
4
+
5
+ Features:
6
+ - Super-dooper fast
7
+ - Compression with gzip or JSON
8
+ - Comes as a single, small static binary
9
+ - Natively supports blob storage on AWS, Google Cloud and Azure
10
+ - Detailed progress output and logging
11
+ - Written in Rust :crab :
12
+
13
+ ![ ] ( ./images/readme.gif )
4
14
5
15
## Installation
6
16
@@ -10,6 +20,53 @@ Dump elasticsearch indexes to blob storage, really-really fast :rocket:
10
20
11
21
## Usage
12
22
23
+ Pass the Elasticsearch or OpenSearch HTTP(s) URL and a blob storage URL. Set the credentials in the environment
24
+ (see [ example.env] ( ./example.env ) ), and run!
25
+
26
+ ``` shell
27
+ $ esdump-rs http://localhost:9200 s3://es-dump/test/ \
28
+ --index=test-index \
29
+ --batches-per-file=5 \
30
+ --batch-size=5000 \
31
+ --concurrency=10
32
+ ```
33
+
34
+ Settings such as the batch size and concurrency can be set as flags
35
+
13
36
``` shell
14
- cargo run --profile release http://localhost:9200 --index=test-index --batches-per-file=5 --batch-size=5000 s3://es-dump/test/ --env-file=test.env --concurrency=10
37
+ Usage: esdump-rs [OPTIONS] --index < INDEX> --concurrency < CONCURRENCY> --batch-size < BATCH_SIZE> --batches-per-file < BATCHES_PER_FILE> < ELASTICSEARCH_URL> < OUTPUT_LOCATION>
38
+
39
+ Arguments:
40
+ < ELASTICSEARCH_URL> Elasticsearch cluster to dump
41
+ < OUTPUT_LOCATION> Location to write results. Can be a file://, s3:// or gs:// URL
42
+
43
+ Options:
44
+ -i, --index < INDEX>
45
+ Index to dump
46
+ -c, --concurrency < CONCURRENCY>
47
+ Number of concurrent requests to use
48
+ -l, --limit < LIMIT>
49
+ Limit the total number of records returned
50
+ -b, --batch-size < BATCH_SIZE>
51
+ Number of records in each batch
52
+ --batches-per-file < BATCHES_PER_FILE>
53
+ Number of batches to write per file
54
+ -q, --query < QUERY>
55
+ A file path containing a query to execute while dumping
56
+ -f, --field < FIELD>
57
+ Specific fields to fetch
58
+ --compression < COMPRESSION>
59
+ Compress the output files [default: zstd] [possible values: gzip, zstd]
60
+ --concurrent-uploads < CONCURRENT_UPLOADS>
61
+ Max chunks to concurrently upload * per task*
62
+ --upload-size < UPLOAD_SIZE>
63
+ Size of each uploaded [default: 15MB]
64
+ -d, --distribution < DISTRIBUTION>
65
+ Distribution of the cluster [possible values: elasticsearch, opensearch]
66
+ --env-file < ENV_FILE>
67
+ Distribution of the cluster [default: .env]
68
+ -h, --help
69
+ Print help
70
+ -V, --version
71
+ Print version
15
72
` ` `
0 commit comments