Skip to content

Commit f2267e7

Browse files
committed
Add a README
1 parent e221fe8 commit f2267e7

File tree

3 files changed

+61
-2
lines changed

3 files changed

+61
-2
lines changed

README.md

Lines changed: 59 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,16 @@
11
# esdump-rs
22

3-
Dump elasticsearch indexes to blob storage, really-really fast :rocket:
3+
Dump Elasticsearch or OpenSearch indexes to blob storage, really-really fast :rocket:
4+
5+
Features:
6+
- Super-dooper fast
7+
- Compression with gzip or JSON
8+
- Comes as a single, small static binary
9+
- Natively supports blob storage on AWS, Google Cloud and Azure
10+
- Detailed progress output and logging
11+
- Written in Rust :crab:
12+
13+
![](./images/readme.gif)
414

515
## Installation
616

@@ -10,6 +20,53 @@ Dump elasticsearch indexes to blob storage, really-really fast :rocket:
1020

1121
## Usage
1222

23+
Pass the Elasticsearch or OpenSearch HTTP(s) URL and a blob storage URL. Set the credentials in the environment
24+
(see [example.env](./example.env)), and run!
25+
26+
```shell
27+
$ esdump-rs http://localhost:9200 s3://es-dump/test/ \
28+
--index=test-index \
29+
--batches-per-file=5 \
30+
--batch-size=5000 \
31+
--concurrency=10
32+
```
33+
34+
Settings such as the batch size and concurrency can be set as flags
35+
1336
```shell
14-
cargo run --profile release http://localhost:9200 --index=test-index --batches-per-file=5 --batch-size=5000 s3://es-dump/test/ --env-file=test.env --concurrency=10
37+
Usage: esdump-rs [OPTIONS] --index <INDEX> --concurrency <CONCURRENCY> --batch-size <BATCH_SIZE> --batches-per-file <BATCHES_PER_FILE> <ELASTICSEARCH_URL> <OUTPUT_LOCATION>
38+
39+
Arguments:
40+
<ELASTICSEARCH_URL> Elasticsearch cluster to dump
41+
<OUTPUT_LOCATION> Location to write results. Can be a file://, s3:// or gs:// URL
42+
43+
Options:
44+
-i, --index <INDEX>
45+
Index to dump
46+
-c, --concurrency <CONCURRENCY>
47+
Number of concurrent requests to use
48+
-l, --limit <LIMIT>
49+
Limit the total number of records returned
50+
-b, --batch-size <BATCH_SIZE>
51+
Number of records in each batch
52+
--batches-per-file <BATCHES_PER_FILE>
53+
Number of batches to write per file
54+
-q, --query <QUERY>
55+
A file path containing a query to execute while dumping
56+
-f, --field <FIELD>
57+
Specific fields to fetch
58+
--compression <COMPRESSION>
59+
Compress the output files [default: zstd] [possible values: gzip, zstd]
60+
--concurrent-uploads <CONCURRENT_UPLOADS>
61+
Max chunks to concurrently upload *per task*
62+
--upload-size <UPLOAD_SIZE>
63+
Size of each uploaded [default: 15MB]
64+
-d, --distribution <DISTRIBUTION>
65+
Distribution of the cluster [possible values: elasticsearch, opensearch]
66+
--env-file <ENV_FILE>
67+
Distribution of the cluster [default: .env]
68+
-h, --help
69+
Print help
70+
-V, --version
71+
Print version
1572
```

example.env

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
ES_DUMP_USERNAME=foo
2+
ES_DUMP_PASSWORD=bar

images/readme.gif

742 KB
Loading

0 commit comments

Comments
 (0)