-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Hi.
Thanks for writing this tool.
Unfortunately I'm not able to figure out how the query parameter is working. It seems not to be used at all. See below:
I'm trying to use this tool to dump parts of our datastreams by using this query (in file /query.json)
{
"query": {
"range": {
"@timestamp": {
"gte": "2025-05-16T00:00:00.00000+00:00",
"lt": "2025-05-16T00:10:00.00000+00:00"
}
}
}
}
when executing this query in our elasticsearch i get the following data with 3702 hits reported.
GET /logs-mail.rspamd-default/_search
{
"query": {
"range": {
"@timestamp": {
"gte": "2025-05-16T00:00:00.00000+00:00",
"lt": "2025-05-16T00:10:00.00000+00:00"
}
}
}
}
response:
{
"took": 14,
"timed_out": false,
"_shards": {
"total": 14,
"successful": 14,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3702,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
the same with running the esdump_rs tool:
+ docker run --rm --name export_datastream -v /tmp/query.json:/query.json -v /mnt/<REDACTED>/incoming/20250515:/export -e ES_DUMP_USERNAME=testuser -e 'ES_DUMP_PASSWORD=<REDACTED>' ghcr.io/gitguardian/esdump-rs -c 4 --batch-size=10000 --batches-per-file=40 --index logs-mail.rspamd-default -q /query.json -d elasticsearch https://elastic-node-1<REDACTED>:9200 file:///export/logs-mail.rspamd-default
2025-05-16T06:13:18.486086Z WARN esdump_rs: Error reading env file: path not found
2025-05-16T06:13:18.486350Z INFO esdump_rs: Dumping index logs-mail.rspamd-default https://elastic-node-1<REDACTED>:9200/ to file:///export/logs-mail.rspamd-default
2025-05-16T06:13:18.486364Z INFO esdump_rs: Using None concurrent uploads, with 4 concurrent fetchers
2025-05-16T06:13:18.486370Z INFO esdump_rs: Using query {
"range": {
"@timestamp": {
"gte": "2025-05-16T00:00:00.00000+00:00",
"lt": "2025-05-16T00:10:00.00000+00:00"
}
}
}
2025-05-16T06:13:18.503822Z INFO esdump_rs::elasticsearch: Counted 1863776 records
2025-05-16T06:13:18.503846Z INFO esdump_rs: Got 187 batches to process in 5 chunks
Metadata
Metadata
Assignees
Labels
No labels