Skip to content

howto use query parameter #44

@gweiss76

Description

@gweiss76

Hi.
Thanks for writing this tool.

Unfortunately I'm not able to figure out how the query parameter is working. It seems not to be used at all. See below:

I'm trying to use this tool to dump parts of our datastreams by using this query (in file /query.json)

{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2025-05-16T00:00:00.00000+00:00",
        "lt": "2025-05-16T00:10:00.00000+00:00"
      }
    }
  }
} 

when executing this query in our elasticsearch i get the following data with 3702 hits reported.

GET /logs-mail.rspamd-default/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2025-05-16T00:00:00.00000+00:00",
        "lt": "2025-05-16T00:10:00.00000+00:00"
      }
    }
  }
}

response:
{
  "took": 14,
  "timed_out": false,
  "_shards": {
    "total": 14,
    "successful": 14,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3702,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {

the same with running the esdump_rs tool:

+ docker run --rm --name export_datastream -v /tmp/query.json:/query.json -v /mnt/<REDACTED>/incoming/20250515:/export -e ES_DUMP_USERNAME=testuser -e 'ES_DUMP_PASSWORD=<REDACTED>' ghcr.io/gitguardian/esdump-rs -c 4 --batch-size=10000 --batches-per-file=40 --index logs-mail.rspamd-default -q /query.json -d elasticsearch https://elastic-node-1<REDACTED>:9200 file:///export/logs-mail.rspamd-default
2025-05-16T06:13:18.486086Z  WARN esdump_rs: Error reading env file: path not found
2025-05-16T06:13:18.486350Z  INFO esdump_rs: Dumping index logs-mail.rspamd-default https://elastic-node-1<REDACTED>:9200/ to file:///export/logs-mail.rspamd-default
2025-05-16T06:13:18.486364Z  INFO esdump_rs: Using None concurrent uploads, with 4 concurrent fetchers
2025-05-16T06:13:18.486370Z  INFO esdump_rs: Using query {
  "range": {
    "@timestamp": {
      "gte": "2025-05-16T00:00:00.00000+00:00",
      "lt": "2025-05-16T00:10:00.00000+00:00"
    }
  }
}
2025-05-16T06:13:18.503822Z  INFO esdump_rs::elasticsearch: Counted 1863776 records
2025-05-16T06:13:18.503846Z  INFO esdump_rs: Got 187 batches to process in 5 chunks

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions