Solving Elasticsearch 8.x Query Failures with Sigma and Elastalert

[m-aouzal]

Background: The Challenge

Elasticsearch 8.x introduced updates to its analyzer and mapping mechanisms, which impact how text fields are indexed and queried. Without proper handling, this mismatch can lead to query failures, especially in pipelines like ecs_windows, where field naming and text processing are critical.

---

Key Issues and Solutions

To address these challenges, I developed a robust, three-pronged approach:

1. Lowercasing Query Values

• Problem: By default, Elasticsearch keyword fields are case-sensitive. For example, a Sigma rule searching for Process won’t match process in the index.
• Solution: I modified the query generation process to lowercase all query values. This ensures consistency, aligning with Sigma’s case-insensitive design.
• Impact: Queries now reliably match data regardless of casing variations, a common issue in Windows event logs.

2. Implementing a Normalizer in Kibana

• Problem: Indexed data may retain mixed casing, leading to mismatches with lowercase queries.
• Solution: I applied a normalizer to keyword fields in Kibana, enforcing lowercase storage at index time. For example, a field mapping might look like:

  {
    "properties": {
      "event.action": {
        "type": "keyword",
        "normalizer": {
          "type": "custom",
          "filter": ["lowercase"]
        }
      }
    }
  }

• Impact: This creates uniformity between indexed data and queries, critical for large-scale or legacy datasets.

3. Appending .keyword Suffix

• Problem: In Elasticsearch 8.x, text fields are analyzed by default, breaking exact-match queries (e.g., wildcards or regex). The unanalyzed version requires the .keyword suffix (e.g., event.action.keyword).
• Solution: My conversion script automatically appends .keyword to relevant ECS (Elastic Common Schema) fields, with logic to exclude non-text fields like timestamps.
• Impact: This ensures compatibility with modern Elasticsearch mappings, supporting complex Sigma rules with wildcards (e.g., file.name:*.exe).

---

Validation and Resources

This approach has been rigorously tested with diverse datasets, including Windows logs, and supports advanced queries like regex and wildcards. For a detailed implementation, refer to my repository: sigma_to_elastalert README.

---

Conclusion

By lowercasing queries, normalizing data, and adapting to Elasticsearch’s .keyword convention, this solution bridges the gap between Sigma’s flexibility and Elasticsearch 8.x’s precision. I welcome feedback from the community on alternative strategies or enhancements!