Custom Play for apache.error message field does not match any data

[joachim-zauner-epcom]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4

RAM

16GB

Storage for /

75

Storage for /nsm

200

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi everyone!

I want a simple play that alerts me every time when the string "ValidateDataException" occures in the error log message field.

My Sigma:

title: PHP Parameter ValidateDataException
description: Whenever a get or post parameter in website project is not in a valid format an php exception called  ValidateDataException is thrown
status: experimental
author: Joachim Zauner, epcom
tags:
- attack.discovery
logsource:
  category: webserver
  service: apache
  definition: 'Requirements: Must be able to collect the error log files'
detection:
  selection:
    message|contains: ValidateDataException
  condition: selection
falsepositives:
- unknown
level: low

This converts to a yaml file with the "eql" filter

- eql: >
   any where message : "*ValidateDataException*"

However, the Problem is that no data is found.
If i search through the Hunt Interface for this text i get results.

I tried to search for the date with Kibana dev console without any hits with

GET /.ds-logs*/_eql/search
{
  "query": """
    any where message : "*ValidateDataException*"
  """
}

and with like:

GET /.ds-logs*/_eql/search
{
  "query": """
    any where message like "*ValidateDataException*"
  """
}

BUT if i do

GET /.ds-logs-*/_search
{
  "query": {
    "match_phrase": {
      "message": "ValidateDataException"
    }
  }
}

i get the hits!

How can i modify my play so i can find the data with the specified substring in the message field?

Thank you, best regards
Joachim

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

What about using keywords condition?

detection:
    keywords:
        - 'ValdateDataException'
    condition: keywords

[joachim-zauner-epcom]

Thank you for the tip.
Unfortunately this leads to the same ElastAlert Config and no data is found:
any where message : "ValidateDataException"

Is it possible that the field message cannot be matched against substrings?
If i try to filter on an other field (agent.name for example) the code works and data is available
any where agent.name : "hosting"

[reyesj2]

I think you just need to add * before and after ValidateDataException in your play.

I created test doc and got hits with wildcards, but not without them.

POST my-index-000001/_doc/
{
  "@timestamp": "2099-11-15T13:12:00",
  "message": "foo ValidateDataException bar"
}

GET /my-index-000001*/_eql/search
{
"query": """
any where message : "*ValidateDataException*"
"""
}

Results

{
  "is_partial": false,
  "is_running": false,
  "took": 8,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "events": [
      {
        "_index": "my-index-000001",
        "_id": "cbMC140BBFergj4cKJ_5",
        "_source": {
          "@timestamp": "2099-11-15T13:12:00",
          "message": "foo ValidateDataException bar"
        }
      }
    ]
  }
}

[joachim-zauner-epcom]

Tried again but without luck

Search for substring

GET /.ds-logs*/_eql/search
{
  "query": """
    any where message : "*ValidateDataException*" 
  """
}

Result

{
  "is_partial": false,
  "is_running": false,
  "took": 1732,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "events": []
  }
}

Search with match_pattern

GET /.ds-logs-*/_search
{
  "query": {
    "match_phrase": {
      "message": "ValidateDataException"
    }
  }
}

**First hit of result with some content removed **

{
  "took": 148,
  "timed_out": false,
  "_shards": {
    "total": 33,
    "successful": 33,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3612,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-logs-apache.error-default-2024.02.15-000001",
        "_id": "vbxl5I0Bs80Wpz3oplHQ",
        "_score": 1,
          "log": {
            "file": {
              "path": "/var/log/apache2/error.log-project.domain.tld"
            },
            "offset": 620751,
            "level": "error"
          },
          "elastic_agent": {
            "id": "4a6c9f9c-f270-4673-8aba-fe12dc9285d8",
            "version": "8.10.4",
            "snapshot": false
          },
          "message": "AH01071: Got error 'PHP message: [audit-warn] [MSG-VD01] Type mismatch at addreserv [Description]: ValidateDataException at URI [/sample/uri]', referer: https://project.domain.tld/",
          "tags": [
            "apache-error",
            "elastic-agent",
            "input-so-man",
            "beats_input_codec_plain_applied",
            "error"
          ],
          "input": {
            "type": "log"
          },
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "apache.error"
          }
        }
      },
      -- removed hits --
    ]
  }
}

[joachim-zauner-epcom]

Is it possible that there is a problem with the field definition?
GET /.ds-logs-*/_field_caps?fields=message
shows the following.

{
  "indices": [
    ".ds-logs-apache.access-default-2024.02.23-000001",
    ".ds-logs-apache.error-default-2024.02.15-000001",
    ".ds-logs-auditd.log-default-2024.02.15-000001",
    ".ds-logs-elastic_agent-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.endpoint_security-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.filebeat-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.fleet_server-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.metricbeat-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.osquerybeat-default-2024.02.15-000001",
    ".ds-logs-elasticsearch.server-default-2024.02.15-000001",
    ".ds-logs-endpoint.events.api-default-2024.02.19-000001",
    ".ds-logs-endpoint.events.file-default-2024.02.15-000001",
    ".ds-logs-endpoint.events.library-default-2024.02.19-000001",
    ".ds-logs-endpoint.events.network-default-2024.02.15-000001",
    ".ds-logs-endpoint.events.process-default-2024.02.15-000001",
    ".ds-logs-endpoint.events.registry-default-2024.02.19-000001",
    ".ds-logs-endpoint.events.security-default-2024.02.19-000001",
    ".ds-logs-kratos-so-2024.02.15-000001",
    ".ds-logs-osquery_manager.result-default-2024.02.19-000001",
    ".ds-logs-pfsense.log-default-2024.02.16-000001",
    ".ds-logs-playbook.alerts-so-2024.02.15-000001",
    ".ds-logs-soc-so-2024.02.15-000001",
    ".ds-logs-strelka-so-2024.02.15-000001",
    ".ds-logs-suricata-so-2024.02.15-000001",
    ".ds-logs-system.application-default-2024.02.19-000001",
    ".ds-logs-system.auth-default-2024.02.15-000001",
    ".ds-logs-system.security-default-2024.02.19-000001",
    ".ds-logs-system.syslog-default-2024.02.15-000001",
    ".ds-logs-system.system-default-2024.02.19-000001",
    ".ds-logs-windows.powershell-default-2024.02.19-000001",
    ".ds-logs-windows.powershell_operational-default-2024.02.19-000001",
    ".ds-logs-zeek-so-2024.02.15-000001"
  ],
  "fields": {
    "message": {
      "text": {
        "type": "text",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    }
  }
}

When i query the field agent.name for example - with this field substrings in same format can be searched
GET /.ds-logs-*/_field_caps?fields=agent.name
i get

{
  "indices": [
    ".ds-logs-apache.access-default-2024.02.23-000001",
    ".ds-logs-apache.error-default-2024.02.15-000001",
    ".ds-logs-auditd.log-default-2024.02.15-000001",
    ".ds-logs-elastic_agent-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.endpoint_security-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.filebeat-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.fleet_server-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.metricbeat-default-2024.02.15-000001",
    ".ds-logs-elastic_agent.osquerybeat-default-2024.02.15-000001",
    ".ds-logs-elasticsearch.server-default-2024.02.15-000001",
    ".ds-logs-endpoint.events.api-default-2024.02.19-000001",
    ".ds-logs-endpoint.events.file-default-2024.02.15-000001",
    ".ds-logs-endpoint.events.library-default-2024.02.19-000001",
    ".ds-logs-endpoint.events.network-default-2024.02.15-000001",
    ".ds-logs-endpoint.events.process-default-2024.02.15-000001",
    ".ds-logs-endpoint.events.registry-default-2024.02.19-000001",
    ".ds-logs-endpoint.events.security-default-2024.02.19-000001",
    ".ds-logs-kratos-so-2024.02.15-000001",
    ".ds-logs-osquery_manager.result-default-2024.02.19-000001",
    ".ds-logs-pfsense.log-default-2024.02.16-000001",
    ".ds-logs-playbook.alerts-so-2024.02.15-000001",
    ".ds-logs-soc-so-2024.02.15-000001",
    ".ds-logs-strelka-so-2024.02.15-000001",
    ".ds-logs-suricata-so-2024.02.15-000001",
    ".ds-logs-system.application-default-2024.02.19-000001",
    ".ds-logs-system.auth-default-2024.02.15-000001",
    ".ds-logs-system.security-default-2024.02.19-000001",
    ".ds-logs-system.syslog-default-2024.02.15-000001",
    ".ds-logs-system.system-default-2024.02.19-000001",
    ".ds-logs-windows.powershell-default-2024.02.19-000001",
    ".ds-logs-windows.powershell_operational-default-2024.02.19-000001",
    ".ds-logs-zeek-so-2024.02.15-000001"
  ],
  "fields": {
    "agent": {
      "object": {
        "type": "object",
        "metadata_field": false,
        "searchable": false,
        "aggregatable": false
      }
    },
    "agent.name": {
      "keyword": {
        "type": "keyword",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": true
      }
    }
  }
}

so the field message is of type text and agent.name is of type keyword. Maybe this is the problem?