[Required Change] Update Elastic Security Requirements - Third-party collectors mapped to ECS #2934
Description
What can we change to make the docs better?
Some users may expect that they can use third-party OTel collectors to ship data to Elastic Security and have all functions work properly. This is not a supported configuration for the reasons stated below. Although our documentation already makes it clear that ECS is required for Elastic Security, we need to add an explicit mention of third-party OTel collectors not being supported.
Suggestions:
Keep current text in Third-party collectors mapped to ECS
Add new section titled "Third-party collectors not mapped to ECS"
The use of third party connectors that do not map data to ECS, including third-party and open source OTel Collectors, is not currently supported by Elastic Security.
BACKGROUND INFORMATION
(from public information available to Claude.ai about a recent Elastic Customer issue)
QUESTION: For users of Elastic SIEM, which requires Elastic Common Schema (ECS), would data shipped by the SumoLogic Windows OTel collector work properly in detection rules, dashboards, queries, and other downstream workflows?
ANSWER: No, data shipped by the SumoLogic Windows OTel collector would not work properly in Elastic SIEM without significant transformation, due to fundamental schema incompatibilities between OpenTelemetry and Elastic Common Schema (ECS).
Schema Incompatibility Issues
Field Naming Conflicts:
- OTel uses attributes.host.name vs ECS uses host.name
- OTel severity vs ECS log.level
- OTel body vs ECS message
Structural Differences:
- OTel nests Windows-specific fields under winlog.* namespace
- ECS has more granular field categorization (event., process., user.*, etc.)
- ECS requires specific field types and formats that OTel may not provide
Specific Impact on Elastic SIEM Components
Detection Rules:
- Pre-built Elastic detection rules expect ECS field names
- Custom KQL/DSL queries would fail due to missing expected fields
- Timeline investigations rely on ECS event categorization
Dashboards:
- Visualizations targeting ECS fields like event.category, event.action would show no data
- Host and user drill-downs expect host.name, user.name in ECS format
- Security metrics dashboards would be broken
Machine Learning Jobs:
- Anomaly detection models trained on ECS field patterns wouldn't function
- Entity analytics expecting ECS user/host normalization would fail
Potential Solutions
Data Transformation Required:
- Logstash Pipeline - Transform OTel format to ECS during ingestion
- Ingest Pipelines - Use Elasticsearch ingest processors for field mapping
- Custom Processors - Modify OTel collector configuration for ECS output
- Alternative Approach: Use Elastic's own Elastic Agent with the Windows integration, which natively produces ECS-compliant data for optimal SIEM functionality.
Summary
While technically possible to make it work through extensive transformation, using SumoLogic's OTel collector data in Elastic SIEM would require significant engineering effort and ongoing maintenance. The schemas serve different ecosystems with different priorities - OTel focuses on vendor neutrality while ECS optimizes for Elastic's security analytics capabilities.
For production Elastic SIEM deployments, native Elastic Agent collection would be the recommended approach for reliability and full feature compatibility.
Doc URL
Please include the doc URL and any other related information where applicable:
Doc URL: https://www.elastic.co/docs/solutions/security/get-started/elastic-security-requirements#security-requirements-overview-third-party-collectors-mapped-to-ecs
Which documentation set needs improvement?
ESS and serverless
Software version
This change SHOULD be made as soon as possible, and MUST be made before the 9.2 stack release.