[Feature Request] Implement field_mapping mapper type for pull-based ingestion #20728
Description
Is your feature request related to a problem? Please describe
Implements the field_mapping mapper type introduced in #20722, enabling pull-based ingestion to extract document metadata from configurable fields in raw stream messages.
Mapper settings
| Setting | Description |
|---|---|
id_field |
Source field to use as document _id. Auto-generated if absent. |
version_field |
Source field to use as document _version (external versioning). |
op_type_field |
Source field to determine operation type (index vs delete). |
op_type_field.delete_value |
Optional. The value that indicates a delete. If not set, field is parsed as boolean. |
Describe the solution you'd like
Example
PUT /my-index
{
"settings": {
"ingestion_source": {
"type": "kafka",
"mapper_type": "field_mapping",
"mapper_settings.id_field": "user_id",
"mapper_settings.version_field": "timestamp",
"mapper_settings.op_type_field": "is_deleted"
}
}
}Raw message {"user_id": "abc", "timestamp": 123, "is_deleted": false, "name": "alice"} gets mapped to {_id=abc, _version=123, _op_type=index, _source={"name": "alice"}.
Related component
No response
Describe alternatives you've considered
No response
Additional context
No response