[Streams] Add new ingest pipeline field access flag

[jbaiera]

This PR introduces a new flag to ingest pipeline configurations which will be used to control how fields are accessed from within that pipeline.

The field_access_pattern config option is used to denote which strategy to use for resolving field names in ingest pipelines:

{
  "description": "My pipeline description",
  "field_access_pattern": "flexible",
  "processors": [...]
}

This config accepts two values currently:

• classic - the default value and current behavior
• flexible - defines the new field access behavior that will allow for dotted field name access

The configuration requires a new node feature to be present on all nodes in the cluster in order to be accepted. This node feature is only added to the cluster if the streams feature flag is enabled.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[Copilot]

Pull Request Overview

This PR adds a new field_access_pattern option to ingest pipeline configs, introduces the corresponding enum and node feature, and propagates the flag through pipeline creation and execution.

• Extend Pipeline.create and constructors to accept a hasFeature predicate and the new fieldAccessPattern
• Introduce IngestPipelineFieldAccessPattern enum and register it via IngestFeatures/FeatureSpecification
• Update all call sites, transport actions, and tests to pass the feature predicate and handle the new flag

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/common/validation/SourceDestValidatorTests.java	Updated Pipeline.create invocation to include the new feature predicate.
server/src/test/java/org/elasticsearch/ingest/PipelineProcessorTests.java	Adjusted test pipelines to include IngestPipelineFieldAccessPattern.CLASSIC.
server/src/test/java/org/elasticsearch/ingest/PipelineFactoryTests.java	Expanded Pipeline.create calls and added tests for unsupported patterns.
server/src/test/java/org/elasticsearch/action/ingest/SimulatePipelineRequestParsingTests.java	Updated parse calls to accept a feature predicate lambda.
server/src/main/resources/META-INF/services/org.elasticsearch.features.FeatureSpecification	Registered the new IngestFeatures spec to expose the node feature.
server/src/main/java/org/elasticsearch/ingest/TrackingResultProcessor.java	Propagated fieldAccessPattern into the pipeline decoration.
server/src/main/java/org/elasticsearch/ingest/SimulateIngestService.java	Passed the feature‐checking lambda into substitution pipeline creation.
server/src/main/java/org/elasticsearch/ingest/Pipeline.java	Added FIELD_ACCESS_PATTERN constant, field, enum handling in create(), and a getter.
server/src/main/java/org/elasticsearch/ingest/IngestService.java	Declared the FIELD_ACCESS_PATTERN node feature and local support check.
server/src/main/java/org/elasticsearch/ingest/IngestPipelineFieldAccessPattern.java	Introduced enum to represent classic vs. flexible access patterns.
server/src/main/java/org/elasticsearch/ingest/IngestFeatures.java	Implemented FeatureSpecification to advertise the feature.
server/src/main/java/org/elasticsearch/action/ingest/SimulatePipelineTransportAction.java	Updated transport action to supply the feature predicate to parsing.
server/src/main/java/org/elasticsearch/action/ingest/SimulatePipelineRequest.java	Modified parse() signature to take a Predicate<NodeFeature>.
server/src/main/java/org/elasticsearch/action/ingest/SimulateExecutionService.java	Propagated fieldAccessPattern into simulated pipeline execution.
server/src/main/java/module-info.java	Added org.elasticsearch.ingest.IngestFeatures to the module declarations.

Comments suppressed due to low confidence (2)

server/src/main/java/org/elasticsearch/ingest/Pipeline.java:30

• [nitpick] Update the class-level Javadoc for Pipeline to mention the new field_access_pattern configuration option and its effect, and document the getFieldAccessPattern() method so that users understand its purpose.

public final class Pipeline {

server/src/main/java/org/elasticsearch/ingest/TrackingResultProcessor.java:219

• Add a unit test for TrackingResultProcessor verifying that the injected fieldAccessPattern is correctly passed through and applied during pipeline execution.

pipeline.getFieldAccessPattern(),

[lukewhiting]

Clean, tidy & super easy to follow. LGTM 👍🏻

[masseyke]

Very minor thing, but it's the node that doesn't support the parameter, not the pipeline, right? Also, it's always one, never "or more" since it's a hard-coded list of one right?

[masseyke]

Looks good to me. I assume rest tests, documentation, and rest-spec changes (and probably other things I'm forgetting) are all coming once the new field does what it's supposed to do.

[masseyke]

Oh, and it needs the 8.19.0 label, right?