ES-11331 streams params restriction

[lukewhiting]

This pull request introduces endpoint parameter restrictions for streams, specifically targeting the logs stream, and enforces allowed parameters for bulk and index operations based on the API designs of Streams.

New YAML rest suite added to verify these restrictions.

[lukewhiting]

Originally we were going to create a new REST action for logs streams that did it's checks and then delegated back down to the relevant existing REST action.

I'm not 100% happy with either approach TBH... I'm loath to have streams leak out across the codebase but I also don't want the maintenance headache of unnecessary duplication as the REST framework isn't set up for that kind of delegation and relies on a lot on interface methods to provide configuration that end up duplicated on the new endpoint.

On balance, I think just doing it this way is the lesser of two evils for and there is precedent for other features doing it this way however that opinion might change if we added any additional streams logic to the indexing endpoints.

Comments and differing opinions most welcome 😃

[elasticsearchmachine]

Hi @lukewhiting, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[Copilot]

Pull Request Overview

This PR introduces endpoint parameter restrictions for streams, specifically targeting the logs stream, to ensure only allowed parameters are used in bulk and index operations according to streams API design specifications.

• Adds validation to restrict parameters when writing to streams, allowing only index, op_type, error_trace, and timeout
• Implements parameter validation in both bulk and single document operations for streams
• Adds comprehensive test coverage via YAML REST tests to verify these restrictions

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
RestIndexAction.java	Added constructor parameters and stream parameter validation logic
RestBulkAction.java	Defined allowed stream parameters and added validation tracking
TransportAbstractBulkAction.java	Refactored stream validation into dedicated method
BulkRequest.java	Added field to track restricted parameter usage with transport version support
IncrementalBulkService.java	Updated to propagate restricted parameter usage flag
ActionModule.java	Updated handler registrations to include required dependencies
TransportVersions.java	Added new transport version for stream parameter restrictions
30_param_restrictions.yml	Added comprehensive test cases for parameter restriction validation
20_substream_restrictions.yml	Minor test optimization to avoid refresh parameter
132967.yaml	Added changelog entry

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[masseyke]

Two things about this:
(1) It would be useful if it also printed out which parameters were passed, because I had to debug in to see why my request had failed
(2) I think it is accidentally a little too restrictive. If I run this command:

POST logs/_doc
{
  "@timestamp": "2099-11-15T13:12:00",
  "message": "GET /search HTTP/1.1 200 1070000",
  "user": {
    "id": "kimchy"
  }
}

then I get rejected. Through debugging, I discovered that that is implicitly adding pretty as a parameter. I think we ought to allow that. If I use the form of that API that passes in an ID, that also gets blocked by this. Maybe we want to prevent users from providing their own IDs, but you can still pas an ID in the bulk API, so that seems inconsistent.

[masseyke]

Rethinking my last comment, I think there are 3 things that need to be done:
(1) List the request params in the error message
(2) Add pretty and _id to the whitelisted params
(3) Decide if we want to prevent users from setting the ID, and if so do it at a lower level so that it's consistent

[lukewhiting]

Thanks @masseyke :-) I have refactored this to pass the list of param names used so we get better error messages. This is at the expense of additional traffic over the wire on each bulk request line.

I also white listed pretty and id as allowed but will follow up with product when they are back on Monday if they want ID or not.

[lukewhiting]

Product have agreed that ID should not be blocked for now so I think we're good to go on this PR from that perspective

[masseyke]

Just to make sure we're explicit, here's where it currently stands:

Param	Allowed
error_trace	✅
filter_path	✅
id	✅
include_source_on_error boolean	❌
index	✅
list_executed_pipelines	❌
op_type	✅
pipeline	❌
pretty	✅
refresh	✅
require_alias	❌
require_data_stream	❌
routing	❌
_source	❌
_source_excludes	❌
_source_includes	❌
timeout	✅
wait_for_active_shards	❌

Some of those seem kind of useful -- list_executed_pipelines for example.

EDIT: Updated 2025-08-24@13:47 by LW to include addition of filter_path and refresh

[masseyke]

LGTM. The Observability team is currently setting the refresh and filter_path params when they are loading to a stream, so we might need to whitelist a few more params at some point.