x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector by efd6 · Pull Request #49614 · elastic/beats

efd6 · 2026-03-23T22:41:16Z

Proposed commit message

x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector

Implement the Redirector interface on httpjson's InputManager so
that inputs with run_as_cel: true and a cel.program are transparently
redirected to the cel input type.

convertHttpjsonToCel translates shared fields (interval, URL, auth,
retry, redirect, keep_alive, tracer, TLS, proxy, timeout), cel-
namespaced fields (program, max_executions, state, regexp, xsd,
redact), and passthrough fields (id) into a cel input config.

migrateCursor reads the httpjson cursor from the persistent store
during Redirect and injects it into the translated config's
state.cursor so that the cel input continues from where httpjson
left off. Direct store writes are not viable because the Loader
initialises all plugin managers' in-memory state tables before any
Configure call where Redirect runs. The httpjson store entry is
preserved for rollback safety.

Note

Best reviewed commitwise.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

Author's Checklist

[ ]

How to test this PR locally

Related issues

Depends filebeat/input/{v2,logv2}: add Redirector interface for general input migration #49613

Use cases

Screenshots

Logs

github-actions · 2026-03-23T22:41:27Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

github-actions · 2026-03-23T22:42:07Z

Vale Linting Results

Summary: 2 suggestions found

💡 Suggestions (2)

File	Line	Rule	Message
docs/reference/filebeat/filebeat-input-httpjson.md	1772	Elastic.WordChoice	Consider using 'run, start' instead of 'execute', unless the term is in the UI.
docs/reference/filebeat/filebeat-input-httpjson.md	1781	Elastic.WordChoice	Consider using 'can, might' instead of 'May', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

github-actions · 2026-03-23T22:43:16Z

🔍 Preview links for changed docs

docs/reference/filebeat/filebeat-input-httpjson.md

efd6 · 2026-03-23T22:56:29Z

re: vale linter

Spelling changed to ASE.
"execute" retained as it is conventional language for the input.
The suggestion is semantically incorrect; the sense here is permission, not capacity or possibility.

Implement the Redirector interface on httpjson's InputManager so that when run_as_cel is true and cel.program is present, the Loader redirects to the cel input with a translated config. convertHttpjsonToCel maps the full set of shared fields between the two inputs: interval, auth, request.url to resource.url, retry, redirect, keep_alive, tracer, and inline transport settings (ssl, timeout, proxy_url, proxy_headers, proxy_disable, idle_connection_timeout). cel-namespaced fields (program, state, regexp, xsd, max_executions, redact) are extracted to their top-level equivalents. Fields specific to httpjson's declarative config (method, body, transforms, response, cursor, chain) are excluded since the CEL program handles them directly. max_executions uses Int/SetInt to match its *int type in the cel config rather than round-tripping through strings.

Read the httpjson cursor from the persistent store during Redirect and inject it into the translated cel config's state.cursor field. The cel input uses config.State["cursor"] as the initial value when no stored cursor exists, so the first run after redirect continues from where httpjson left off. Subsequent runs use the cel-stored cursor and ignore the injected value. Direct store writes are not viable because the Loader calls Init on all plugin managers (loading the cel cursor manager's in-memory table) before any Configure call where Redirect runs. A cel:: key written to the persistent store during Configure would not appear in the in-memory table, and findOrCreate would overwrite it with an empty resource on first flush. The httpjson store entry is preserved for rollback safety.

Add reference documentation for the run_as_cel option and the cel.program, cel.state, cel.max_executions, cel.regexp, cel.xsd, and cel.redact configuration fields. Includes a field mapping table, a note on cursor state migration, and a worked example.

elasticmachine · 2026-03-31T22:23:32Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine · 2026-03-31T22:23:34Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

coderabbitai · 2026-03-31T22:28:45Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1f4d6877-8de8-401c-9932-76d6df89d982

📥 Commits

Reviewing files that changed from the base of the PR and between f5b8809 and 649e859.

📒 Files selected for processing (2)

x-pack/filebeat/input/httpjson/redirect.go
x-pack/filebeat/input/httpjson/redirect_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

x-pack/filebeat/input/httpjson/redirect.go

📝 Walkthrough

Walkthrough

Adds a redirect mechanism that converts httpjson inputs into cel inputs when run_as_cel is true and cel.program is provided. The change implements a Redirect method that translates shared configuration fields, lifts cel.* options into the cel config, and migrates persisted httpjson cursor state into the resulting cel state.cursor. Documentation and a changelog fragment for the new options were added, and comprehensive unit tests cover configuration translation, cursor migration, and error cases.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

🛠️ Update Documentation: Commit on current branch
🛠️ Update Documentation: Create PR

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

kcreddy · 2026-04-01T13:19:17Z

x-pack/filebeat/input/httpjson/redirect.go

+		}
+	}
+	if err := dst.SetChild("state.cursor", -1, cursorCfg); err != nil {
+		m.cursor.Logger.Warnw("cursor migration: cannot inject cursor into config", "error", err)


Since these are warnings and not erros, they wouldn't DEGRADE the input, correct?
So when this warnings occur, does it start with a default cursor with cel input, or previous cursor with httpjson?

No, there is no degradation. The warnings use m.cursor.Logger.Warnw, not UpdateStatus(status.Degraded, …), so the input health stays healthy.

If cursor migration fails, the cel input falls back to config.State["cursor"] which is whatever the user put in cel.state.cursor (the bootstrap default). It doesn't fall back to the httpjson cursor, but starts fresh from the config default. This is safe because blocking startup would be worse than a re-fetch.

This is safe because blocking startup would be worse than a re-fetch.

Got it. In the worst case scenario if a re-fetch is attempted, we may have duplicates. Do you think we should make it mandatory to have fingerprint processors inside the ingest pipeline for all run_as_cel: true cases?

x-pack/filebeat/input/httpjson/redirect.go

x-pack/filebeat/input/httpjson/redirect_test.go

Parse the URL through url.Parse before constructing the cursor store key so that the key matches source.Name() normalization. Without this, URL round-tripping through net/url could cause a mismatch between the key we look up and the key the input wrote. Add input.Name() assertion to TestRedirect_EndToEnd to verify the Loader actually redirected to the cel plugin.

kcreddy

Changes LGTM, just a minor follow-up question.

efd6 self-assigned this Mar 23, 2026

efd6 added enhancement Filebeat Filebeat backport-skip Skip notification from the automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:Security-Service Integrations Security Service Integrations Team labels Mar 23, 2026

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 23, 2026

github-actions bot deployed to docs-preview March 23, 2026 22:41 View deployment

efd6 force-pushed the id6318-director-httpjson branch from 4cbe48d to 78c339c Compare March 23, 2026 22:48

github-actions bot deployed to docs-preview March 23, 2026 22:48 View deployment

efd6 force-pushed the id6318-director-httpjson branch from 78c339c to 118dc97 Compare March 23, 2026 23:08

github-actions bot deployed to docs-preview March 23, 2026 23:09 View deployment

efd6 force-pushed the id6318-director-httpjson branch from 118dc97 to cdb29b4 Compare March 23, 2026 23:20

github-actions bot deployed to docs-preview March 23, 2026 23:21 View deployment

efd6 force-pushed the id6318-director-httpjson branch from cdb29b4 to f7641f4 Compare March 24, 2026 00:42

github-actions bot deployed to docs-preview March 24, 2026 00:43 View deployment

efd6 force-pushed the id6318-director-httpjson branch from f7641f4 to d2a44e8 Compare March 24, 2026 00:53

github-actions bot deployed to docs-preview March 24, 2026 00:54 View deployment

efd6 mentioned this pull request Mar 24, 2026

filebeat/input/{v2,logv2}: add Redirector interface for general input migration #49613

Merged

6 tasks

efd6 added 3 commits April 1, 2026 08:53

efd6 force-pushed the id6318-director-httpjson branch from d2a44e8 to f5b8809 Compare March 31, 2026 22:23

efd6 marked this pull request as ready for review March 31, 2026 22:23

efd6 requested a review from a team as a code owner March 31, 2026 22:23

theletterf approved these changes Apr 1, 2026

View reviewed changes

kcreddy reviewed Apr 1, 2026

View reviewed changes

efd6 requested a review from kcreddy April 1, 2026 19:47

kcreddy approved these changes Apr 2, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector#49614

x-pack/filebeat/input/httpjson: add httpjson-to-cel Redirector#49614
efd6 wants to merge 4 commits intoelastic:mainfrom
efd6:id6318-director-httpjson

efd6 commented Mar 23, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 23, 2026

Uh oh!

github-actions bot commented Mar 23, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 23, 2026 •

edited

Loading

Uh oh!

efd6 commented Mar 23, 2026 •

edited

Loading

Uh oh!

elasticmachine commented Mar 31, 2026

Uh oh!

elasticmachine commented Mar 31, 2026

Uh oh!

coderabbitai bot commented Mar 31, 2026 •

edited

Loading

Walkthrough

Uh oh!

kcreddy Apr 1, 2026 •

edited

Loading

Uh oh!

efd6 Apr 1, 2026

Uh oh!

kcreddy Apr 2, 2026

Uh oh!

Uh oh!

Uh oh!

kcreddy left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

efd6 commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Disruptive User Impact

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

github-actions bot commented Mar 23, 2026

🤖 GitHub comments

Uh oh!

github-actions bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Vale Linting Results

Uh oh!

github-actions bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

efd6 commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Mar 31, 2026

Uh oh!

elasticmachine commented Mar 31, 2026

Uh oh!

coderabbitai bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Uh oh!

kcreddy Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

kcreddy Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

efd6 commented Mar 23, 2026 •

edited

Loading

github-actions bot commented Mar 23, 2026 •

edited

Loading

github-actions bot commented Mar 23, 2026 •

edited

Loading

efd6 commented Mar 23, 2026 •

edited

Loading

coderabbitai bot commented Mar 31, 2026 •

edited

Loading

kcreddy Apr 1, 2026 •

edited

Loading