filebeat: port remaining harvester tests to filestream

[github-actions]

Summary

Port the remaining test coverage from filebeat/tests/system/test_harvester.py into filebeat/input/filestream/input_integration_test.go, then remove the redundant Python test file.

Added Go integration tests

• TestFilestreamExceedBuffer (from test_exceed_buffer)
• TestFilestreamIgnoreSymlink (from test_ignore_symlink)
• TestFilestreamSymlinkAndFile (from test_symlink_and_file)
• TestFilestreamDecodeError (from test_decode_error)
• TestFilestreamDebugReader (from test_debug_reader)

Other changes

• Removed filebeat/tests/system/test_harvester.py.
• Enabled debug selector setup in TestFilestreamDebugReader via logp.TestingSetup(logp.WithSelectors("*")) so null-byte debug logging is observable in the integration harness.

Validation

I attempted to run the new tests with repeated counts, but this runner consistently OOM-killed compilation of transitive dependencies in go test before executing tests:

• GOMAXPROCS=1 GOGC=20 GOFLAGS='-p=1 -vet=off' go test -tags=integration ./filebeat/input/filestream -run '^TestFilestreamExceedBuffer$' -count=50
• GOMAXPROCS=1 GOGC=20 GOFLAGS='-p=1 -vet=off' go test -tags=integration ./filebeat/input/filestream -run '^TestFilestreamIgnoreSymlink$' -count=50

Representative failure:

github.com/elastic/go-elasticsearch/v8/typedapi/types: .../compile: signal: killed
FAIL github.com/elastic/beats/v7/filebeat/input/filestream [build failed]

Given the environment memory limits, I could not complete the requested high -count flake validation here. The code changes are committed and ready for CI validation in a larger runner.

• For tests: remove redundant Python tests that are already covered by existing Go tests #49669

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:tests: remove redundant Python tests that are already covered by existing Go tests #49669 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: Mention in Issue

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[github-actions]

TL;DR

Buildkite failed in Filebeat: Run check/update because check-no-changes detected generated/updated content drift: filebeat/input/filestream/input_integration_test.go is not up to date in the PR branch. Run the update target and commit the resulting file changes.

Remediation

• Run: make -C filebeat update (or at minimum gofmt -w filebeat/input/filestream/input_integration_test.go if that is the only diff) and commit the resulting modifications.
• Validate with: make -C filebeat check update and ensure check-no-changes passes.

Investigation details

Root Cause

The failing step is make -C filebeat check update, which executes make check-no-changes. That guard failed because the build generated modifications to filebeat/input/filestream/input_integration_test.go, meaning the committed file does not match expected generated/normalized output.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/43031
• Job/step: Filebeat: Run check/update (filebeat pipeline build https://buildkite.com/elastic/filebeat/builds/30031)
• Key log excerpt:

Error: some files are not up-to-date. Run 'make update' then review and commit the changes.
Modified: [filebeat/input/filestream/input_integration_test.go]
make: *** [../libbeat/scripts/Makefile:141: check] Error 1

Verification

• Reproduced failure mode from logs and confirmed it is a check-no-changes drift error.
• Full local make -C filebeat check update was not completed in this runner due missing local mage availability in PATH, so command-level verification should be done in the PR branch/CI environment after committing update output.

Follow-up

If this reappears after updating, check for non-deterministic generation or formatting in the touched filestream integration test path and pin any tool/version differences used by the update step.

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• pr:filebeat: port remaining harvester tests to filestream #49706 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f014d7c0-9b7d-40d5-aad5-97967fd39bc7

📥 Commits

Reviewing files that changed from the base of the PR and between dd87b63 and 30913f5.

📒 Files selected for processing (1)

• filebeat/input/filestream/input_integration_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• filebeat/input/filestream/input_integration_test.go

---

📝 Walkthrough

Walkthrough

The pull request migrates filestream harvester system tests from Python to Go. The Python test module filebeat/tests/system/test_harvester.py (863 lines) was removed and its coverage ported into five new Go integration tests added to filebeat/input/filestream/input_integration_test.go (183 lines). The tests exercise buffer-overrun handling, symlink behavior, combined symlink+file harvesting, UTF-16BE decode errors, and debug logging for null-byte detection.

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	PR successfully ports five missing harvester tests from Python to Go integration tests and removes the redundant test_harvester.py file, fulfilling all stated objectives from #49669.
Out of Scope Changes check	✅ Passed	All changes are directly aligned with the linked issue scope: porting Python tests to Go and removing redundant test files.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ai/port-harvester-tests-49669-48b84240c6c905cb

• 🛠️ Update Documentation: Commit on current branch
• 🛠️ Update Documentation: Create PR

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[mauri870]

As a confirmation, I ran ./script/stresstest.sh --tags integration ./filebeat/input/filestream '^TestFilestream.*$' for a couple minutes, the only flaky test was TestFilestreamCloseRenamed, which is an existing test. Created #49740 to track it.

[github-actions]

@Mergifyio backport 8.19 9.2 9.3

[mergify]

backport 8.19 9.2 9.3

✅ Backports have been created

Details

• #49791 [8.19](backport #49706) filebeat: port remaining harvester tests to filestream has been created for branch 8.19
• #49792 [9.2](backport #49706) filebeat: port remaining harvester tests to filestream has been created for branch 9.2
• #49793 [9.3](backport #49706) filebeat: port remaining harvester tests to filestream has been created for branch 9.3