Play around with E2E tests to improve. #377

mashhurs · 2025-11-20T21:32:11Z

Description

This change suggestions:

Better visibility for the entire stack by printing docker container [LS, ES, elastic-agent] logs
it was found that with 9.3.0-SNAPSHOT stack version, elastic-agent doesn't use SSL params. Introduced a separate config for such cases
increases resource/VM-spec to make sure all stack can spin up
fixes flaky unit test

Test CI: https://buildkite.com/elastic/logstash-filter-elastic-integration-e2e/builds/250#019aa917-7cd2-42bd-9fe9-7a44e840ae53

mergify · 2025-11-20T21:32:48Z

This pull request does not have a backport label. Could you fix it @mashhurs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

.buildkite/scripts/e2e-pipeline/bootstrap.py

mashhurs · 2025-11-21T23:31:57Z

...java/co/elastic/logstash/filters/elasticintegration/ElasticsearchRestClientWireMockTest.java

-                assertThat(ex.getMessage(), stringContainsInOrder("fatal", "bad_certificate"));
+                assertThat(ex.getMessage(), allOf(
+                    containsString("fatal"),
+                    anyOf(containsString("bad_certificate"), containsString("certificate_required"))


In some hosts (mostly with recent builds), this test fails with the below error, looks like a JVM specific.
I can't reproduce it on my local (my JVM always uses build in certificates) but taking a look at RFC-8446, the unit test is not sending any certificates, so certificate_required looks a correct validation here.

The CI failure link: https://buildkite.com/elastic/logstash-filter-elastic-integration-pull-request/builds/659#019aa856-f319-4ae0-9c3d-4eba70714a6c/74-1400

The error log

logstash-1 \| Test testBasicConnectivityDisablingVerification() FAILED -- logstash-1 \| logstash-1 \| java.lang.AssertionError: logstash-1 \| Expected: a string containing "fatal", "bad_certificate" in order logstash-1 \| but: was "(certificate_required) Received fatal alert: certificate_required" logstash-1 \| at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) logstash-1 \| at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) logstash-1 \| at co.elastic.logstash.filters.elasticintegration.ElasticsearchRestClientWireMockTest$MutualHttps.testBasicConnectivityDisablingVerification(ElasticsearchRestClientWireMockTest.java:178) logstash-1 \| at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) logstash-1 \| at java.base/java.lang.reflect.Method.invoke(Method.java:580)

…tic_agent without setting SSL. With this change, Logstash is also opting out SSL settings for the versions greater than 9.3

mashhurs · 2025-11-22T01:00:10Z

.buildkite/scripts/e2e-pipeline/config/pipeline-ea-without-ssl.conf

+input {
+  elastic_agent {
+    port => 5044
+  }


I have found that elastic-package defining greater than 9.3 stack version is not using SSL params in elastic-agent. So, Logstash is getting Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: error, CI link: https://buildkite.com/elastic/logstash-filter-elastic-integration-e2e/builds/249#019aa897-ce17-466d-b457-d5a9334aa87b

I have pinged @bhapas to clarify if this is an intention.

@mashhurs Not sure since I was not part of this change..

mashhurs · 2025-11-22T01:02:33Z

.buildkite/scripts/e2e-pipeline/generate-steps.py

        "agents": {
            "provider": "gcp",
-            "machineType": "n2-standard-4",
+            "machineType": "n2-standard-16",


Sometimes ES is not reachable, when I spin up VM and tried many times, it doesn't response. It looks like elastic-package fires up so much resource that ES is very slow. Changing the machine type and decreasing number of packages (only m365_defender which has more processors) to test.

mashhurs · 2025-11-22T01:03:35Z

.buildkite/scripts/e2e-pipeline/plugin_test.py

                for result_line in result.stdout.splitlines(): print(f"{result_line}")

        # although there was an error, le's check how LS performed and make sure errors weren't because of Logstash
+        time.sleep(2) # make sure LS processes the event way to downstream ES


This happened mostly when ES is very slow or irresponsive but in case to make sure events are processed, leaving some window before hitting _node/stats API

mashhurs · 2025-11-22T01:04:11Z

.buildkite/scripts/e2e-pipeline/util.py

+        container_logs = container.logs().decode('utf-8')
+        for log_line in container_logs.splitlines():
+            print(f"  {log_line}")
+        print(f"{separator}\n")


This gives us better visibility what happened with entire stack.

donoghuc · 2025-11-25T22:27:15Z

.buildkite/scripts/e2e-pipeline/main.py

 import util

-INTEGRATION_PACKAGES_TO_TEST = ["apache", "m365_defender", "nginx", "tomcat"]
+INTEGRATION_PACKAGES_TO_TEST = ["m365_defender"]


Now that we are using a more capable VM are we adding these back?

I am thinking to opt out them as they do not provide meaningful values to the test cases. m365_defender has more complex conditions and more processors which, I consider, is the best candidate (or TI integration) for the test. nginx, apache & tomcat are simple integrations with countable processors.
If you have strong opinion I am happy to rollback (or at least add one/couple of them).

donoghuc · 2025-11-25T22:28:13Z

.buildkite/scripts/e2e-pipeline/bootstrap.py

        local_config_file_path = ".buildkite/scripts/e2e-pipeline/config/"
-        config_file = "serverless_pipeline.conf" if self.project_type == "serverless" else "pipeline.conf"
+        if self.__is_version_gte(self.stack_version, 9, 3):
+            config_file = "pipeline-ea-without-ssl.conf"


I cant think of a reason to special case this. I think we should figure out why this all the sudden does not use ssl rather than letting it use unencrypted.

Yup, let's wait a bit to get a confirmation (or the plan for their fix). I would like to make sure the pipeline gets 🟢. I am thinking we may not need if there is a bug in elastic-package with elastic-agent settings.

donoghuc

I'm good with the added logging, the more flexible assertian for the bad_certificate vs certificate_required, and the more capable VM. But from what i can tell the elastic_agent stopping using ssl seems like a legit bug we need to fix rather than encoding the unencrypted communication into our tests. It feel like a legit regression we need to address. Can we split that out?

… separately if requires.

elasticmachine · 2025-11-26T19:32:33Z

💚 Build Succeeded

Buildkite Build
Commit: 27111e1

History

💚 Build #662 succeeded c40c17b
💛 Build #660 was flaky 04baf2a
💔 Build #659 failed ba95d99
💔 Build #658 failed ce078c3
💔 Build #657 failed ac25d17
💔 Build #656 failed ac25d17

cc @mashhurs

Play around with E2E tests to improve.

ac25d17

mashhurs added 3 commits November 21, 2025 11:06

Display container logs, increase VM spec.

ce078c3

Let's print more container logs to figure out what is happening.

ba95d99

Flaky unit test fix and simplify log print method.

04baf2a

mashhurs commented Nov 21, 2025

View reviewed changes

mashhurs added 2 commits November 21, 2025 16:52

Greater than 9.3 stack versions defined elastic-package env uses elas…

58ed51c

…tic_agent without setting SSL. With this change, Logstash is also opting out SSL settings for the versions greater than 9.3

Uncomment after test.

c40c17b

mashhurs commented Nov 22, 2025

View reviewed changes

mashhurs self-assigned this Nov 22, 2025

mashhurs added the backport-skip Skip automated backport with mergify label Nov 22, 2025

donoghuc self-requested a review November 25, 2025 18:29

donoghuc reviewed Nov 25, 2025

View reviewed changes

donoghuc requested changes Nov 25, 2025

View reviewed changes

Remove elastic-agent side affect changes to confirm first and address…

27111e1

… separately if requires.

mashhurs requested a review from donoghuc November 26, 2025 22:35

Play around with E2E tests to improve. #377

Are you sure you want to change the base?

Play around with E2E tests to improve. #377

Conversation

mashhurs commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

mergify bot commented Nov 20, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mashhurs Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

donoghuc left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Nov 26, 2025

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mashhurs commented Nov 20, 2025 •

edited

Loading

mashhurs Nov 26, 2025 •

edited

Loading