Update buildkite script to look for new logstash releases location #358

donoghuc · 2025-08-06T23:10:23Z

This commit updates the buildkite scripts to look in the .ci repo in the logstash-plugins org for the new location of logstash releases.

Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/

mergify · 2025-08-06T23:10:59Z

This pull request does not have a backport label. Could you fix it @donoghuc? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

donoghuc · 2025-08-06T23:48:06Z

Looks like there is no ruby interpreter on the path for this test runner. @mashhurs why do we use

logstash-filter-elastic_integration/.buildkite/scripts/pull-request-pipeline/generate-steps.py

Lines 62 to 67 in 6a03524

    
           "agents": { 
        
               "provider": "gcp", 
        
               "machineType": "n1-standard-4", 
        
               "image": "family/core-ubuntu-2204" 
        
           }, 
        
           "steps": []}

a vanilla gcp image instead of the ones we build for logstash ci https://github.com/elastic/logstash/blob/3b13df1d6aa21f5d47828df4d93e61a1c07019e2/.buildkite/pull_request_pipeline.yml#L45

mashhurs · 2025-08-07T00:22:26Z

Looks like there is no ruby interpreter on the path for this test runner. @mashhurs why do we use

logstash-filter-elastic_integration/.buildkite/scripts/pull-request-pipeline/generate-steps.py

Lines 62 to 67 in 6a03524

"agents": {

"provider": "gcp",

"machineType": "n1-standard-4",

"image": "family/core-ubuntu-2204"

},

"steps": []}

a vanilla gcp image instead of the ones we build for logstash ci https://github.com/elastic/logstash/blob/3b13df1d6aa21f5d47828df4d93e61a1c07019e2/.buildkite/pull_request_pipeline.yml#L45

We didn't have range of our custom VMs when we setup these pipelines :) The only reason (as I remember) was docker compose was available in this VM type.
I think now, we can change to platform-ingest-logstash-multi-jdk-ubuntu-2204 (as used in E2E tests)?!

donoghuc · 2025-08-07T18:57:54Z

@mashhurs I think i have this working (now its just failing due to #357

mashhurs · 2025-08-07T19:11:20Z

@mashhurs I think i have this working (now its just failing due to #357

Great! Thank you.
Let's also land #357 if you don't have any concern. I have added more context under the PR and more work will be covered with move-to-bridge (working with my branch)

mashhurs · 2025-08-07T19:12:58Z

.buildkite/scripts/build-pipeline/generate-steps.py

+            "machineType": "n2-standard-4",
+            "imageProject": "elastic-images-prod",
+            "image": "family/platform-ingest-logstash-multi-jdk-ubuntu-2204",
+            "diskSizeGb": 120


Do you what is the default disk size?
I did increase in the E2E since we need to spin up elastic-package but I am thinking here the default (32 or 64) should be sufficient.

I just copy pasted from the other. I'll remove it and see if default works (which i suspect it will). Good catch.

Pushed a commit to check if default is sufficient. Thanks

mashhurs

LGTM if you please clarify disk size and make sure we are not allocating over resource.

mashhurs · 2025-08-07T19:13:52Z

.buildkite/scripts/run_tests.sh

+
+export JAVA_HOME="/opt/buildkite-agent/.java/adoptiumjdk_21"
+export PATH="/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:/opt/buildkite-agent/.java/bin:$JAVA_HOME:$PATH"
+eval "$(rbenv init -)"


ah right, this was a requirement!

Yeah, i will be re-visiting this as part of https://github.com/elastic/ingest-dev/issues/5827

mashhurs

LGTM!

This commit updates the buildkite scripts to look in the `.ci` repo in the logstash-plugins org for the new location of logstash releases.

Use images we are building for LS testing for all BK pipelines.

elasticmachine · 2025-08-07T20:25:36Z

💚 Build Succeeded

Buildkite Build
Commit: 03643ad

History

💔 Build #625 failed b11baaf
💔 Build #624 failed 8db7002
💔 Build #623 failed 32e1ea8
💔 Build #622 failed 6355586
💔 Build #621 failed b3fc29d

donoghuc · 2025-08-07T20:40:14Z

Green!

github-actions · 2025-08-07T20:40:43Z

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

mergify · 2025-08-07T20:41:06Z

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

#359 [8.17] (backport #358) Update buildkite script to look for new logstash releases location has been created for branch 8.17 but encountered conflicts
#360 [8.18] (backport #358) Update buildkite script to look for new logstash releases location has been created for branch 8.18 but encountered conflicts
#361 [8.19] (backport #358) Update buildkite script to look for new logstash releases location has been created for branch 8.19 but encountered conflicts
#362 [9.0] (backport #358) Update buildkite script to look for new logstash releases location has been created for branch 9.0
#363 [9.1] (backport #358) Update buildkite script to look for new logstash releases location has been created for branch 9.1

) (cherry picked from commit febb207) # Conflicts: # .buildkite/scripts/build-pipeline/generate-steps.py # .buildkite/scripts/e2e-pipeline/generate-steps.py # .buildkite/scripts/pull-request-pipeline/generate-steps.py # .buildkite/scripts/run_tests.sh

) (cherry picked from commit febb207)

…sh releases location (#359) * Update buildkite script to look for new logstash releases location (#358) (cherry picked from commit febb207) # Conflicts: # .buildkite/scripts/build-pipeline/generate-steps.py # .buildkite/scripts/e2e-pipeline/generate-steps.py # .buildkite/scripts/pull-request-pipeline/generate-steps.py # .buildkite/scripts/run_tests.sh * fix merge conflict --------- Co-authored-by: Cas Donoghue <[email protected]>

…sh releases location (#360) * Update buildkite script to look for new logstash releases location (#358) (cherry picked from commit febb207) # Conflicts: # .buildkite/scripts/build-pipeline/generate-steps.py # .buildkite/scripts/e2e-pipeline/generate-steps.py # .buildkite/scripts/pull-request-pipeline/generate-steps.py # .buildkite/scripts/run_tests.sh * fix merge conflicts --------- Co-authored-by: Cas Donoghue <[email protected]>

…sh releases location (#361) * Update buildkite script to look for new logstash releases location (#358) (cherry picked from commit febb207) # Conflicts: # .buildkite/scripts/build-pipeline/generate-steps.py # .buildkite/scripts/e2e-pipeline/generate-steps.py # .buildkite/scripts/pull-request-pipeline/generate-steps.py # .buildkite/scripts/run_tests.sh * fix merge conflicts --------- Co-authored-by: Cas Donoghue <[email protected]>

) (#362) (cherry picked from commit febb207) Co-authored-by: Cas Donoghue <[email protected]>

) (#363) (cherry picked from commit febb207) Co-authored-by: Cas Donoghue <[email protected]>

donoghuc added the backport-active-all Automated backport with mergify to all the active branches label Aug 6, 2025

donoghuc force-pushed the logstash-releases-update branch 3 times, most recently from b3fc29d to 6355586 Compare August 6, 2025 23:35

donoghuc requested a review from mashhurs August 7, 2025 18:57

mashhurs reviewed Aug 7, 2025

View reviewed changes

mashhurs previously approved these changes Aug 7, 2025

View reviewed changes

donoghuc dismissed mashhurs’s stale review via b11baaf August 7, 2025 19:25

mashhurs approved these changes Aug 7, 2025

View reviewed changes

donoghuc added 4 commits August 7, 2025 13:05

Update buildkite script to look for new logstash releases location

c5987ae

This commit updates the buildkite scripts to look in the `.ci` repo in the logstash-plugins org for the new location of logstash releases.

Use the same vm for all tests

ef0ce13

Use images we are building for LS testing for all BK pipelines.

set java home

19bd809

Use default disk size on test runners

03643ad

donoghuc force-pushed the logstash-releases-update branch from b11baaf to 03643ad Compare August 7, 2025 20:05

donoghuc merged commit febb207 into elastic:main Aug 7, 2025
4 checks passed

mergify bot mentioned this pull request Aug 7, 2025

[8.17] (backport #358) Update buildkite script to look for new logstash releases location #359

Merged

This was referenced Aug 7, 2025

[8.18] (backport #358) Update buildkite script to look for new logstash releases location #360

Merged

[8.19] (backport #358) Update buildkite script to look for new logstash releases location #361

Merged

mergify bot pushed a commit that referenced this pull request Aug 7, 2025

Update buildkite script to look for new logstash releases location (#358

1806f70

) (cherry picked from commit febb207)

mergify bot pushed a commit that referenced this pull request Aug 7, 2025

Update buildkite script to look for new logstash releases location (#358

f112444

) (cherry picked from commit febb207)

This was referenced Aug 7, 2025

[9.0] (backport #358) Update buildkite script to look for new logstash releases location #362

Merged

[9.1] (backport #358) Update buildkite script to look for new logstash releases location #363

Merged

donoghuc added a commit that referenced this pull request Aug 8, 2025

Update buildkite script to look for new logstash releases location (#358

e1f8272

) (#362) (cherry picked from commit febb207) Co-authored-by: Cas Donoghue <[email protected]>

donoghuc added a commit that referenced this pull request Aug 8, 2025

Update buildkite script to look for new logstash releases location (#358

724f933

) (#363) (cherry picked from commit febb207) Co-authored-by: Cas Donoghue <[email protected]>

Update buildkite script to look for new logstash releases location #358

Update buildkite script to look for new logstash releases location #358

Uh oh!

Conversation

donoghuc commented Aug 6, 2025

Uh oh!

mergify bot commented Aug 6, 2025

Uh oh!

donoghuc commented Aug 6, 2025

Uh oh!

mashhurs commented Aug 7, 2025

Uh oh!

donoghuc commented Aug 7, 2025

Uh oh!

mashhurs commented Aug 7, 2025

Uh oh!

mashhurs Aug 7, 2025

Choose a reason for hiding this comment

Uh oh!

donoghuc Aug 7, 2025

Choose a reason for hiding this comment

Uh oh!

donoghuc Aug 7, 2025

Choose a reason for hiding this comment

Uh oh!

mashhurs left a comment

Choose a reason for hiding this comment

Uh oh!

mashhurs Aug 7, 2025

Choose a reason for hiding this comment

Uh oh!

donoghuc Aug 7, 2025

Choose a reason for hiding this comment

Uh oh!

mashhurs left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Aug 7, 2025

💚 Build Succeeded

History

Uh oh!

donoghuc commented Aug 7, 2025

Uh oh!

Uh oh!

github-actions bot commented Aug 7, 2025

Uh oh!

mergify bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Uh oh!

mergify bot commented Aug 7, 2025 •

edited

Loading