Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

jozala · 2025-06-18T14:36:41Z

Gradle task added to modify branches.json. Possibility to add, remove and update branches in the file with the version field added next to the branch.
After the change the file looks like this:

{
    "branches": [
        {
            "branch": "main",
            "version": "9.1.0"
        },
        {
            "branch": "9.0",
            "version": "9.0.3"
        },
        {
            "branch": "8.19",
            "version": "8.19.1"
        }
    ]
}

BwC test configuration has been changed to use changed branches.json file to calculate unreleased ES versions, instead of combining data from Version.java and branches.json.

breskeby · 2025-06-23T07:04:55Z

...internal/src/main/java/org/elasticsearch/gradle/internal/release/UpdateBranchesJsonTask.java

+
+    @TaskAction
+    public void executeTask() throws IOException {
+        File branchesFile = new File(Util.locateElasticsearchWorkspace(project.getGradle()), "branches.json");


referencing project here while executing task is deprecated. In this case I think we should just inject ProjectLayout instead of project in the ctor and then use layout.getSettingsDirectory().file('branches.json').asFile.

The Util.locateElasticsearchWorkspace is just a utility method only used to resolve files relateive from internal builds within this repository, which we don't need here as we never run this task anywhere else but in the :elasticsearch root build

Also we probably want to make this file an outputFile property of the task

I've now changed the way to reference branches.json file location using ProjectLayout and also made the branches file an @OutputFile property of the task.

breskeby · 2025-06-23T07:06:51Z

...internal/src/main/java/org/elasticsearch/gradle/internal/release/UpdateBranchesJsonTask.java

+    private final ObjectMapper objectMapper;
+    private final BranchesFileParser branchesFileParser;
+
+    @Nullable


Usually we do not use that annotation. In Gradle tasks, you usually mark properties that are optional as @optional and as @input

I've made all three parameters @Input and @Optional.

jozala · 2025-07-07T17:29:36Z

There is still a problem with GlobalBuildInfoPluginSpec failing because BwcVersions directly use static version from VersionProperties. It needs to be solved before merging to main.

jozala · 2025-07-24T08:23:18Z

...ols-internal/src/main/java/org/elasticsearch/gradle/internal/info/GlobalBuildInfoPlugin.java

+        project.getPlugins().apply(VersionPropertiesPlugin.class);
+        Properties versionProperties = (Properties) project.getExtensions().getByName(VERSIONS_EXT);
+        JavaVersion minimumCompilerVersion = JavaVersion.toVersion(versionProperties.get("minimumCompilerJava"));
+        JavaVersion minimumRuntimeVersion = JavaVersion.toVersion(versionProperties.get("minimumRuntimeJava"));
+
+        String bundledJdkVersion = versionProperties.getProperty("bundled_jdk");
+        String bundledJdkMajorVersion = bundledJdkVersion.split("[.+]")[0];
+        Version elasticsearchVersionProperty = Version.fromString(versionProperties.getProperty("elasticsearch"));


Not using static VersionProperties any more here as this make the class untestable.
This solves the problem described here: #129637 (comment)

When VersionProperties is static it is loaded only once per JVM. It means it cannot be changed in the tests. This is a problem, because we want to test that GlobalBuildInfoPlugin can calculate unreleased versions with BwcVersions for some fixed configuration. This was impossible when VersionProperties reads the real version.properties file which changes with every release.

jozala · 2025-07-24T08:28:12Z

...al/src/integTest/resources/org/elasticsearch/gradle/internal/fake_git/remote/settings.gradle

+include ":distribution:bwc:major1"
+include ":distribution:bwc:major2"
+include ":distribution:bwc:major3"
+include ":distribution:bwc:major4"
+include ":distribution:bwc:minor1"
+include ":distribution:bwc:minor2"
+include ":distribution:bwc:minor3"
+include ":distribution:bwc:minor4"


This is simplification of the naming convention.
Name minorN means previous minor on the same major where number N is a distance from the current.
Name majorN means previous major, still actively developed versions, where N is a distance from the current, omitting all the previous minor versions.

Related changes in other files across this PR.

jozala · 2025-07-24T08:31:23Z

build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/BwcVersions.java

+        Map<String, List<DevelopmentBranch>> bwcBranches = developmentBranches.stream()
+            .filter(developmentBranch -> developmentBranch.version().before(currentVersion))
+            .sorted(reverseOrder(comparing(DevelopmentBranch::version)))
+            .collect(Collectors.groupingBy(branch -> {
+                if (branch.version().getMajor() == currentVersion.getMajor()) {
+                    return "minor";
+                } else if (branch.version().getMajor() == currentVersion.getMajor() - 1) {
+                    return "major";
+                }
+                return "older";
+            }));
+
+        developmentBranches.stream()
+            .filter(branch -> branch.version().equals(currentVersion))
+            .findFirst()
+            .ifPresent(
+                developmentBranch -> result.put(
+                    currentVersion,
+                    new UnreleasedVersionInfo(currentVersion, developmentBranch.name(), ":distribution")
+                )
+            );

-        // The current version is always in development
-        String currentBranch = getBranchFor(currentVersion, developmentBranches);
-        result.put(currentVersion, new UnreleasedVersionInfo(currentVersion, currentBranch, ":distribution"));
-
-        // Check for an n.x branch as well
-        if (currentBranch.equals("main") && developmentBranches.stream().anyMatch(s -> s.endsWith(".x"))) {
-            // This should correspond to the latest new minor
-            Version version = versions.stream()
-                .sorted(Comparator.reverseOrder())
-                .filter(v -> v.getMajor() == (currentVersion.getMajor() - 1) && v.getRevision() == 0)
-                .findFirst()
-                .orElseThrow(() -> new IllegalStateException("Unable to determine development version for branch"));
-            String branch = getBranchFor(version, developmentBranches);
-            assert branch.equals(currentVersion.getMajor() - 1 + ".x") : "Expected branch does not match development branch";
-
-            result.put(version, new UnreleasedVersionInfo(version, branch, ":distribution:bwc:minor"));
+        List<DevelopmentBranch> previousMinorBranches = bwcBranches.getOrDefault("minor", Collections.emptyList());
+        for (int i = 0; i < previousMinorBranches.size(); i++) {
+            DevelopmentBranch previousMinorBranch = previousMinorBranches.get(i);
+            result.put(
+                previousMinorBranch.version(),
+                new UnreleasedVersionInfo(previousMinorBranch.version(), previousMinorBranch.name(), ":distribution:bwc:minor" + (i + 1))
+            );
        }

-        // Now handle all the feature freeze branches
-        List<String> featureFreezeBranches = developmentBranches.stream()
-            .filter(b -> Pattern.matches("[0-9]+\\.[0-9]+", b))
-            .sorted(reverseOrder(comparing(s -> Version.fromString(s, Version.Mode.RELAXED))))
-            .toList();
-
-        int bugfixCount = 0;
-        boolean existingStaged = false;
-        for (int i = 0; i < featureFreezeBranches.size(); i++) {
-            String branch = featureFreezeBranches.get(i);
-            Version version = versions.stream()
-                .sorted(Comparator.reverseOrder())
-                .filter(v -> v.toString().startsWith(branch))
-                .findFirst()
-                .orElse(null);
-
-            // If we don't know about this version we can ignore it
-            if (version == null) {
-                continue;
-            }
-
-            // If this is the current version we can ignore as we've already handled it
-            if (version.equals(currentVersion)) {
-                continue;
-            }
-
-            // We only maintain compatibility back one major so ignore anything older
-            if (currentVersion.getMajor() - version.getMajor() > 1) {
-                continue;
-            }
-
-            // This is the maintenance version
-            if (i == featureFreezeBranches.size() - 1) {
-                result.put(version, new UnreleasedVersionInfo(version, branch, ":distribution:bwc:maintenance"));
-            } else if (version.getRevision() == 0) { // This is the next staged minor
-                String project = existingStaged ? "staged2" : "staged";
-                result.put(version, new UnreleasedVersionInfo(version, branch, ":distribution:bwc:" + project));
-                existingStaged = true;
-            } else { // This is a bugfix
-                bugfixCount++;
-                String project = "bugfix" + (bugfixCount > 1 ? bugfixCount : "");
-                result.put(version, new UnreleasedVersionInfo(version, branch, ":distribution:bwc:" + project));
-            }
+        List<DevelopmentBranch> previousMajorBranches = bwcBranches.getOrDefault("major", Collections.emptyList());
+        for (int i = 0; i < previousMajorBranches.size(); i++) {
+            DevelopmentBranch previousMajorBranch = previousMajorBranches.get(i);
+            result.put(
+                previousMajorBranch.version(),
+                new UnreleasedVersionInfo(previousMajorBranch.version(), previousMajorBranch.name(), ":distribution:bwc:major" + (i + 1))
+            );


This is the main change. Now we use branches.json to get information about the unreleased versions instead of calculating it from Version.java.

jozala · 2025-07-24T08:34:30Z

...ols-internal/src/main/java/org/elasticsearch/gradle/internal/info/GlobalBuildInfoPlugin.java

+    private List<DevelopmentBranch> getDevelopmentBranches() {
+        String branchesFileLocation = project.getProviders()
+            .gradleProperty(BRANCHES_FILE_LOCATION_PROPERTY)
+            .getOrElse(DEFAULT_BRANCHES_FILE_URL);
+        LOGGER.info("Reading branches.json from {}", branchesFileLocation);
+        byte[] branchesBytes;
+        if (branchesFileLocation.startsWith("http")) {
+            try (InputStream in = URI.create(branchesFileLocation).toURL().openStream()) {
+                branchesBytes = in.readAllBytes();
+            } catch (IOException e) {
+                throw new UncheckedIOException("Failed to download branches.json from: " + branchesFileLocation, e);
+            }
+        } else {
+            try {
+                branchesBytes = Files.readAllBytes(new File(branchesFileLocation).toPath());
+            } catch (IOException e) {
+                throw new UncheckedIOException("Failed to read branches.json from: " + branchesFileLocation, e);
            }
-        } catch (IOException e) {
-            throw new UncheckedIOException(e);
        }

-        return branches;
+        return branchesFileParser.parse(branchesBytes);


It reads development branches (with current versions of ES on these branches) from main branch using raw GitHub link. This URL can be overridden with org.elasticsearch.build.branches-file-location Gradle property if needed.

# Conflicts: # branches.json

By default, it now reads branches.json from main branch of Elasticsearch (through GitHub raw link) . This can be overridden for testing or when there is a problem with remote call.

This will be done automatically when changes in release-tools are merged and these changes are on main

It has been unused earlier, but now used again in `AbstractTransportVersionFuncTest`

…ing unreleased versions for BwC tests. Backporting changes from elastic#129637 Gradle task added to modify `branches.json`. Possibility to add, remove and update branches in the file with the version field added next to the branch. BwC test configuration has been changed to use changed `branches.json` file to calculate unreleased ES versions, instead of combining data from `Version.java` and `branches.json`.

This will be done automatically when changes in release-tools are merged and these changes are on main

…ing unreleased versions for BwC tests. Backporting changes from elastic#129637 Gradle task added to modify `branches.json`. Possibility to add, remove and update branches in the file with the version field added next to the branch. BwC test configuration has been changed to use changed `branches.json` file to calculate unreleased ES versions, instead of combining data from `Version.java` and `branches.json`.

This change cleans up the configuration by removing the property used for testing changes in BwC before the version field was introduced in branches.json. The associated TODO comment has also been removed.

Versions from branches.json are now used for calculating unreleased versions for BwC tests. Gradle task added to modify branches.json. Possibility to add, remove and update branches in the file with the version field added next to the branch. BwC test configuration has been changed to use changed branches.json file to calculate unreleased ES versions, instead of combining data from Version.java and branches.json.

Gradle task added to modify `branches.json`. Possibility to add, remove and update branches in the file with the version field added next to the branch. BwC test configuration has been changed to use changed `branches.json` file to calculate unreleased ES versions, instead of combining data from `Version.java` and `branches.json`.

jozala · 2025-09-03T09:14:56Z

Backports

Status	Branch	Result
✅	9.1
✅	9.0
✅	8.19
✅	8.18

…lastic#133524) Versions from branches.json are now used for calculating unreleased versions for BwC tests. Gradle task added to modify branches.json. Possibility to add, remove and update branches in the file with the version field added next to the branch. BwC test configuration has been changed to use changed branches.json file to calculate unreleased ES versions, instead of combining data from Version.java and branches.json.

jozala added >non-issue v9.1.0 labels Jun 18, 2025

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Jun 18, 2025

jozala force-pushed the bwc-explicit-branches-version branch from 1e1a591 to d84fef9 Compare June 20, 2025 10:24

jozala self-assigned this Jun 20, 2025

breskeby reviewed Jun 23, 2025

View reviewed changes

breskeby approved these changes Jun 23, 2025

View reviewed changes

jozala force-pushed the bwc-explicit-branches-version branch from 4d569dc to 1e2fb07 Compare June 26, 2025 12:04

elasticsearchmachine added v9.2.0 and removed v9.1.0 labels Jun 26, 2025

jozala force-pushed the bwc-explicit-branches-version branch 2 times, most recently from 022297f to 4d3aa2f Compare July 3, 2025 09:33

jozala force-pushed the bwc-explicit-branches-version branch 2 times, most recently from e0f1357 to 2bf9688 Compare July 24, 2025 08:18

jozala commented Jul 24, 2025

View reviewed changes

jozala and others added 10 commits July 24, 2025 16:06

Initial work for having explicit active branch with version connection

c18c24c

# Conflicts: # branches.json

Basic implementation with single test passing

67e07c4

Tests for UpdateBranchesJsonTask

1c1a531

Allow changing branch.json location in GlobalBuildInfoPlugin

f6fedd5

By default, it now reads branches.json from main branch of Elasticsearch (through GitHub raw link) . This can be overridden for testing or when there is a problem with remote call.

Change branch.json location temporarily to point to the forked repo

588c2dd

[CI] Auto commit changes from spotless

d4c0bbc

Artificial Gradle projects for BwC tests added

c51044d

Tests corrected to be in line with new BwC test naming

79a9c0a

Register the task in the plugin

5206a31

Configure JSON formatting for branches.json

8f3d4c2

jozala and others added 4 commits August 12, 2025 12:35

Merge branch 'main' into bwc-explicit-branches-version

72f6f1c

Merge branch 'main' into bwc-explicit-branches-version

5b4c655

Update branches.json manually after releases

8583e31

This will be done automatically when changes in release-tools are merged and these changes are on main

[CI] Auto commit changes from spotless

b1d8282

jozala mentioned this pull request Aug 21, 2025

[9.1] Add explicit versions in the branches.json (#129637) #133338

Merged

Revert the deletion of setupLocalGitRepo method

8b0e9c8

It has been unused earlier, but now used again in `AbstractTransportVersionFuncTest`

brianseeders approved these changes Aug 21, 2025

View reviewed changes

Merge branch 'main' into bwc-explicit-branches-version

a63e333

jozala mentioned this pull request Aug 22, 2025

[8.17] Add explicit versions in the branches.json (#129637) #133404

Closed

This was referenced Aug 25, 2025

[8.18] Add explicit versions in the branches.json (#129637) #133476

Merged

[8.19] Add explicit versions in the branches.json (#129637) #133524

Merged

[9.0] Add explicit versions in the branches.json (#129637) #133525

Merged

jozala and others added 4 commits September 1, 2025 10:49

Merge branch 'main' into bwc-explicit-branches-version

96b9648

Fix the compilation issues after merge

0eb15bf

[CI] Auto commit changes from spotless

4de4a5a

Update branches.json manually after releases

f156bf0

This will be done automatically when changes in release-tools are merged and these changes are on main

jozala mentioned this pull request Sep 2, 2025

Add explicit versions to branches.json #133965

Merged

jozala added 2 commits September 2, 2025 12:40

Merge branch 'main' into bwc-explicit-branches-version

a71d4f5

Remove temporary branches-file-location property in gradle.properties

88774ef

This change cleans up the configuration by removing the property used for testing changes in BwC before the version field was introduced in branches.json. The associated TODO comment has also been removed.

jozala removed the serverless-linked Added by automation, don't add manually label Sep 2, 2025

jozala merged commit b1081c0 into elastic:main Sep 2, 2025
33 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

Uh oh!

jozala commented Jun 18, 2025 •

edited

Loading

Uh oh!

breskeby Jun 23, 2025

Uh oh!

breskeby Jun 23, 2025

Uh oh!

jozala Jul 7, 2025

Uh oh!

breskeby Jun 23, 2025 •

edited

Loading

Uh oh!

jozala Jul 7, 2025

Uh oh!

jozala commented Jul 7, 2025

Uh oh!

jozala Jul 24, 2025

Uh oh!

jozala Jul 24, 2025 •

edited

Loading

Uh oh!

jozala Jul 24, 2025

Uh oh!

jozala Jul 24, 2025

Uh oh!

Uh oh!

jozala commented Sep 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

Add explicit versions in the branches.json and use them for calculating unreleased versions for BwC tests #129637

Uh oh!

Conversation

jozala commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

breskeby Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

breskeby Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

jozala Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

breskeby Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jozala Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

jozala commented Jul 7, 2025

Uh oh!

jozala Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

jozala Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jozala Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

jozala Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jozala commented Sep 3, 2025

Backports

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jozala commented Jun 18, 2025 •

edited

Loading

breskeby Jun 23, 2025 •

edited

Loading

jozala Jul 24, 2025 •

edited

Loading