Run mixed mode tests in a workflow, and publish results to release notes (#3166)

This adds a workflow to run the yaml-tests at a given tag against the
last 50 releases, and adds the results to `docs/ReleaseNotes.md`.
Note: Currently it will only download back to `4.0.559.1`, because older
builds didn't have a server to do mixed-mode testing against.

Currently this is a manual workflow that someone would run after a
successful build, but once proven, we can change it to automatically run
after a release.

There is some discussion in @alecgrieser's review, but I will also point
out here that the results are somewhat weakened by the use of
`!supported_version`. For example, `prepared.yamsql` is set to
`!supported_version: 4.1.6.0` because that support only exists in
`4.1.6.0`, but it was also added to `4.0.559.6` (which was released
after the test run in the sample below). We also have issues where the
behavior may change between two versions. None of that is visible here,
and would have to be noted in the release notes, and found there. But
hopefully the fact that we run single version mixed-mode tests during
PRB and release will prevent issues from completely slipping through.

Sample output (an actual run, but the link is to a different workflow):

> #### Mixed Mode Test Results
> 
> Mixed mode testing run against the following previous versions:
> ✅`4.0.559.1`, ✅`4.0.559.2`, ✅`4.0.559.3`, ✅`4.0.559.4`, ❌`4.0.561.0`,
✅`4.0.562.0`, ✅`4.0.564.0`, ✅`4.0.565.0`, ✅`4.0.566.0`, ✅`4.0.567.0`,
✅`4.0.568.0`, ✅`4.0.569.0`, ✅`4.0.570.0`, ✅`4.0.571.0`, ✅`4.0.572.0`,
✅`4.0.573.0`, ✅`4.0.574.0`, ✅`4.0.575.0`, ✅`4.1.4.0`, ✅`4.1.5.0`,
✅`4.1.6.0`
> 
> [See full test
run](https://github.com/FoundationDB/fdb-record-layer/actions/runs/13331380671)

Author: ScottDugas

.github/workflows/mixed_mode_test.yml

@@ -0,0 +1,46 @@
+name: Mixed Mode Test
+
+on:
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: 'Tag to test from'
+        required: true
+
+jobs:
+  gradle:
+    runs-on: ubuntu-latest
+    permissions:
+      pull-requests: write
+    steps:
+      - name: Checkout sources
+        uses: actions/[email protected]
+        with:
+          ref: ${{ inputs.tag }}
+          ssh-key: ${{ secrets.DEPLOY_KEY }}
+      - name: Fetch Main
+        run: git fetch --depth=1 origin main
+      - name: Setup Base Environment
+        uses: ./actions/setup-base-env
+      - name: Setup FDB
+        uses: ./actions/setup-fdb
+
+      # Push a version bump back to main. There are failure scenarios that can result
+      # in published artifacts but an erroneous build, so it's safer to bump the version
+      # at the beginning
+      - name: Configure git
+        run: |
+          git config --global user.name 'FoundationDB CI'
+          git config --global user.email '[email protected]'
+      - name: Run Gradle Test
+        uses: ./actions/gradle-test
+        with:
+          gradle_command: mixedModeTest
+          gradle_args: -PreleaseBuild=false -PpublishBuild=false
+      - name: Checkout Main
+        run: git checkout main
+      - name: Update release notes
+        run: python build/publish-mixed-mode-results.py ${{ inputs.tag }} --release-notes docs/ReleaseNotes.md --commit --run-link ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+      - name: Push release notes update
+        run: git push
+

build/publish-mixed-mode-results.py

@@ -0,0 +1,111 @@
+#!/usr/bin/env python3
+
+#
+# publish_mixed_mode_results.py
+#
+# This source file is part of the FoundationDB open source project
+#
+# Copyright 2015-2025 Apple Inc. and the FoundationDB project authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script generates a list of mixed-mode testing results generated by the task
+# ./gradlew mixedModeTest
+
+import argparse
+import subprocess
+import sys
+
+def run(command):
+    try:
+        process = subprocess.run(command, check=True, capture_output=True, text=True)
+        return process.stdout
+    except subprocess.CalledProcessError as e:
+        print("Failed: " + str(e.cmd))
+        print(e.stdout)
+        print(e.stderr)
+        exit(e.returncode)
+
+def get_results(results_path):
+    results = {}
+    with open(results_path) as f:
+        for line in f:
+            split = line.strip().split(' ')
+            if len(split) != 2:
+                raise Exception("Line is not valid: " + line)
+            result = split[0]
+            version = split[1]
+            if result == 'FAILURE':
+                results[version] = result
+            elif result == 'SUCCESS' and version not in results:
+                results[version] = result
+    return results
+
+def emoji(result_word):
+    if result_word == 'FAILURE':
+        return '❌'
+    elif result_word == 'SUCCESS':
+        return '✅'
+    else:
+        raise Exception('Invalid result type: ' + result_word)
+
+def generate_markdown(version, results, header_size):
+    sorted_keys = sorted(results.keys(), key=lambda raw: [int(part) for part in raw.split('.')])
+
+    return header_size + " Mixed Mode Test Results\n\nMixed mode testing run against the following previous versions:\n\n" + \
+        ', '.join([emoji(results[version]) + '`' + version + '`' for version in sorted_keys])
+
+def update_release_notes_file(markdown, version, filename):
+    with open(filename, 'r') as fin:
+        lines = fin.read().split('\n')
+    i = 0
+    target = '<!-- MIXED_MODE_RESULTS ' + version + ' PLACEHOLDER -->'
+    while i < len(lines) and not lines[i].startswith(target):
+        i+= 1
+    if i == len(lines):
+        raise Exception('Could not find placeholder in release notes file')
+    return '\n'.join(lines[:i]
+                     + [markdown]
+                     + lines[i+1:])
+
+def commit_updates(filename, version):
+    subprocess.run(['git', 'commit', '-m', "Recording " + version + "mixed mode test results in release notes", filename],
+                           check=True)
+
+def main(argv):
+    '''Process the output of a mixedModeTest run and convert it into a short markdown'''
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--results-path', help='Path to the results', default='.out/reports/mixed-mode-results.log')
+    parser.add_argument('--release-notes', help='If provided, the results will be injected into this file')
+    parser.add_argument('--header-size', help='Markdown header level (e.g. # or ##)', default='####')
+    parser.add_argument('--run-link', help='A link to the test run that generated the results')
+    parser.add_argument('--commit', action='store_true', default=False, help='Commit the updates to the release notes')
+    parser.add_argument('version', help='Version of the server that was tested')
+    args = parser.parse_args(argv)
+
+    markdown = generate_markdown(args.version, get_results(args.results_path), args.header_size)
+    if args.run_link is not None:
+        markdown = markdown + "\n\n[See full test run](" + args.run_link +")"
+    if args.release_notes is None:
+        print(markdown)
+    else:
+        new_content = update_release_notes_file(markdown, args.version, args.release_notes)
+        with open(args.release_notes, 'w') as fout:
+            fout.write(new_content)
+        if args.commit:
+            commit_updates(args.release_notes, args.version)
+        print(f'Updated {args.release_notes} with test results for {args.version}')
+
+if __name__ == '__main__':
+    main(sys.argv[1:])

build/update_release_notes.py

@@ -106,6 +106,7 @@ def get_new_contents(filename, new_version):
         + [template]
         + ['// end next release', '-->', '']
         + updated_next_release_notes
+        + ['', '<!-- MIXED_MODE_RESULTS ' + new_version + ' PLACEHOLDER -->', '']
         + lines[next_release_end+3:])
 
 

docs/ReleaseNotes.md

@@ -52,6 +52,8 @@ e** Change 1 [(Issue #NNN)](https://github.com/FoundationDB/fdb-record-layer/iss
 * **Feature** Allow scrubbing of indexes in READABLE_UNIQUE_PENDING state [(Issue #3135)](https://github.com/FoundationDB/fdb-record-layer/issues/3135)
 * **Feature** Support Lucene index scrubbing [(Issue #3008)](https://github.com/FoundationDB/fdb-record-layer/issues/3008)
 
+<!-- MIXED_MODE_RESULTS 4.1.6.0 PLACEHOLDER -->
+
 ### 4.1.4.0
 
 * **Bug fix** Ungrouped GROUP BY queries result in infinite continuations when maxRows is 1 [(Issue #3093)](https://github.com/FoundationDB/fdb-record-layer/issues/3093)

gradle/testing.gradle

@@ -1,3 +1,5 @@
+import java.util.regex.Matcher
+
 /*
  * testing.gradle
  *
@@ -56,9 +58,58 @@ task quickTest(type: Test) {
     }
 }
 
+def getMixedModeVersion(descriptor) {
+    while (descriptor != null) {
+        // don't display this or any of the parents. Let's assume that nobody ever
+        // sets the display name in code to start with "Gradle Test Executor".
+        // it appears to be suffixed with a number, but I didn't investigate why
+        if (descriptor.displayName.startsWith("Gradle Test Executor")) {
+            break
+        }
+
+        Matcher versionMatch = descriptor.displayName =~ /^MultiServer \((?:((?:\d+\.)+\d+) then Embedded|Embedded then ((?:\d+\.)+\d+))\)/
+        if (versionMatch.size() != 0) {
+            def version = versionMatch[0][1]
+            if (version == null) {
+                version = versionMatch[0][2]
+            }
+            return version
+        }
+        descriptor = descriptor.parent
+    }
+    return null
+}
+
+task mixedModeTest(type: Test) {
+    useJUnitPlatform {
+        includeTags 'MixedMode'
+    }
+
+    ignoreFailures = true
+
+    // Skip non-MultiServer yaml tests
+    systemProperties['tests.mixedModeOnly'] = 'true'
+
+    def markdownReports = new File("$rootDir/.out/reports/")
+    def mixedModeResults = new File(markdownReports, "mixed-mode-results.log")
+    doFirst {
+        // This may have issues with running tasks concurrently
+        if (!markdownReports.exists()) {
+            markdownReports.mkdirs()
+        }
+    }
+    afterTest { descriptor, result ->
+        def version = getMixedModeVersion(descriptor)
+        if (version == null) {
+            throw new RuntimeException("Could not find version " + getFullDisplayName(descriptor))
+        } else {
+            mixedModeResults.append("${result.resultType} ${version}\n")
+        }
+    }
+}
+
 def getFullDisplayName(descriptor) {
-    def fullName = ""
-    fullName = descriptor.displayName
+    def fullName = descriptor.displayName
     descriptor = descriptor.parent
     while (descriptor != null) {
         // don't display this or any of the parents. Let's assume that nobody ever
@@ -89,8 +140,9 @@ def configureTestTask = { propertyPrefix, task ->
                    propertyPrefix + '.forkEvery']
     System.properties.each { prop ->
         def prefix = "${propertyPrefix}.sysProp."
-        if (!prop.key.startsWith(prefix.toString()))
-            return;
+        if (!prop.key.startsWith(prefix.toString())) {
+            return
+        };
         def setkey = prop.key.substring(prefix.length())
         task.systemProperties[setkey] = prop.value
         logger.debug "Set system property ${setkey} = ${prop.value} on ${propertyPrefix}"

yaml-tests/src/main/java/com/apple/foundationdb/relational/yamltests/YamlTest.java

@@ -20,6 +20,7 @@
 
 package com.apple.foundationdb.relational.yamltests;
 
+import org.junit.jupiter.api.Tag;
 import org.junit.jupiter.api.extension.ExtendWith;
 
 import java.lang.annotation.Retention;
@@ -43,6 +44,9 @@
  */
 @Retention(RetentionPolicy.RUNTIME)
 @ExtendWith(YamlTestExtension.class)
+// Right now these are the only tests that have the capability to run in mixed mode, but if we create other tests, this
+// should be moved to a shared static location
+@Tag("MixedMode")
 public @interface YamlTest {
     /**
      * Simple interface to run a {@code .yamsql} file, based on the config.

yaml-tests/src/main/java/com/apple/foundationdb/relational/yamltests/YamlTestExtension.java

@@ -81,9 +81,13 @@ public void beforeAll(final ExtensionContext context) throws Exception {
             for (ExternalServer server : servers) {
                 server.start();
             }
+            final boolean mixedModeOnly = Boolean.parseBoolean(System.getProperty("tests.mixedModeOnly", "false"));
+            final Stream<YamlTestConfig> localTestingConfigs = mixedModeOnly ?
+                                                               Stream.of() :
+                                                               Stream.of(new EmbeddedConfig(), new JDBCInProcessConfig());
             testConfigs = Stream.concat(
                     // The configs for local testing (single server)
-                    Stream.of(new EmbeddedConfig(), new JDBCInProcessConfig()),
+                    localTestingConfigs,
                     // The configs for multi-server testing (4 configs for each server available)
                     servers.stream().flatMap(server ->
                             Stream.of(new MultiServerConfig(0, server),

yaml-tests/yaml-tests.gradle

@@ -1,3 +1,6 @@
+import java.util.jar.JarFile
+import java.util.regex.Matcher
+
 /*
  * yaml-tests.gradle
  *
@@ -104,20 +107,55 @@ ext.resolveOtherServer = { Set<String> rejectedVersions ->
                 })
         def configuration = configurations.getByName(configurationName, { })
 
-        def resolution = configuration.resolve()[0]
-        def versionMatch = resolution.getName() =~ /^fdb-relational-server-(.*-SNAPSHOT)-all.jar$/
+        File resolution = configuration.resolve()[0]
+        Matcher versionMatch = resolution.getName() =~ /^fdb-relational-server-(.*-SNAPSHOT)-all.jar$/
         if (versionMatch.size() != 0) {
-            System.out.println("Rejecting old external server: " + resolution.getName())
+            println("Rejecting old external server: " + resolution.getName())
             def version = versionMatch[0][1]
             // check that the version is new, to more obviously catch potential infinite loops
             assert rejectedVersions.add(version)
             return resolveOtherServer(rejectedVersions)
         }
-        System.out.println("Downloaded old external server: " + resolution.getName())
+        println("Downloaded old external server: " + resolution.getName())
         return resolution
     }
 }
 
+static def getAttributesFromJar(File file) {
+    try (JarFile jarFile = new JarFile(file)) {
+        java.util.jar.Manifest manifest = jarFile.getManifest()
+        java.util.jar.Attributes mainAttributes = manifest.getMainAttributes()
+        String version = mainAttributes.getValue("Specification-Version")
+        // It looks like `Build-Date` is not currently being included, if we had that we could guarantee the last
+        // x weeks of builds. There may need to be some ways to handle the fact that built date may not be consistent
+        // with the ordering of the versions
+        if (version != null) {
+            return [version: version]
+        } else {
+            throw new RuntimeException("Server does not specify a version in the manifest: " + file.getAbsolutePath())
+        }
+    }
+}
+
+ext.resolveManyServers = { ->
+    Set<File> selectedServers = new HashSet<>();
+    Set<String> rejectedVersions = new HashSet<>();
+    while (selectedServers.size() < 50) {
+        def serverFile = resolveOtherServer(rejectedVersions)
+        def attributes = getAttributesFromJar(serverFile)
+        // 4.0.559.0 is the first version that introduced the server, so we won't be able to find anything
+        // older than that. Eventually we can remove this check because we'll be long enough away in terms of versions
+        // and times.
+        if (attributes.version == "4.0.559.0") {
+            break
+        }
+        rejectedVersions.add(attributes.version)
+        selectedServers.add(serverFile)
+    }
+    println("Found ${selectedServers.size()} to test against")
+    return selectedServers
+}
+
 task cleanExternalServerDirectory(type: Delete) {
     delete project.layout.buildDirectory.dir('externalServer')
 }
@@ -131,6 +169,19 @@ task serverJars(type: Copy) {
     }
 }
 
+task downloadManyExternalServers(type: Copy) {
+    dependsOn "cleanExternalServerDirectory"
+    from resolveManyServers()
+    into project.layout.buildDirectory.dir('externalServer')
+}
+
+mixedModeTest {
+    dependsOn("downloadManyExternalServers")
+    systemProperty("yaml_testing_external_server", project.layout.buildDirectory.dir('externalServer').get().asFile)
+    // this is specified in testing.gradle, but it looks like it needs to be repeated here.
+    ignoreFailures = true
+}
+
 test {
     dependsOn "serverJars"
     systemProperty("yaml_testing_external_server", project.layout.buildDirectory.dir('externalServer').get().asFile)