Merge pull request #3719 from buildkite/test-experiment-documentation

Test experiment documentation

Author: moskyb

EXPERIMENTS.md

@@ -37,18 +37,6 @@ After repository checkout, resolve `BUILDKITE_COMMIT` to a commit hash. This mak
 
 **Status**: broadly useful, we'd like this to be the standard behaviour in 4.0. 👍👍
 
-### `polyglot-hooks`
-
-Allows the agent to run hooks written in languages other than bash. This enables the agent to run hooks written in any language, as long as the language has a runtime available on the agent. Polyglot hooks can be in interpreted languages, so long as they have a valid shebang, and the interpreter specified in the shebang is installed on the agent.
-
-This experiment also allows the agent to run compiled binaries (such as those produced by Go, Rust, Zig, C et al.) as hooks, so long as they are executable.
-
-Hooks are run in a subshell, so they can't modify the environment of the agent process. However, they can use the [job-api](#job-api) to modify the environment of the job.
-
-Binary hooks are available on all platforms, but interpreted hooks are unfortunately unavailable on Windows, as Windows does not support shebangs.
-
-**Status:** Experimental while we try to cover the various corner cases. We'll probably promote this to non-experiment soon™️.
-
 ### `agent-api`
 
 This exposes a local API for interacting with the agent process.
@@ -58,19 +46,6 @@ The API is exposed via a Unix Domain Socket. The path to the socket is not avail
 
 **Status:** Experimental while we iron out the API and test it out in the wild. We'll probably promote this to non-experiment soon™.
 
-### `use-zzglob`
-
-Uses a different library for resolving glob expressions used for `artifact upload`.
-The new glob library should resolve a few issues experienced with the old library:
-
-- Because `**` is used to mean "zero or more path segments", `/**/` should match `/`.
-- Directories that cannot match the glob pattern shouldn't be walked while resolving the pattern. Failure to do this makes `artifact upload` difficult to use when run in a directory containing a mix of subdirectories with different permissions.
-- Failures to walk potential file paths should be reported individually.
-
-The new library should handle all syntax supported by the old library, but because of the chance of incompatibilities and bugs, we're providing it via experiment only for now.
-
-**Status:** Since using the old library causes problems, we hope to promote this to be the default soon™️.
-
 ### `pty-raw`
 
 Set PTY to raw mode, to avoid mapping LF (\n) to CR,LF (\r\n) in job command output.
@@ -96,7 +71,7 @@ a cancelled job should appear as a failure, regardless of the OS the agent is ru
 
 ### `interpolation-prefers-runtime-env`
 
-When interpolating the pipeline level environment block, a pipeline level environment variable could take precedence over environment variables depending on the ordering. This may contravene Buildkite's [documentation](https://buildkite.com/docs/pipelines/environment-variables#environment-variable-precedence) that suggests the Job runtime environment takes precedence over that defined by combining environment variables defined in a pipeline. 
+When interpolating the pipeline level environment block, a pipeline level environment variable could take precedence over environment variables depending on the ordering. This may contravene Buildkite's [documentation](https://buildkite.com/docs/pipelines/environment-variables#environment-variable-precedence) that suggests the Job runtime environment takes precedence over that defined by combining environment variables defined in a pipeline.
 
 We previously made this the default behaviour of the agent (as of v3.63.0) but have since reverted it.
 
@@ -122,3 +97,15 @@ has a different effect depending on this experiment:
 - With `allow-artifact-path-traversal` enabled, `foo.txt` is downloaded to `../../foo.txt`.
 
 **Status:** This experiment is an escape hatch for a security fix. While the new behaviour is more secure, it may break downloading of legitimately-uploaded artifacts.
+
+### `descending-spawn-priority`
+
+When using `--spawn` with `--spawn-with-priority`, the agent assigns ascending priorities to each spawned agent (1, 2, 3, ...). This experiment changes the priorities to be descending (-1, -2, -3, ...) instead. This helps jobs be assigned across all hosts in cases where the value of `--spawn` varies between hosts.
+
+**Status:** Experimental as an escape hatch to default behaviour. Will soon be promoted to a regular flag.
+
+### `propagate-agent-config-vars`
+
+Prepends agent configuration variables (such as `BUILDKITE_GIT_*`, `BUILDKITE_SHELL`, `BUILDKITE_CANCEL_GRACE_PERIOD`, etc.) to the environment file used by the job runner. This is useful in environments like Docker where the agent configuration is not otherwise available to the job process.
+
+**Status:** Experimental while we test the impact on job environments

internal/experiments/experiments_test.go

@@ -0,0 +1,23 @@
+package experiments
+
+import (
+	"fmt"
+	"os"
+	"strings"
+	"testing"
+)
+
+func TestAvailableExperimentsDocumented(t *testing.T) {
+	data, err := os.ReadFile("../../EXPERIMENTS.md")
+	if err != nil {
+		t.Fatalf("reading EXPERIMENTS.md: %v", err)
+	}
+	contents := string(data)
+
+	for name := range Available {
+		heading := fmt.Sprintf("### `%s`", name)
+		if !strings.Contains(contents, heading) {
+			t.Errorf("available experiment %q is missing a %q section in EXPERIMENTS.md", name, heading)
+		}
+	}
+}