Custom integrations support?

[be9]

Are there any plans to support custom integrations for bktec? We've got some integration test suites that:

• are written in Go
• are run with Bazel
• are sharded because they are quite slow
  • are sharded by test name, not by package

Currently, sharding is done by having something like:

if hashFunc(t.Name()) % shardCount != currentShardID {
    t.Skip()
}

in the test code. We are interested in test splitting feature, but bktec is currently very far from what we need. I learned that Bazel support is on the roadmap, but the description says that it's mostly about piping BEP to Buildkite and not necessarily about test splitting.

[jameshill]

Hey @be9! Thanks for reaching out.

That's right the Bazel support is primarily about adding support for data ingest via BEP and the Bazel event stream. I'd love to know what you had in mind for supporting custom integrations and how it might fit in with Bazel and bin packing the selected tests.

[be9]

@jameshill thanks for quick response! So as of now, you've got this interface

type TestRunner interface {
	// Run takes testCases as input, executes the test against the test cases, and mutates the runner.RunResult with the test results.
	Run(result *runner.RunResult, testCases []plan.TestCase, retry bool) error
	// GetExamples discovers all tests within given files.
	// This function is only used for split by example use case. Currently only supported by RSpec.
	GetExamples(files []string) ([]plan.TestCase, error)
	// GetFiles discover all test files that the runner should execute.
	// This is sent to server-side when creating test plan.
	// This is also used to obtain a fallback non-intelligent test splitting mechanism.
	GetFiles() ([]string, error)
	Name() string
}

with a few implementations. For a custom integration, what if GetExamples and GetFiles read from a provided JSON file, and Run writes example names / files to be run by the current shard to a JSON file (without running anything)? Our test runner will then use these data to skip tests for other shards.

It's basically equivalent to making Test Plan API public.

[jameshill]

Hey @be9, just wanted to let you know i've raised this with the team and we are exploring option. I don't have a timeframe at the moment though.