feat: Support unit test (non-analysis test) types of tests in rules_testing

This adds support to write tests of general Starlark code that doesn't require
analysis phase information. While this was possible to do with enough work,
it's a bit of boiler plate to do so. This moves that boiler plate into
rules_testing, so that users just need to define their verification function.

Also makes `test_suite` able to handle both basic tests (ones that are only
an implementation function, like most unit tests) and regular tests (tests
that require a loading-phase setup process, like analyss tests).

Also creates a best practices doc. Some of the same advice pertains across
test suite, unit test, and analysis test.

Closes #37

PiperOrigin-RevId: 539743721

Author: rickeylev

docs/source/best_practices.md

@@ -0,0 +1,21 @@
+# Best Practices
+
+Here we collection tips and techniques for keeping your tests maintainable and
+avoiding common pitfalls.
+
+### Put each suite of tests in its own sub-package
+
+It's recommended to put a given suite of unit tests in their own sub-package
+(directory with a BUILD file). This is because the names of your test functions
+become target names in the BUILD file, which makes it easier to create name
+conflicts. By moving them into their own package, you don't have to worry about
+unit test function names in one `.bzl` file conflicting with names in another.
+
+### Give test functions private names
+
+It's recommended to give test functions private names, i.e. start with a leading
+underscore. This is because if you forget to add a test to the list of tests (an
+easy mistake to make in a file with many tests), the test won't run, and it'll
+appear as if everything is OK. By using a leading underscore, tools like
+buildifier can detect the unused private function and will warn you that it's
+unused, preventing you from accidentally forgetting it.

docs/source/test_suite.md

@@ -0,0 +1,78 @@
+# Test suites
+
+The `test_suite` macro is a front-end for easily instantiating groups of
+Starlark tests. It can handle both analysis tests and unit tests. Under the
+hood, each test is its own target with an aggregating `native.test_suite`
+for the group of tests.
+
+## Basic tests
+
+Basic tests are tests that don't require any custom setup or attributes. This is
+the common case for tests of utility code that doesn't interact with objects
+only available to rules (e.g. Targets). These tests are created using
+`unit_test`.
+
+To write such a test, simply write a `unit_test` compatible function (one that
+accepts `env`) and pass it to `test_suite.basic_tests`.
+
+```starlark
+# BUILD
+
+load(":my_tests.bzl", "my_test_suite")
+load("@rules_testing//lib:test_suite.bzl", "test_suite")
+
+my_test_suite(name = "my_tests")
+
+# my_tests.bzl
+
+def _foo_test(env):
+  env.expect.that_str(...).equals(...)
+
+def my_test_suite(name):
+  test_suite(
+    name = name,
+    basic_tests = [
+      _foo_test,
+    ]
+  )
+```
+
+Note that it isn't _required_ to write a custom test suite function, but doing
+so is preferred because it's uncommon for BUILD files to pass around function
+objects, and tools won't be confused by it.
+
+## Regular tests
+
+A regular test is a macro that acts as a setup function and is expected to
+create a target of the given name (which is added to the underlying test suite).
+
+The setup function can perform arbitrary logic, but in the end, it's expected to
+call `unit_test` or `analysis_test` to create a target with the provided name.
+
+If you're writing an `analysis_test`, then you're writing a regular test.
+
+```starlark
+# my_tests.bzl
+def _foo_test(name):
+  analysis_test(
+    name = name,
+    impl = _foo_test_impl,
+    attrs = {"myattr": attr.string(default="default")}
+  )
+
+def _foo_test_impl(env):
+  env.expect.that_str(...).equals(...)
+
+def my_test_suite(name):
+  test_suite(
+    name = name,
+    tests = [
+      _foo_test,
+    ]
+  )
+```
+
+Note that a using a setup function with `unit_test` test is not required to
+define custom attributes; the above is just an example. If you want to define
+custom attributes for every test in a suite, the `test_kwargs` argument of
+`test_suite` can be used to pass additional arguments to all tests in the suite.

docs/source/unit_tests.md

@@ -0,0 +1,73 @@
+# Unit tests
+
+Unit tests are for Starlark code that isn't specific to analysis-phase or
+loading phase cases; usually utility code of some sort. Such tests typically
+don't require a rule `ctx` or instantiating other targets to verify the code
+under test.
+
+To write such a test, simply write a function accepting `env` and pass it to
+`test_suite`. The test suite will pass your verification function to
+`unit_test()` for you.
+
+```starlark
+# BUILD
+
+load(":my_tests.bzl", "my_test_suite")
+load("@rules_testing//lib:test_suite.bzl", "test_suite")
+
+my_test_suite(name = "my_tests")
+
+# my_tests.bzl
+
+def _foo_test(env):
+  env.expect.that_str(...).equals(...)
+
+def my_test_suite(name):
+  test_suite(
+    name = name,
+    basic_tests = [
+      _foo_test,
+    ]
+  )
+```
+
+Note that it isn't _required_ to write a custom test suite function, but doing
+so is preferred because it's uncommon for BUILD files to pass around function
+objects, and tools won't be confused by it.
+
+## Customizing setup
+
+If you want to customize the setup (loading phase) of a unit test, e.g. to add
+custom attributes, then you need to write in the same style as an analysis test:
+one function is a verification function, and another function performs setup and
+calls `unit_test()`, passing in the verification function.
+
+Custom tests are like basic tests, except you can hook into the loading phase
+before the actual unit test is defined. Because you control the invocation of
+`unit_test`, you can e.g. define custom attributes specific to the test.
+
+```starlark
+# my_tests.bzl
+def _foo_test(name):
+  unit_test(
+    name = name,
+    impl = _foo_test_impl,
+    attrs = {"myattr": attr.string(default="default")}
+  )
+
+def _foo_test_impl(env):
+  env.expect.that_str(...).equals(...)
+
+def my_test_suite(name):
+  test_suite(
+    name = name,
+    custom_tests = [
+      _foo_test,
+    ]
+  )
+```
+
+Note that a custom test is not required to define custom attributes; the above
+is just an example. If you want to define custom attributes for every test in a
+suite, the `test_kwargs` argument of `test_suite` can be used to pass additional
+arguments to all tests in the suite.

lib/BUILD

@@ -13,6 +13,7 @@
 # limitations under the License.
 
 load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
+load("//lib/private:util.bzl", "do_nothing")
 
 licenses(["notice"])
 
@@ -26,7 +27,10 @@ bzl_library(
     srcs = ["analysis_test.bzl"],
     visibility = ["//visibility:public"],
     deps = [
-        "//lib:truth_bzl",
+        ":test_suite_bzl",
+        ":truth_bzl",
+        "//lib/private:analysis_test_bzl",
+        "//lib/private:util_bzl",
     ],
 )
 
@@ -57,6 +61,25 @@ bzl_library(
     ],
 )
 
+bzl_library(
+    name = "unit_test_bzl",
+    srcs = ["unit_test.bzl"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//lib/private:analysis_test_bzl",
+    ],
+)
+
+bzl_library(
+    name = "test_suite_bzl",
+    srcs = ["test_suite.bzl"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":unit_test_bzl",
+        "//lib/private:util_bzl",
+    ],
+)
+
 filegroup(
     name = "test_deps",
     testonly = True,
@@ -81,3 +104,9 @@ exports_files(
         "//docgen:__pkg__",
     ],
 )
+
+# Unit tests need some target because they're based upon analysis tests.
+do_nothing(
+    name = "_stub_target_for_unit_tests",
+    visibility = ["//visibility:public"],
+)