Fix custom twitter_scrooge toolchain breakages (#1747)

* Fix custom `twitter_scrooge` toolchain breakages

Adds the `examples/twitter_scrooge` repository, adds
`test_twitter_scrooge_example` to `test/shell/test_examples.sh`, and
enables overriding all `twitter_scrooge` dependencies. Fixes #1744.

Adds `docs/twitter_scrooge.md` and includes extensive comments in
`examples/twitter_scrooge/{BUILD,MODULE.bazel,WORKSPACE}`.

Also updates `test/shell/test_examples.sh`:

- It now uses `run_tests` from `test/shell/test_runner.sh`, and all test
  functions begin with `test_`. This enables automatic discovery and
  easy skipping of test cases (using the prefix `_test_`).

- Replaces `test_example` with `run_in_example_dir`. This removes a
  subshell per test case, and allows for passing individually quoted
  arguments.

---

Fixes two breakages related to overriding default `twitter_scrooge`
dependencies under Bzlmod (neither problem manifested under `WORKSPACE).

The first happened when using `setup_scrooge_toolchain` without
instantiating the default Scala and `twitter_scrooge` toolchains and
importing some of their repos. Even with all the available dependency
parameters to `setup_scrooge_toolchain` specified, the build would break
with errors like:

```txt
no such package
  '@@[unknown repo 'io_bazel_rules_scala_scopt_2_13_16'
  requested from @@]//':

The repository
  '@@[unknown repo 'io_bazel_rules_scala_scopt_2_13_16'
  requested from @@]'
  could not be resolved:

No repository visible as '@io_bazel_rules_scala_scopt_2_13_16'
  from main repository and referenced by
  '//tools/scala:compiler_classpath_provider'
```

This was because `setup_scala_toolchain` still contained hardcoded
references to `io_bazel_rules_scala_scopt` and other internal dependency
repos. The workaround was to instantiate the builtin toolchains, then
use `use_repo` to bring these internal repositories into scope.

The fix was to make all of these repos overridable via
`setup_scala_toolchain` and the `scala_deps.twitter_scrooge()` tag
class. Now providing all the dependency arguments to
`setup_scala_toolchain` avoids the need instantiate the builtin
toolchains.

The second breakage happened when specifying overrides for the
`scala_deps.twitter_scrooge()` tag class:

```txt
@@rules_scala+//scala/extensions:rules_scala++scala_deps+rules_scala_toolchains:
  expected value of type 'string' for dict value element,
  but got
  Label("@@rules_jvm_external++maven+maven//:org_apache_thrift_libthrift")
```

This happened because:

- The `scala_deps` module extension compiled its `twitter_scrooge` tag
  class Labels into a dictionary.

- It passed this `string` to `Label` dict to `scala_toolchains()`, which
  assigned it to the `twitter_scrooge_options` attribute of
  `scala_toolchains_repo`.

- The `scala_toolchains_repo` attribute is of type `attr.string_dict`,
  not `attr.string_keyed_label_dict`.

The fix was to translate the Labels in this dictionary into strings at
the point of `scala_toolchains_repo` assignment. We don't officially
support Bazel 6 anymore, but I didn't want to change the attribute to
`string_keyed_label_dict` just yet. Things still work fine without it,
and it gives Bazel 6 users a little more wiggle room and time to
migrate.

* Remove examples/twitter_scrooge/BUILD first line

This is left over from my first draft, when I'd written the new
tests in `test/shell/test_twitter_scrooge_toolchains.sh`.

Author: mbland

.bazelignore

@@ -13,6 +13,7 @@ examples/semanticdb
 examples/testing/multi_frameworks_toolchain
 examples/testing/scalatest_repositories
 examples/testing/specs2_junit_repositories
+examples/twitter_scrooge
 test/proto_cross_repo_boundary/repo
 test_cross_build
 third_party/test/example_external_workspace

README.md

@@ -30,6 +30,9 @@ This project defines core build rules for [Scala](https://www.scala-lang.org/) t
 - [scala_import](./docs/scala_import.md)
 - [scala_doc](./docs/scala_doc.md)
 
+See the [docs](./docs/) directory for documentation on other `rules_scala`
+capabilities as well.
+
 ## Getting started
 
 [Install Bazel][], preferably using the [Bazelisk][] wrapper. See the

docs/twitter_scrooge.md

@@ -0,0 +1,79 @@
+# Using the `twitter_scrooge` toolchain and rules
+
+## Rules
+
+```py
+load("@rules_scala//thrift:thrift.bzl", "thrift_library")
+load(
+    "@rules_scala//twitter_scrooge/toolchain:toolchain.bzl",
+    "setup_scrooge_toolchain",
+)
+load(
+    "@rules_scala//twitter_scrooge:twitter_scrooge.bzl",
+    "scrooge_java_library",
+    "scrooge_scala_library",
+)
+```
+
+## Examples
+
+The [`//test/src/main/scala/scalarules/test/twitter_scrooge`][] package provides
+extensive examples of `twitter_scrooge` rule usage.
+
+## Toolchain configuration
+
+### Default builtin toolchain
+
+To use the builtin toolchain with its default dependencies under Bzlmod:
+
+```py
+# MODULE.bazel
+
+scala_deps = use_extension("//scala/extensions:deps.bzl", "scala_deps")
+dev_deps.scala()
+dev_deps.twitter_scrooge()
+```
+
+And under `WORKSPACE`:
+
+```py
+# WORKSPACE
+load(
+    "@rules_scala//scala:toolchains.bzl",
+    "scala_register_toolchains",
+    "scala_toolchains",
+)
+
+scala_toolchains(twitter_scrooge = True)
+
+scala_register_toolchains()
+```
+
+### Builtin toolchain dependency overrides
+
+The [`examples/twitter_scrooge`][] repository shows how to configure the
+toolchains for `twitter_scrooge` rules in both [`MODULE.bazel`][] and
+[`WORKSPACE`][]. Both use [`rules_jvm_external`][] to import Maven artifacts for
+overriding the builtin `twitter_scrooge` toolchain defaults.
+
+### Defining a custom toolchain
+
+[`examples/twitter_scrooge/BUILD`][] shows how to use `setup_scrooge_toolchain`
+to define a custom `twitter_scrooge` toolchain with [`rules_jvm_external`][]
+artifacts.
+
+### More information
+
+See the comments in the above [`examples/twitter_scrooge`][] files for
+configuration details.
+
+See the [Bazel manual on toolchain resolution](
+https://bazel.build/extending/toolchains#toolchain-resolution) for guidance on
+selecting a specific toolchain.
+
+[`//test/src/main/scala/scalarules/test/twitter_scrooge`]: ../test/src/main/scala/scalarules/test/twitter_scrooge
+[`examples/twitter_scrooge`]: ../examples/twitter_scrooge/
+[`MODULE.bazel`]: ../examples/twitter_scrooge/MODULE.bazel
+[`WORKSPACE`]: ../examples/twitter_scrooge/WORKSPACE
+[`examples/twitter_scrooge/BUILD`]: ../examples/twitter_scrooge/BUILD
+[`rules_jvm_external`]: https://github.com/bazel-contrib/rules_jvm_external

examples/twitter_scrooge/.bazelrc

@@ -0,0 +1 @@
+import ../../.bazelrc

examples/twitter_scrooge/.bazelversion

@@ -0,0 +1 @@
+7.6.1

examples/twitter_scrooge/BUILD

@@ -0,0 +1,65 @@
+# Targets adapted from //test/src/main/scala/scalarules/test/twitter_scrooge.
+load("@rules_scala//scala:scala.bzl", "scala_library")
+load("@rules_scala//thrift:thrift.bzl", "thrift_library")
+load(
+    "@rules_scala//twitter_scrooge:twitter_scrooge.bzl",
+    "scrooge_scala_library",
+)
+load(
+    "@rules_scala//twitter_scrooge/toolchain:toolchain.bzl",
+    "setup_scrooge_toolchain",
+)
+
+# When using `setup_scrooge_toolchain` with all its dependencies specified, you
+# don't need to instantiate the builtin toolchain. In that case, make sure to
+# register your custom toolchain via `register_toolchains` in `MODULE.bazel` or
+# `WORKSPACE`. See the comments in those files for further details.
+#
+# It's OK to remove any of these overrides in order to use the builtin defaults
+# for those dependencies instead. However, in that case, you _must_ instantiate
+# the default `twitter_scrooge` toolchain in `MODULE.bazel` or `WORKSPACE`,
+# without the corresponding dependency overrides. This allows `rules_scala` to
+# generate the necessary builtin dependency repositories, even if you don't use
+# the default toolchain.
+#
+# However, if you remove the `scrooge_generator` override, the toolchain will
+# also depend on the builtin `mustache` and `scopt` repos. You will need to
+# remove the `mustache` and `scopt` overrides, too, to use the builtin repos
+# instead.
+setup_scrooge_toolchain(
+    name = "toolchain_from_build_file",
+    javax_annotation_api = "@maven//:javax_annotation_javax_annotation_api",
+    libthrift = "@maven//:org_apache_thrift_libthrift",
+    mustache = "@maven//:com_github_spullara_mustache_java_compiler",
+    scopt = "@maven//:com_github_scopt_scopt_2_12",
+    scrooge_core = "@maven//:com_twitter_scrooge_core_2_12",
+    scrooge_generator = "@maven//:com_twitter_scrooge_generator_2_12",
+    util_core = "@maven//:com_twitter_util_core_2_12",
+    util_logging = "@maven//:com_twitter_util_logging_2_12",
+)
+
+scala_library(
+    name = "justscrooge",
+    srcs = ["JustScrooge.scala"],
+    exports = [":scrooge"],
+    deps = [":scrooge"],
+)
+
+scrooge_scala_library(
+    name = "scrooge",
+    visibility = ["//visibility:public"],
+    deps = [":thrift"],
+)
+
+thrift_library(
+    name = "thrift",
+    srcs = ["Thrift1.thrift"],
+    visibility = ["//visibility:public"],
+    deps = [":thrift2"],
+)
+
+thrift_library(
+    name = "thrift2",
+    srcs = ["Thrift2.thrift"],
+    visibility = ["//visibility:public"],
+)

examples/twitter_scrooge/JustScrooge.scala

@@ -0,0 +1,9 @@
+package examples.twitter_scrooge
+
+object JustScrooge {
+  val classes = Seq(classOf[Struct1])
+
+  def main(args: Array[String]) {
+    print(s"classes ${classes.mkString(",")}")
+  }
+}

examples/twitter_scrooge/MODULE.bazel

@@ -0,0 +1,128 @@
+# Test configuration for test/shell/test_twitter_scrooge_toolchains.sh.
+module(name = "twitter_scrooge_toolchains")
+
+bazel_dep(name = "rules_scala")
+local_path_override(
+    module_name = "rules_scala",
+    path = "../..",
+)
+
+bazel_dep(name = "latest_dependencies")
+local_path_override(
+    module_name = "latest_dependencies",
+    path = "../../deps/latest",
+)
+
+bazel_dep(
+    name = "protobuf",
+    version = "31.1",
+    repo_name = "com_google_protobuf",
+)
+
+# Temporarily required for `protoc` toolchainization until resolution of
+# protocolbuffers/protobuf#19679.
+single_version_override(
+    module_name = "protobuf",
+    patch_strip = 1,
+    patches = ["//:protobuf.patch"],
+    version = "31.1",
+)
+
+bazel_dep(name = "rules_jvm_external", version = "6.7")
+
+maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven")
+maven.install(
+    artifacts = [
+        "com.github.scopt:scopt_2.12:4.0.0-RC2",
+        "com.github.spullara.mustache.java:compiler:0.8.18",
+        "com.twitter:scrooge-core_2.12:21.2.0",
+        "com.twitter:scrooge-generator_2.12:21.2.0",
+        "com.twitter:util-core_2.12:21.2.0",
+        "com.twitter:util-logging_2.12:21.2.0",
+        "javax.annotation:javax.annotation-api:1.3.2",
+        "org.apache.thrift:libthrift:0.10.0",
+    ],
+    lock_file = "//:maven_install.json",
+)
+use_repo(maven, "maven")
+
+scala_protoc = use_extension(
+    "@rules_scala//scala/extensions:protoc.bzl",
+    "scala_protoc",
+    dev_dependency = True,
+)
+use_repo(scala_protoc, "rules_scala_protoc_toolchains")
+
+register_toolchains("@rules_scala_protoc_toolchains//...:all")
+
+scala_config = use_extension(
+    "@rules_scala//scala/extensions:config.bzl",
+    "scala_config",
+)
+scala_config.settings(scala_version = "2.12.20")
+
+scala_deps = use_extension(
+    "@rules_scala//scala/extensions:deps.bzl",
+    "scala_deps",
+    dev_dependency = True,
+)
+scala_deps.scala()
+
+# When using `setup_scrooge_toolchain` with all its dependencies specified in a
+# `BUILD` file, you don't need to instantiate this builtin toolchain. In that
+# case, make sure to register your custom toolchain via `register_toolchains`
+# (see below). See the `//:toolchain_from_build_file` comments in the `BUILD`
+# file for further details.
+#
+# It's OK to remove any of these overrides in order to use the builtin defaults
+# for those dependencies instead.
+#
+# However, if you remove the `scrooge_generator` override, the toolchain will
+# also depend on the builtin `mustache` and `scopt` repos. You will need to
+# remove the `mustache` and `scopt` overrides, too, to use the builtin repos
+# instead.
+scala_deps.twitter_scrooge(
+    javax_annotation_api = "@maven//:javax_annotation_javax_annotation_api",
+    libthrift = "@maven//:org_apache_thrift_libthrift",
+    mustache = "@maven//:com_github_spullara_mustache_java_compiler",
+    scopt = "@maven//:com_github_scopt_scopt_2_12",
+    scrooge_core = "@maven//:com_twitter_scrooge_core_2_12",
+    scrooge_generator = "@maven//:com_twitter_scrooge_generator_2_12",
+    util_core = "@maven//:com_twitter_util_core_2_12",
+    util_logging = "@maven//:com_twitter_util_logging_2_12",
+)
+
+# If you want to depend on any of the builtin repos when using
+# `setup_scala_toolchain` in a `BUILD` file, you will need to:
+#
+# - Remove the `scala_deps.twitter_scrooge()` overrides for those repos. This
+#   enables the module extension to generate the builtin repos for those
+#   dependencies.
+#
+# - Uncomment the `use_repo` call below to import the builtin repos into the
+#   main module's scope.
+#
+# `version_suffix` should match the `scala_version` argument to
+# `scala_config.settings()`.
+#version_suffix = "_2_12_20"
+#[
+#    use_repo(scala_deps, repo + version_suffix)
+#    for repo in [
+#        "libthrift",
+#        "io_bazel_rules_scala_scrooge_core",
+#        "io_bazel_rules_scala_scrooge_generator",
+#        "io_bazel_rules_scala_util_core",
+#        "io_bazel_rules_scala_util_logging",
+#        "io_bazel_rules_scala_javax_annotation_api",
+#        "io_bazel_rules_scala_mustache",
+#        "io_bazel_rules_scala_scopt",
+#    ]
+#]
+
+# To depend on the toolchain defined by `setup_scala_toolchain` by default,
+# instead of the builtin toolchain, uncomment this line. You can also specify it
+# on demand via:
+#
+#   bazel build --extra_toolchains=//:toolchain_from_build_file //...
+#
+#register_toolchains("//:toolchain_from_build_file")

examples/twitter_scrooge/Thrift1.thrift

@@ -0,0 +1,7 @@
+namespace java examples.twitter_scrooge
+
+include "Thrift2.thrift"
+
+struct Struct1 {
+  1: Thrift2.Struct2 msg
+}

examples/twitter_scrooge/Thrift2.thrift

@@ -0,0 +1,5 @@
+namespace java examples.twitter_scrooge
+
+struct Struct2 {
+  1: string msg
+}