Add expression compiler (#22597)

This PR migrates the expression compiler from
[scalacenter/scala-debug-adapter](https://github.com/scalacenter/scala-debug-adapter)
to the scala3 repository. This change was agreed upon a few months ago
during a core meeting to simplify the release process. By doing so, we
eliminate the need to release parts of
[scalacenter/scala-debug-adapter](https://github.com/scalacenter/scala-debug-adapter)
during each Scala 3 release.

The `ExpressionCompiler` and `ExpressionCompilerBridge` are now parts of
the main compiler module. This integration streamlines usage for
external tools like sbt, Metals, Bloop, IJ Scala plugin. They can now
rely on the compiler JAR directly without needing to download additional
artifacts.

## The expression compiler

The expression compiler extends the main compiler with 3 phases:

**InsertExpression**:
  - Just after parsing
  - Parses and inserts the expression in the original source tree.
- Inserts the expression class, with a placeholder `def evaluate: Any`
method.

**ExtractExpression**:
- After `Typer`, the pickler phases, `ExtensionMethods` and
`ElimByName`.
- Extracts the expression before `LambdaLift` to prevent interference:
the expression itself can change how lambdas are lifted. Notably it can
contains lambda that needs to be lifted to the expression class.
- Replaces references to local variables and inaccessible members with
magic calls to reflectEval, annotated with ReflectEvalStrategy
attachments.

**ResolveReflectEval**:
- At the end of the transform phases
- Transforms `reflectEval` calls into actual reflective calls based on
`ReflectEvalStrategy` metadata.

## Usage of the expression compiler

The expression compiler powers the debug console in Metals and the IJ
Scala plugin, enabling evaluation of arbitrary Scala expressions at
runtime (even macros). The expression compiler produce class files that
can be loaded the running Scala program, to compute the evaluation
output.

The `ExpressionCompilerBridge` can be invoked by reflection with the
following parameters:
- `outputDir: Path`: Directory where compiled classes are written
- `classPath: String`: Classpath used during compilation
- `options: Array[String]`: Compiler options
- `sourceFile: Path`: Source file where the expression is evaluated
- `config: ExpressionCompilerConfig`:
  - `packageName`: Package name of the current evaluation frame
- `outputClassName: String`: Name of the main generated class, to be
loaded by the debuggee
  - `breakpointLine: Int`: Line number where the expression is evaluated
  - `expression: String`: Expression to compile
- `localVariables: Set[String]`: Set of visible local variables (used
for checking evaluation of captured variables)
  - `errorReporter: Consumer[String]`: Callback for reporting errors
  
The `ExpressionCompilerConfig` is designed as a factory class to allow
binary-compatible evolution across versions, enabling
cross-compatibility between different debugger and compiler versions.
 
 ## The debug tests
 
A significant part of this PR is the migration of
[ScalaEvaluationTests](https://github.com/scalacenter/scala-debug-adapter/blob/main/modules/tests/src/test/scala/ch/epfl/scala/debugadapter/ScalaEvaluationTests.scala),
notably the integration with the vulpix test infrastructure:
- In `dotty.tools.vulpix.RunnerOrchestration`:
  - added the `debugMode` to start runners with JVM debugging options
- added the `debugMain` method to connect a debugger, start a main
method and execute the debug steps
- Added `dotty.tools.debug.Debugger`: a lightweight implementation of a
JVM debugger
- Added `dotty.tools.debug.ExpressionEvaluator`: to compile and evaluate
expressions inside `dotty.tools.debug.Debugger`
- Added `dotty.tools.debug.DebugStepAssert`: to describe a debug
scenario as a series of steps (`break`, `step`, `next` and `eval`)
- Added `dotty.tools.debug.DebugTests` test class:
  - it reads all Scala files and folders from `tests/debug`
- for each Scala file, it parses its associated `.check` file as a
series of `DebugStepAssert`
- for each Scala file, it compiles it, runs it with `debugMain`, and
executes the debug steps

I migrated all relevant tests from
[ScalaEvaluationTests](https://github.com/scalacenter/scala-debug-adapter/blob/main/modules/tests/src/test/scala/ch/epfl/scala/debugadapter/ScalaEvaluationTests.scala)
to `tests/debug` folder. Thanks to Vulpix parallel execution, the 47
debug tests now complete in approximately 30-40 seconds.

Author: adpi2

.gitignore

@@ -62,7 +62,6 @@ testlogs/
 
 # Put local stuff here
 local/
-compiler/test/debug/Gen.jar
 
 /bin/.cp
 

compiler/src/dotty/tools/debug/ExpressionCompiler.scala

@@ -0,0 +1,31 @@
+package dotty.tools.debug
+
+import dotty.tools.dotc.Compiler
+import dotty.tools.dotc.core.Contexts.Context
+import dotty.tools.dotc.core.Phases.Phase
+import dotty.tools.dotc.transform.ElimByName
+
+/**
+  * The expression compiler powers the debug console in Metals and the IJ Scala plugin,
+  * enabling evaluation of arbitrary Scala expressions at runtime (even macros).
+  * It produces class files that can be loaded by the running Scala program,
+  * to compute the evaluation output.
+  * 
+  * To do so, it extends the Compiler with 3 phases:
+  *  - InsertExpression: parses and inserts the expression in the original source tree
+  *  - ExtractExpression: extract the typed expression and places it in the new expression class
+  *  - ResolveReflectEval: resolves local variables or inacessible members using reflection calls
+  */
+class ExpressionCompiler(config: ExpressionCompilerConfig) extends Compiler:
+
+  override protected def frontendPhases: List[List[Phase]] =
+    val parser :: others = super.frontendPhases: @unchecked
+    parser :: List(InsertExpression(config)) :: others
+
+  override protected def transformPhases: List[List[Phase]] =
+    val store = ExpressionStore()
+    // the ExtractExpression phase should be after ElimByName and ExtensionMethods, and before LambdaLift
+    val transformPhases = super.transformPhases
+    val index = transformPhases.indexWhere(_.exists(_.phaseName == ElimByName.name))
+    val (before, after) = transformPhases.splitAt(index + 1)
+    (before :+ List(ExtractExpression(config, store))) ++ (after :+ List(ResolveReflectEval(config, store)))

compiler/src/dotty/tools/debug/ExpressionCompilerBridge.scala

@@ -0,0 +1,38 @@
+package dotty.tools.debug
+
+import java.nio.file.Path
+import java.util.function.Consumer
+import java.{util => ju}
+import scala.jdk.CollectionConverters.*
+import scala.util.control.NonFatal
+import dotty.tools.dotc.reporting.StoreReporter
+import dotty.tools.dotc.core.Contexts.Context
+import dotty.tools.dotc.Driver
+
+class ExpressionCompilerBridge:
+  def run(
+      outputDir: Path,
+      classPath: String,
+      options: Array[String],
+      sourceFile: Path,
+      config: ExpressionCompilerConfig
+  ): Boolean =
+    val args = Array(
+      "-d",
+      outputDir.toString,
+      "-classpath",
+      classPath,
+      "-Yskip:pureStats"
+      // Debugging: Print the tree after phases of the debugger
+      // "-Vprint:insert-expression,resolve-reflect-eval",
+    ) ++ options :+ sourceFile.toString
+    val driver = new Driver:
+      protected override def newCompiler(using Context): ExpressionCompiler = ExpressionCompiler(config)
+    val reporter = ExpressionReporter(error => config.errorReporter.accept(error))
+    try
+      driver.process(args, reporter)
+      !reporter.hasErrors
+    catch
+      case NonFatal(cause) =>
+        cause.printStackTrace()
+        throw cause

compiler/src/dotty/tools/debug/ExpressionCompilerConfig.scala

@@ -0,0 +1,65 @@
+package dotty.tools.debug
+
+import dotty.tools.dotc.ast.tpd.*
+import dotty.tools.dotc.core.Symbols.*
+import dotty.tools.dotc.core.Types.*
+import dotty.tools.dotc.core.Names.*
+import dotty.tools.dotc.core.Flags.*
+import dotty.tools.dotc.core.Contexts.*
+import dotty.tools.dotc.core.SymUtils
+
+import java.{util => ju}
+import ju.function.Consumer
+
+class ExpressionCompilerConfig private[debug] (
+  packageName: String,
+  outputClassName: String,
+  private[debug] val breakpointLine: Int,
+  private[debug] val expression: String,
+  private[debug] val localVariables: ju.Set[String],
+  private[debug] val errorReporter: Consumer[String],
+  private[debug] val testMode: Boolean
+):
+  def this() = this(
+    packageName = "",
+    outputClassName = "",
+    breakpointLine = -1,
+    expression = "",
+    localVariables = ju.Collections.emptySet,
+    errorReporter = _ => (),
+    testMode = false,
+  )
+
+  def withPackageName(packageName: String): ExpressionCompilerConfig = copy(packageName = packageName)
+  def withOutputClassName(outputClassName: String): ExpressionCompilerConfig = copy(outputClassName = outputClassName)
+  def withBreakpointLine(breakpointLine: Int): ExpressionCompilerConfig = copy(breakpointLine = breakpointLine)
+  def withExpression(expression: String): ExpressionCompilerConfig = copy(expression = expression)
+  def withLocalVariables(localVariables: ju.Set[String]): ExpressionCompilerConfig = copy(localVariables = localVariables)
+  def withErrorReporter(errorReporter: Consumer[String]): ExpressionCompilerConfig = copy(errorReporter = errorReporter)
+
+  private[debug] val expressionTermName: TermName = termName(outputClassName.toLowerCase.toString)
+  private[debug] val expressionClassName: TypeName = typeName(outputClassName)
+
+  private[debug] def expressionClass(using Context): ClassSymbol =
+    if packageName.isEmpty then requiredClass(outputClassName)
+    else requiredClass(s"$packageName.$outputClassName")
+
+  private[debug] def evaluateMethod(using Context): Symbol =
+    expressionClass.info.decl(termName("evaluate")).symbol
+
+  private def copy(
+    packageName: String = packageName,
+    outputClassName: String = outputClassName,
+    breakpointLine: Int = breakpointLine,
+    expression: String = expression,
+    localVariables: ju.Set[String] = localVariables,
+    errorReporter: Consumer[String] = errorReporter,
+  ) = new ExpressionCompilerConfig(
+    packageName,
+    outputClassName,
+    breakpointLine,
+    expression,
+    localVariables,
+    errorReporter,
+    testMode
+  )

compiler/src/dotty/tools/debug/ExpressionReporter.scala

@@ -0,0 +1,17 @@
+package dotty.tools.debug
+
+import dotty.tools.dotc.core.Contexts.*
+import dotty.tools.dotc.reporting.AbstractReporter
+import dotty.tools.dotc.reporting.Diagnostic
+
+private class ExpressionReporter(reportError: String => Unit) extends AbstractReporter:
+  override def doReport(dia: Diagnostic)(using Context): Unit =
+    // Debugging: println(messageAndPos(dia))
+    dia match
+      case error: Diagnostic.Error =>
+        val newPos = error.pos.source.positionInUltimateSource(error.pos)
+        val errorWithNewPos = new Diagnostic.Error(error.msg, newPos)
+        reportError(stripColor(messageAndPos(errorWithNewPos)))
+      case _ =>
+        // TODO report the warnings in the expression
+        ()

compiler/src/dotty/tools/debug/ExpressionStore.scala

@@ -0,0 +1,24 @@
+package dotty.tools.debug
+
+import dotty.tools.dotc.ast.tpd.*
+import dotty.tools.dotc.core.Symbols.*
+import dotty.tools.dotc.core.Types.*
+import dotty.tools.dotc.core.Names.*
+import dotty.tools.dotc.core.Flags.*
+import dotty.tools.dotc.core.Contexts.*
+import dotty.tools.dotc.core.SymUtils
+
+private class ExpressionStore:
+  var symbol: TermSymbol | Null = null
+  // To resolve captured variables, we store:
+  // - All classes in the chain of owners of the expression
+  // - The first local method enclosing the expression
+  var classOwners: Seq[ClassSymbol] = Seq.empty
+  var capturingMethod: Option[TermSymbol] = None
+
+  def store(exprSym: Symbol)(using Context): Unit =
+    symbol = exprSym.asTerm
+    classOwners = exprSym.ownersIterator.collect { case cls: ClassSymbol => cls }.toSeq
+    capturingMethod = exprSym.ownersIterator
+      .find(sym => (sym.isClass || sym.is(Method)) && sym.enclosure.is(Method)) // the first local class or method
+      .collect { case sym if sym.is(Method) => sym.asTerm } // if it is a method