Gather more test cases and a better working definition of "stability"

[retronym]

I'm gathering test cases for unstable output for Scalac with a view to improving the compiler to make them stable. See the README for the tests I've come up with so far.

This ticket is a 🦇 -signal to some people that might be interested in this effort, and hopefully might be able to help me come up with additional test cases.

I'm also interested in what "stability" means to you.

@dragos (who fixed an bug if this ilk in scala/scala#5965 and might be able to exploit from stability in https://triplequote.com/hydra)
@johnynek @ittaiz (from https://github.com/bazelbuild/rules_scala)
@jvican (from Scala Center, and who I recall is interested in this topic)
@eed3si9n / @dwijnand (from sbt)

[dragos]

Great initiative! I have a WIP branch for lamdbalift ordering, I thought I filed a ticket already but it seems I missed it. Anyway, maybe this helps: dragos/scala@7c0f351

[dragos]

Stability, for us, means that bytecode should be identical for the same inputs, where "same" allows some minor differences:

• the order of source files need not be the same
• you can recompile a subset of all project files as long as the other files were compiled previously, didn't change, and are on the classpath (touch-ing some files and recompiling should give exactly the same results)

[johnynek]

I would like a bit simpler (I think) definition. To have a resident compiler in bazel, we have a process that bazel manages (called a "worker") that gets passed the arguments to the compiler run. We then allocate a new scala.tools.nsc.MainClass and just pass it Array[String] args to the process method:

https://github.com/bazelbuild/rules_scala/blob/master/src/java/io/bazel/rulesscala/scalac/ScalacProcessor.java#L237

I would like the contract that if you do this multiple times, if the input args are identical (that Array[String]), then the output to be bit-for-bit identical (and for the names of the files generated to be identical).

Note, this is not currently true for us. We occasionally see reproducibility bugs (as we call them) and have yet to diagnose the cause.

In the most extreme case, bazel took the output from a scalac that did not error, but the jar we got was incorrect, such that downstream compilations failed. But bazel didn't rerun the compilation because its inputs had not changed (and it even got cached on our CI).

We don't know if this bug is in bazel (highly unlikely since google uses it at such scale and does heavily cache), in the bazel scala rules somehow (again, we can't see how we can mess it up, but it of course is possible), or if scalac has some state that is not removed by allocating a new MainClass. It is very hard for us to be sure about the last one.

[eed3si9n]

I think any effort towards making compilation a pure function would be great.
For the big picture/why, this matters for the purpose of both caching and verification. https://reproducible-builds.org/ says:

Reproducible builds are a set of software development practices that create a verifiable path from human readable source code to the binary code used by computers.

Related to this might be various use of timestamps that might sneak into the final product, like in JAR (sbt/io#58). It would be nice to get good performance on incremental compilation while keeping these properties.

[johnynek]

we definitely zero out the timestamps in bazel when creating jars, but perhaps there could be some non-determinism in the class files.

We have a test for reproducibility in the bazel rules_scala, and it passes. But like I said, internally at stripe we have seen some non-reproducibility (which is almost certainly not timestamps on jars).

[retronym]

Thanks for all the input.

@dragos Thanks for the patch. I'll also try just using a LinkedHashSet there.

@johnynek If you run into that problem, could you please hold onto the pair of differing class files? A diff of the output of javap -v -private might be enough for me to figure out the problem.

@eed3si9n Glad SBT is making moves in this area at the JAR/ZIP level. Here's a related maven plugin that might serve as inspiration: https://zlika.github.io/reproducible-build-maven-plugin/

[retronym]

I've added another test for quasiquotes.

Here's my WIP branch that uses finer grained fresh names counters (one per source-file-defined method/class, rather than one per source file). It fixes all the problems identified in this repo so far.

[retronym]

/cc @smarter

[ittaiz]

@johnynek The reproducibility test still fails for me locally from time to time.
We should take a look at how to make it more fine grain since right now when it fails it's hard to know which jars differ, let alone which class files and we also don't keep a copy of them to give to @retronym

[smarter]

In Dotty, @nicolasstucki implemented idempotency tests for this sort of things: https://github.com/lampepfl/dotty/blob/master/compiler/test/dotty/tools/dotc/IdempotencyTests.scala

[retronym]

Asking @xeno-by for some background on the change to macros/quasiquotes that switched to globally scoped fresh names.

[xeno-by]

Hmmm I'm not sure I can recall this right now. @densh, maybe that was necessary for your work on quasiquotes?

[retronym]

Some progress:

I've ported the test cases from this PR into unit tests in my stable-output WIP branch.

I wrote a script that can run a stability test for any project after exporting a files with the compiler options. I have a local sbt plugin that adds the argsFile task to perform this export.

Using this script on the compiler sources themselves reveals the next problem: the order of default getter methods within the data of the @ScalaSignature annotation depends on the order that their methods were type completed. I've added a failing test for this. Perhaps the pickler could sort decls before writing, at least those of synthetic methods. (That seems to work)

I've also published a new version of jardiff, which I used to inspect and compare class files (or classpaths thereof). The new raw mode -r is useful to highlights differences that the tool normalizes away by default (such as method ordering).