Introduce JKlib entry point

[nikita-nazarov]

Hi @erokhins, @kunyavskiy!

This PR is a preliminary version of the Kotlin compiler entry point for J2CL.
Currently there is a lot of mocking going around and there are a couple of workarounds introduced, so I believe some of the Kotlin API has to be modified to make this code cleaner. But anyway this PR should give you an impression of what we would like to achieve.

The ultimate goal would be to put this extension point in the Kotlin compiler repo together with some IR text tests, so that it would be possible to catch bugs early and to perform the required API migrations as fast as possible.

Here is a link to the related document.

Context

Currently, to transpile Kotlin code, J2CL acts as a compiler plugin. It is facing challenges in inlining Kotlin functions due to its reliance on the experimental and unsupported -Xserialize-ir compiler flag to serialize the IR of a library in its corresponding header JAR file. This approach has led to issues such as orphaned IR nodes and incomplete IR serialization, particularly in complex scenarios like transitive inline calls. Additionally, compiling the standard library (stdlib) with this flag has presented further complications.These issues needed to be addressed before the wider adoption of J2CL’s Kotlin frontend. In that context, the Kotlin compiler team at JetBrains proposed the following solution that relies on klibs.

The Klib pipeline

With this extension point J2CL transpilation happens roughly the following way:

1. Given a set of source files and jar and klib dependencies, compile the source module to a klib (using K2 frontend).
2. Deserialise the klib (using K1) and link the IR, then give the resulting IR to J2CL.
3. At J2CL run custom IR lowerings
4. Perform the finalised IR -> Closure transpilation

Workarounds

When trying to implement this klib based pipeline we have encountered numerous issues (kudos to @kunyavskiy to help us fix them!), which made us introduce a couple of workarounds that are best described in this document:

1. Instead of writing Kotlin signatures of imported Java functions to a klib, we write plain Java signatures to address signature mismatches between K1 and K2.
   For example given this code:

fun <T, U : Comparable<U>> Comparator<T>.foo(
  keyExtractor: Function<in T, out U>
): Comparator<T> =
  thenComparing(keyExtractor)

The imported thenComparing function will have different signatures when serializing to klib with K2, and when deserializing with K1:

thenComparing(...){0§<kotlin.Comparable<in|0:0?>?>} // K2 view
thenComparing(...){0§<kotlin.Comparable<0:0?>?>} // K1 view

To fix this issue, we put this signature to a klib:

// Imported signatures:
java/util/Comparator.thenComparing(Ljava/util/function/Function;)Ljava/util/Comparator;

2. When deserializing Java method signatures of a mapped type, e.g.:

// Imported signatures:
kotlin/Throwable.getSuppressed|getSuppressed(){}[0]

We encountered issues when trying to resolve this declaration, because kotlin.Throwable doesn't define getSuppressed in source code. This method comes from the corresponding java.lang.Throwable class. This made us introduce this workaround, which caches the IR symbols of Java declarations of mapped types.

[nikita-nazarov]

The entire file is copy pasted from the corresponding Kotlin counterparts, except for the mangleTypeArgumentsUsingEffectiveVariance methods. This is needed to allow always using effective variance during mangling like mentioned in this comment .

[nikita-nazarov]

Most of the file is copied. The difference is in the signatureString methods that yield JVM signatures for Java methods.

[nikita-nazarov]

This function is mostly copied from org.jetbrains.kotlin.load.kotlin.MethodSignatureMappingKt#computeJvmDescriptor, except for the workaround described in the comment below.

[nikita-nazarov]

This is the only different line from org.jetbrains.kotlin.load.kotlin.MethodSignatureMappingKt#computeJvmSignature. The original:

 (original as? SimpleFunctionDescriptor ?: return null).computeJvmDescriptor()

[udalov]

Just in case, the original function FunctionDescriptor.computeJvmDescriptor seems like some ad-hoc implementation that is only suitable for some specific purpose, and it would be risky to use it to compute JVM signatures of arbitrary declarations. For example, it doesn't support @JvmName, context parameters, constructors of inner/inline/sealed classes, and probably a lot more.

[nikita-nazarov]

This is the workaround for mapped types that I mentioned in the description of this PR

[nikita-nazarov]

We are mocking an IrPluginContext here, because J2CL is currently operating with this type of context, but this can be changed in the future.

[nikita-nazarov]

This is the method that is called by J2CL to get the wanted IR.

[nikita-nazarov]

This method performs the "first stage" of the compilation - compilation to klib

[nikita-nazarov]

This method performs the "second stage" - deserialze the klib and link IR.

[erokhins]

Hey @nikita-nazarov ! Thanks a lot for sharing code and all relevant info. I'll be able to take a look at this in details next week.

[erokhins]

Hi @nikita-nazarov! As promised, here is the result of preliminary pre-review/internal discussions.
Below is our proposal for moving forward -- it has already been approved internally. I can imagine you would like to change something/have questions about how it will work—don't hesitate to ask.

1. We are fine to take something like this MR into our repo, but with some limitations, see below.
2. We don't want to have this new BE as part of the distribution (i.e. publication to the maven central), because you are the only client of this new BE and you will be building it from sources anyway (with the rest of the K2CL pipeline).
3. You should add some tests that will be running in our repo -- proposal -- create new IRValidator and run tests on JVM black box tests -- i.e. compile to the klib and then -- deserialize it and check with IRValidator that IR is linked + maybe some custom properties of the deserialized IR that is important for the rest of the K2CL pipeline
4. This part -- serialization + deserialization + special rules for JVM signatures in klib world -- we will take to the Kotlin repo, the rest of the K2CL backend -- not a part of this agreement. (You haven't asked for this in the MR, just a note for any future discussions). I.e., custom lowerings and especially codegen are out of scope.
5. Code should be deduplicated -- i.e. you could add some extensibility to the already existing components and re-use them. You don't need to create extension points -- you could just add open methods and extend already existing classes.
6. To make 2) possible, please create a separate module and put all the corresponding code there + tests.
   6.1. For this module, we will have the CODEOWNERS set up in a way that both JetBrains and Google are owners of this code. This is preferable because we need to be able to fix problems quickly (i.e. I if we are doing some refactorings, we don't want to be blocked by waiting for review from you).
   6.2. Please create a README for this folder -- for some random people, so they will know what is going on :)
7. Before the merge, the following set of people from our side should review the code:

• @kunyavskiy -- TechLead of the whole Kotlin Compiler
• @udalov -- JVM TechLead
• @ddolovov -- Common BE TechLead (klib serialization/deserialization is the responsibility of this team)

8. We are going to fix failing tests, but we cannot guarantee that new language features will be automatically supported -- this is your area of responsibility. I.e., if that is a matter of just enabling test -- we will enable it, but if there is something non-trivial -- the best we can do is to communicate to you that you are going to have a problem with it in this place, once you decide to support this new language feature.
9. As always, we are ready to help you with problems that will arise along the way -- feel free to ask us any questions.

P.s. I'm going to be on vacation till December 9. With technical problems -- default gateway @kunyavskiy. With organizational questions, Pavel could also try to help. By default, I will return on December 9 and address all remaining organizational questions.