Skip to content

Add multi-thread to ObjcExportHeaderGenerator when processing packageFragments #5564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+37 −21
Open

Add multi-thread to ObjcExportHeaderGenerator when processing packageFragments #5564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions native/objcexport-header-generator/impl/k1/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,12 @@ sourceSets {
"test" { projectDefault() }
}

repositories {
mavenCentral()
}

dependencies {
implementation(libs.kotlinx.coroutines.core)
Copy link
Contributor

@SvyatoslavScherbina SvyatoslavScherbina Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, no other part of the compiler uses kotlinx.coroutines. If this is the case, this change effectively adds a new dependency.
Can the parallelization be done using standard JDK means?

For example, another part of the K/N compiler uses java.util.concurrent.Executors:

kotlin/kotlin-native/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/driver/phases/TopLevelPhases.kt

Lines 243 to 260 in 3afe02c

val executor = Executors.newFixedThreadPool(threadsCount)
val thrownFromThread = AtomicReference<Throwable?>(null)
val tasks = fragmentsList.zip(generationStates).map { (fragment, generationState) ->
Callable {
try {
// Currently, it's not possible to initialize the correct thread on `PerformanceManager` creation
// because new threads are spawned here when `fragment` with its `PerformanceManager` is already initialized.
fragment.performanceManager?.initializeCurrentThread()
runAfterLowerings(fragment, generationState)
} catch (t: Throwable) {
thrownFromThread.set(t)
}
}
}
executor.invokeAll(tasks.toList())
executor.shutdown()
executor.awaitTermination(1, TimeUnit.DAYS)
thrownFromThread.get()?.let { throw it }

Maybe it is worth checking other parts of the Kotlin compiler to see what is common.

Copy link
Contributor Author

@alfonsocv12 alfonsocv12 Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think theres more places that are using them already. This search returns 17 build files with the dependency

Copy link
Contributor

@SvyatoslavScherbina SvyatoslavScherbina Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This search returns 17 build files with the dependency

Most of them are not in the compiler.
The ones in the compiler seem to add the dependency because some other dependencies require coroutines. Look for usages of coroutines in the source code, not in the build scripts.

api(project(":native:objcexport-header-generator"))
testImplementation(project(":compiler:cli-base"))
implementation(project(":compiler:ir.backend.common"))
Expand Down
53 changes: 32 additions & 21 deletions ...or/impl/k1/src/org/jetbrains/kotlin/backend/konan/objcexport/ObjCExportHeaderGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import org.jetbrains.kotlin.descriptors.ClassDescriptor
import org.jetbrains.kotlin.descriptors.ModuleDescriptor
import org.jetbrains.kotlin.descriptors.SourceFile
import org.jetbrains.kotlin.resolve.scopes.MemberScope
import kotlinx.coroutines.*

abstract class ObjCExportHeaderGenerator @InternalKotlinNativeApi constructor(
val moduleDescriptors: List<ModuleDescriptor>,
Expand Down Expand Up @@ -117,32 +118,42 @@ abstract class ObjCExportHeaderGenerator @InternalKotlinNativeApi constructor(
.flatMap { it.getPackageFragments() }
.makePackagesOrderStable()

packageFragments.forEach { packageFragment ->
packageFragment.getMemberScope().getContributedDescriptors()
.asSequence()
.filterIsInstance<CallableMemberDescriptor>()
.filter { mapper.shouldBeExposed(it) }
.forEach {
val classDescriptor = getClassIfCategory(it)
if (classDescriptor == null) {
topLevel.getOrPut(it.findSourceFile(), { mutableListOf() }) += it
} else {
// If a class is hidden from Objective-C API then it is meaningless
// to export its extensions.
if (!classDescriptor.isHiddenFromObjC()) {
extensions.getOrPut(classDescriptor, { mutableListOf() }) += it
val classesToTranslate = mutableListOf<MutableList<ClassDescriptor>>()

runBlocking {
packageFragments.forEach { packageFragment ->
val memberScope = packageFragment.getMemberScope()

// Keep Order
val inOrderCollector = mutableListOf<ClassDescriptor>()
classesToTranslate.add(inOrderCollector)

launch(Dispatchers.Default) {
memberScope.getContributedDescriptors()
.asSequence()
.filterIsInstance<CallableMemberDescriptor>()
.filter { mapper.shouldBeExposed(it) }
.forEach {
val classDescriptor = getClassIfCategory(it)
if (classDescriptor == null) {
topLevel.getOrPut(it.findSourceFile(), { mutableListOf() }) += it
} else {
// If a class is hidden from Objective-C API then it is meaningless
// to export its extensions.
if (!classDescriptor.isHiddenFromObjC()) {
extensions.getOrPut(classDescriptor, { mutableListOf() }) += it
}
}
}
}
}
}

val classesToTranslate = mutableListOf<ClassDescriptor>()

packageFragments.forEach { packageFragment ->
packageFragment.getMemberScope().collectClasses(classesToTranslate)
launch(Dispatchers.Default) {
memberScope.collectClasses(inOrderCollector)
}
}
}

classesToTranslate.makeClassesOrderStable().forEach { translateClass(it) }
classesToTranslate.flatten().makeClassesOrderStable().forEach { translateClass(it) }

extensions.makeCategoriesOrderStable().forEach { (classDescriptor, declarations) ->
generateExtensions(classDescriptor, declarations)
Expand Down