Kmp Source Sets for gRPC

[Mr3zee]

Subsystem
Gradle Plugin

Problem Description
Current solution uses com.google.protobuf plugin.
It is limiting in different ways:

• No support of KMP sources (and it is hard to add on top of it)
• Limited customisation - no lint, no breaking changes detection
• Somewhat hard Gradle plugin integration

Options:

• Custom Gradle plugin that manages protoc binary
  • Most flexible (+)
  • Most work to do (-)
• builf.buf Gradle plugin
  • All perks of Buf - linter, generator, breaking changes detector, etc (+)
  • Subpar Gradle plugin (-)
    • Users need to manage custom Buf config files
    • Tasks are not source set independent - one task per module.
    • No usage of Gradle Provider API

Solution
Blend of Buf's great capabilities to manage proto files and our Gradle plugin to manage Buf.

We use Buf's executable but we configure its workspace using our Gradle tasks:

• We generate a workspace per source set
• For Kotlin/JVM projects we use src/main/proto and src/test/proto directory sets
• for Kotlin/Multilptaform we use src/jvmMain/proto and src/jvmTest/proto directory sets
  • Will be replaced with src/commonMain/proto and src/commonTest/proto directory sets later
• Other source directory sets are NOT used (nativeMain, jsMain, etc)
• Each sourceSet can be customised using protoSourceSets extension
  • include/exclude files as in a regular source set protoSourceSets.jvmTest { proto.exclude("exclude/**") }
  • Add custom protoc plugins using protoSourceSets.jvmTest { protocPlugin(rpc.grpc.protocPlugins.kxrpc) }
    • Predifined plugins are added by default
• Each source set is a separate Buf workspace
  • It has a generated buf.yaml and buf.gen.yaml files
  • buf.yaml defines modules:
    • proto for files to be used for generation (in both main and test sets)
    • import for files used as imported (resolvable but not used for generation) - used in test sets to import proto files from main
  • buf.gen.yaml defines plugins:
    • Uses plugins from the rpc.grpc.protocPlugins container.
      • Predefined kxrpc, protobufJava, grpcJava and grpcKotlin
    • Plugins can be customized with both protoc options and Buf options using ProtocPlugin class
    • Plugins can be marked as generating java code to be automatically added to java source sets
    • New plugins can be added in rpc.grpc.protocPlugins container.
  • Each workspace has dedicated tasks:
    • generateBufYaml<sourceSet> - generates buf.yaml
    • generateBufGenYaml<sourceSet> - generates buf.gen.yaml
    • processProtoFiles<sourceSet> - copies proto files to the workspace directory
    • bufGenerate<sourceSet> - runs buf generate command
  • buf tasks can be customised using rpc.grpc.buf extension.
    • buf generate task can be customised using rpc.grpc.buf.generate extension.
  • Only buf generate task is supported out-of-the-box
  • Users can add custom buf tasks
    • Plain buf task (only access to the executable) - using registerBufExecTask
    • Workspace tasks (they are run one generated workspace) - using rpc.grpc.buf.tasks.registerWorkspaceTask
    • This way users can add first class support for other functions like lint without explicit support from us
• Buf output can be seen using --debug option
• As we can't publish to BSR that easily - our protoc plugin is published to Maven, dowloaded and used as a local plugin, and executed with java.home or a user specified java executable

Next steps

Comming in next PRs in no particular order and with no particular deadlines:

• Gradle plugin tests
• Android source sets
• Fixes in our protoc plugin for import types
• User managed buf workspace
• Dokka APIs for the gradle plugin
• Docs for new gradle plugin apis
• Ktor Generator plugin

[e5l]

please add link to an issue