Get rid of .release files

[agluszak]

Goals:

• Make MODULE.bazel a single source of truth
• have a more standard project structure and simpler release process
• make it possible to use a git_override/local_path_override

Anti-goals:

• Having to bootstrap the builder jars from scratch
• integration tests run longer on CI

[restingbull]

How would you propose avoiding building the necessary jars from scratch each time?

The current release approach is focused on providing a precompiled toolchain.

Additionally, there are pieces that are not shipped as part of the release: cli toolchain, tests, etc. These would bloat the end artifact.

[agluszak]

How would you propose avoiding building the necessary jars from scratch each time?

My idea here would be to extract the jars/toolchain into a separate artifact that could be downloaded instead of being built from scratch, similarly to https://github.com/aspect-build/toolchains_protoc. rules_rust seems to do something similar.

Additionally, there are pieces that are not shipped as part of the release: cli toolchain, tests, etc. These would bloat the end artifact.

I don't think it's that much of a problem, AFAIK every ruleset does that, these source files are just a few KB.

[restingbull]

I don't think it's that much of a problem

It does represent a regression from the current deliverable, and ends up extending public api.

I'm open to removing the current system, but the new one needs to provide measurable benefits.

[agluszak]

ends up extending public api

how so?

[restingbull]

If you include the cli rules in the bundle, they will be available for loads, etc. While not documented, it extends the possiblity of reliance on shifting apis.

They could be visibility locked, but just not shipping is cleaner.

[agluszak]

Ah, I see what you mean now. One solution I came up with that would address my pain point (having to keep release and non-release files in sync manually) is to derive the release files from non-release ones.

So, for example, we could have marker comments in build files like # RELEASE-CONTENT-START and # RELEASE-CONTENT-END and instead of swapping the development build files with release build files (that are mostly copy-pasted parts of development ones) we could have a script to cut the non-release parts

[Bencodes]

I'm generally in favor of just dropping the release archives here. It definitely creates a ton of complexity for us to deal with, especially with how we do integration testing.

I think one big advantage of simplifying this is that I think it'll simplify the backwards/forward compatibility story a bit since our users will just build the rules using the Kotlin compiler version that's being natively used in their project. It could also make it a lot easier for us to maintain compat. with multiple versions at the same time since we can do source selection during analysis, picking the correct source implementation for the given Kotlin compiler version. We'd need to do some restructuring inside the rules to make this more possible.

[restingbull]

So you do enjoy building protoc from source?

[Bencodes]

So you do enjoy building protoc from source?

Not particularly but we will have to deal with that with or without the changes being discussed in this PR because of https://github.com/bazelbuild/bazel-worker-api and all of the different rules that depend on bazel-worker-api. And with local/remote build caching the upfront cost is minimal.

[Bencodes]

The other downside to discuss here is that we'd be shifting the maven syncing out of the pre-compiled release process and onto the end user. You can eliminate most of the costs by using rules_jvm_external pinning though. https://github.com/bazelbuild/rules_kotlin/blob/master/MODULE.bazel#L78-L117.

We'd have more justification to drop dagger from the rules as well if we go down this route so that the compilation process is more streamlined.

[agluszak]

So you do enjoy building protoc from source?

Not particularly but we will have to deal with that with or without the changes being discussed in this PR because of https://github.com/bazelbuild/bazel-worker-api and all of the different rules that depend on bazel-worker-api. And with local/remote build caching the upfront cost is minimal.

What about #1438 though?

[agluszak]

Let me whip up a POC to see if it solves the integration test issue

[restingbull]

To be honest, we should drop maven entirely if you want to go the compile from scratch route.

Dropping the release archiving pushes complexity to the end user (they have the joyous experience of pulling and managing all of the rule dependencies in addition to their own), and it increases build times. Caching only fixes so much. This was one of the reasons the rules are pushed as a separate archive in the first place.

I know many rulesets do build from scratch, but many of them are either ported from Google or aping those that were. In Google3, building everything from scratch was fine. In fact, ideal. Bandwidth was large, and compute was basically free. (That still didn't stop the core libraries team from trying to segment and release -- to improve build times and reduce churn. It didn't stick.) Outside of Google3, it's a far less enjoyable experience.

Most argument I hear about moving away from the release archive structure come from people actively changing the rules. Since that audience is smaller....