Skip to content

Commit 4212398

Browse files
authored
Build refactoring and cleanups (moving from build scripts to convention plugins) (#14764)
1 parent 1584c05 commit 4212398

File tree

189 files changed

+5540
-5518
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

189 files changed

+5540
-5518
lines changed

CHANGES-record.md

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
-->
17+
18+
This is a record of changes to the modernized gradle build.
19+
20+
### ```buildOptions``` plugin instead of ```propertyOrDefault``` method
21+
22+
All "propertyOrDefault*" methods have been removed and an external
23+
plugin for configuring various build options is now used. This is
24+
the plugin (blame me for any shortcomings):
25+
26+
https://github.com/carrotsearch/gradle-build-infra#plugin-comcarrotsearchgradlebuildinfrabuildoptionsbuildoptionsplugin
27+
28+
this plugin will source the value of a given build option from
29+
several places, in order:
30+
- system property (```-Dfoo=value```),
31+
- gradle property (```-Pfoo=value```),
32+
- environment variable (```foo=value ./gradlew ...```)
33+
- a *non-versioned*, ```build-options.local.properties``` property file
34+
(for your personal, local tweaks),
35+
- a *versioned* ```build-options.properties``` property file.
36+
37+
It works much like before - you can override any build option
38+
temporarily by passing, for example, ```-Ptests.verbose=true```. But you
39+
can also make such changes locally persistent by placing them
40+
in your ```build-options.local.properties``` file. Alternatively,
41+
the generated ```gradle.properties``` is also supported since it declares
42+
project properties (equivalent to ```-Pkey=value```). At some point
43+
in the future, we may want to keep gradle.properties versioned though,
44+
so it's probably a good idea to switch over to
45+
```build-options.local.properties```.
46+
47+
The biggest gain from using build options is that you can now see
48+
all declared options and their values, including their source (where their
49+
value comes from). To see what I mean, try running these:
50+
```
51+
./gradlew :buildOptions
52+
./gradlew -p lucene/core buildOptions
53+
```
54+
55+
### Lucene 'base version' string moved
56+
57+
* Various Lucene's "version" properties are now full build options (as described above).
58+
Their default values are in ```build-options.properties```. You can override
59+
any of ```version.suffix```, ```version.base``` or ```version.release```
60+
using any of the build option plugin's methods. If you'd like to bump
61+
Lucene's base version, it now lives in the versioned ```build-options.properties```.
62+
63+
### Tweaks to properties and options
64+
65+
* ```runtime.java.home``` is a build option now but the support for
66+
```RUNTIME_JAVA_HOME``` env. variable has been implemented for backward
67+
compatibility.
68+
69+
* ```tests.neverUpToDate``` option is renamed ```tests.rerun```.
70+
The value of ```true``` indicates test tasks will re-run even if
71+
nothing has changed.
72+
73+
### Changes to project structure
74+
75+
* ```build-tools/missing-doclet``` is now a regular project within Lucene (not
76+
an included composite project). This simplifies the build and ensures we apply
77+
the same checks to this subproject as we do to everything else.
78+
79+
* all gradle build scripts are converted to convention plugins (in groovy) and
80+
live under ```dev-tools/build-infra/src/main/groovy```. The long-term
81+
plan is to move some or all of these groovy plugins to Java (so that the code is
82+
easier to read and debug for non-groovians).
83+
84+
* ```tidy``` is now applied to groovy/gradle scripts (formatting is enforced).
85+
86+
### Security manager support
87+
88+
* parts of the build supporting security manager settings have been just
89+
removed, without any replacement.
90+
91+
### Other notable changes
92+
93+
* The legacy ```precommit``` task has been removed; use gradle's ```check```.
94+
95+
* Removed dependency on jgit entirely. This is replaced by forking the system's git
96+
in porcelain mode (which should be stable and portable). This logic is implemented
97+
in [this plugin](https://github.com/carrotsearch/gradle-build-infra/?tab=readme-ov-file#plugin-comcarrotsearchgradlebuildinfraenvironmentgitinfoplugin).
98+
An additional benefit is that all features of git should now work (including worktrees).
99+
100+
* Added support for owasp API keys in the form of ```validation.owasp.apikey``` build option. Owasp check is
101+
still very, very slow. We should probably just drop it.
102+
103+
* I've changed the default on ```gradle.ge``` (Gradle Enterprise, develocity) to ```false```
104+
on non-CI builds.
105+
106+
* I've tried to clean up lots and lots of gradle/groovy code related to eager initialization of
107+
tasks as well as update deprecated APIs. This isn't always straightforward and I might have broken
108+
some things...
109+
110+
### Fixes to existing issues
111+
112+
* ```gradlew clean check``` will work now (https://github.com/apache/lucene/issues/13567)

build-options.properties

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#
2+
# These are various build options that the Lucene build uses.
3+
#
4+
# PLEASE READ ./gradlew helpLocalSettings
5+
#
6+
7+
# This is the base Lucene version.
8+
version.base=11.0.0
9+
10+
# This is the "suffix" added to version.base to form the final version.
11+
version.suffix=SNAPSHOT
12+
13+
# This property is computed as ${version.base}-${version.suffix}; you
14+
# can override it manually to any x.y.z string to set an arbitrary
15+
# Lucene version.
16+
# version.release=
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
import com.diffplug.gradle.spotless.SpotlessTask
2+
3+
/*
4+
* Licensed to the Apache Software Foundation (ASF) under one or more
5+
* contributor license agreements. See the NOTICE file distributed with
6+
* this work for additional information regarding copyright ownership.
7+
* The ASF licenses this file to You under the Apache License, Version 2.0
8+
* (the "License"); you may not use this file except in compliance with
9+
* the License. You may obtain a copy of the License at
10+
*
11+
* http://www.apache.org/licenses/LICENSE-2.0
12+
*
13+
* Unless required by applicable law or agreed to in writing, software
14+
* distributed under the License is distributed on an "AS IS" BASIS,
15+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
* See the License for the specific language governing permissions and
17+
* limitations under the License.
18+
*/
19+
20+
plugins {
21+
id "java-gradle-plugin"
22+
}
23+
24+
repositories {
25+
mavenCentral()
26+
gradlePluginPortal()
27+
}
28+
29+
static Provider<String> plugin(Provider<PluginDependency> plugin) {
30+
return plugin.map {
31+
if (it.pluginId == "com.carrotsearch.gradle.buildinfra") {
32+
return "${it.pluginId}:${it.pluginId}.gradle.plugin:${it.version}".toString()
33+
} else if (it.pluginId == "de.thetaphi.forbiddenapis") {
34+
return "de.thetaphi:forbiddenapis:${it.version}".toString()
35+
} else {
36+
return "${it.pluginId}:${it.pluginId.replaceFirst(".*\\.", "")}-plugin-gradle:${it.version}".toString()
37+
}
38+
}
39+
}
40+
41+
dependencies {
42+
implementation gradleApi()
43+
implementation localGroovy()
44+
implementation deps.commons.codec
45+
implementation deps.randomizedtesting.runner
46+
47+
implementation plugin(deps.plugins.carrotsearch.buildinfra)
48+
implementation plugin(deps.plugins.forbiddenapis)
49+
implementation plugin(deps.plugins.spotless)
50+
}
51+
52+
tasks.matching {
53+
it.name in [
54+
"renderSiteJavadoc",
55+
"renderJavadoc",
56+
"validateJarChecksums",
57+
"validateJarLicenses",
58+
"collectJarInfos",
59+
"compileJava",
60+
"compileTestJava",
61+
"assemble"
62+
]
63+
}.configureEach {
64+
enabled = false
65+
}
66+
67+
// check spotless and other tasks into thinking their sources
68+
// are from build-infra.
69+
if (!isIdea) {
70+
plugins.withType(JavaPlugin).configureEach {
71+
SourceSet main = sourceSets.main
72+
main.java.srcDirs = rootProject.files("build-tools/build-infra/src/main/java")
73+
}
74+
75+
tasks.withType(SpotlessTask).configureEach {
76+
it.projectDir.set(rootProject.file("build-tools/build-infra"))
77+
it.target
78+
}
79+
}
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Placeholder file to keep the directory in git. Triggers this project
2+
to be a java-library project (and applies all plugins in order).

build-tools/build-infra/build.gradle

Lines changed: 37 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -17,37 +17,40 @@
1717

1818
plugins {
1919
id "java-gradle-plugin"
20+
id "groovy-gradle-plugin"
21+
2022
alias(deps.plugins.spotless) apply false
2123
alias(deps.plugins.forbiddenapis) apply false
2224
}
2325

2426
repositories {
2527
mavenCentral()
28+
gradlePluginPortal()
2629
}
2730

2831
group = "org.apache"
2932

30-
// Make sure the build environment is consistent.
31-
apply from: file('../../gradle/conventions.gradle')
32-
apply from: file('../../gradle/validation/check-environment.gradle')
33-
34-
// Add spotless/ tidy.
35-
tasks.register("checkJdkInternalsExportedToGradle") {}
36-
apply from: file('../../gradle/validation/spotless.gradle')
37-
apply from: file('../../gradle/validation/forbidden-apis.gradle')
38-
3933
java {
4034
sourceCompatibility = JavaVersion.toVersion(deps.versions.minJava.get())
4135
targetCompatibility = JavaVersion.toVersion(deps.versions.minJava.get())
4236
}
4337

4438
gradlePlugin {
4539
automatedPublishing = false
40+
}
4641

47-
plugins {
48-
buildInfra {
49-
id = 'lucene.build-infra'
50-
implementationClass = 'org.apache.lucene.gradle.buildinfra.BuildInfraPlugin'
42+
// Convert a plugin dependency to a regular dependency so that we can
43+
// use [plugins] section in the top-level toml but declare regular
44+
// project dependencies here.
45+
static Provider<String> plugin(Provider<PluginDependency> plugin) {
46+
return plugin.map {
47+
if (it.pluginId == "de.thetaphi.forbiddenapis") {
48+
// Uwe's forbiddenapis is on Maven Central, directly.
49+
return "de.thetaphi:forbiddenapis:${it.version}".toString()
50+
} else {
51+
// maven artifact pattern for gradle's plugin repositories.
52+
return "${it.pluginId}:${it.pluginId}.gradle.plugin:${it.version}".toString()
53+
// return "${it.pluginId}:${it.pluginId.replaceFirst(".*\\.", "")}-plugin-gradle:${it.version}".toString()
5154
}
5255
}
5356
}
@@ -56,11 +59,32 @@ dependencies {
5659
implementation gradleApi()
5760
implementation localGroovy()
5861
implementation deps.commons.codec
62+
implementation deps.randomizedtesting.runner
63+
implementation deps.rat
64+
implementation deps.zstd
65+
66+
implementation deps.flexmark.core
67+
implementation deps.flexmark.ext.abbreviation
68+
implementation deps.flexmark.ext.attributes
69+
implementation deps.flexmark.ext.autolink
70+
implementation deps.flexmark.ext.tables
71+
72+
implementation plugin(deps.plugins.carrotsearch.buildinfra)
73+
implementation plugin(deps.plugins.forbiddenapis)
74+
implementation plugin(deps.plugins.spotless)
75+
implementation plugin(deps.plugins.owasp.dependencycheck)
76+
implementation plugin(deps.plugins.undercouch.download)
77+
implementation plugin(deps.plugins.errorprone)
5978
}
6079

80+
def hasJavaFlightRecorder = ModuleLayer.boot().findModule('jdk.jfr').map{ otherModule ->
81+
this.getClass().module.canRead(otherModule)
82+
}.orElse(false)
6183
if (!hasJavaFlightRecorder) {
6284
logger.warn('Module jdk.jfr is not available; skipping compilation of Java Flight Recorder support.')
6385
tasks.named('compileJava').configure {
6486
exclude('**/ProfileResults.java')
6587
}
6688
}
89+
90+
tasks.register("tidy", {})

build-tools/build-infra/settings.gradle

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,9 @@
1818
rootProject.name = 'build-infra'
1919

2020
dependencyResolutionManagement {
21-
versionCatalogs {
22-
deps {
23-
from(files('../../gradle/libs.versions.toml'))
24-
}
21+
versionCatalogs {
22+
deps {
23+
from(files('../../gradle/libs.versions.toml'))
2524
}
25+
}
2626
}

gradle/hacks/gradle-archives.gradle renamed to build-tools/build-infra/src/main/groovy/lucene.all-projects.conventions.gradle

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,33 @@
1313
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1414
* See the License for the specific language governing permissions and
1515
* limitations under the License.
16-
*/
16+
*/
1717

18-
// LUCENE-9518: Set up sane defaults for any gradle archiving task.
19-
allprojects {
20-
tasks.withType(AbstractArchiveTask).configureEach { task ->
18+
// Common setup for all projects.
19+
20+
if (project != project.rootProject) {
21+
throw new GradleException("Applicable to rootProject only: " + project.path)
22+
}
23+
24+
allprojects {project ->
25+
// All projects have the base plugin (convention tasks like 'check', etc.)
26+
apply plugin: 'base'
27+
28+
// Project group and version.
29+
group = "org.apache.lucene"
30+
version = rootProject.version
31+
32+
// Source repositories for dependencies.
33+
repositories {
34+
mavenCentral()
35+
}
36+
37+
// Common archive artifact naming.
38+
var baseExt = project.getExtensions().getByType(BasePluginExtension)
39+
baseExt.archivesName.convention(project.path.replaceAll("^:", "").replace(':', '-'))
40+
41+
// Common support for reproducible builds.
42+
tasks.withType(AbstractArchiveTask).configureEach {
2143
duplicatesStrategy = DuplicatesStrategy.FAIL
2244
preserveFileTimestamps = false
2345
reproducibleFileOrder = true
@@ -28,4 +50,10 @@ allprojects {
2850
it.unix(0644)
2951
}
3052
}
53+
54+
// Common tasks or meta-tasks.
55+
tasks.register("tidy", {
56+
description "Applies all code formatters and other enforced cleanups to the project."
57+
group "verification"
58+
})
3159
}

0 commit comments

Comments
 (0)