Skip to content

Commit dcb875a

Browse files
authored
Merge pull request quarkusio#35918 from iocanel/wait-for-config-and-doc
Document init tasks
2 parents c9c0d5a + aeab567 commit dcb875a

File tree

15 files changed

+260
-32
lines changed

15 files changed

+260
-32
lines changed

docs/src/main/asciidoc/flyway.adoc

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,7 @@ You can find more information about this feature in the xref:hibernate-orm.adoc#
285285

286286
== Flyway on Kubernetes
287287
Sometimes, it's helpful not to execute Flyway initialization on each application startup. One such example is when deploying
288+
288289
on Kubernetes, where it doesn't make sense to execute Flyway on every single replica. Instead it's desirable to execute it
289290
once and then start the actual application without Flyway. To support this use case, when generating manifests for Kubernetes
290291
the generated manifests contain a Kubernetes initialization `Job` for Flyway.
@@ -312,15 +313,15 @@ To change the `wait-for` image which by default is `groundnuty/k8s-wait-for:no-r
312313

313314
[source,properties]
314315
----
315-
quarkus.kubernetes.init-task-defaults.wait-for-image=my/wait-for-image:1.0
316+
quarkus.kubernetes.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
316317
----
317318

318319
or on Openshift:
319320

320321

321322
[source,properties]
322323
----
323-
quarkus.openshift.init-task-defaults.wait-for-image=my/wait-for-image:1.0
324+
quarkus.openshift.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
324325
----
325326

326327
**Note**: In this context globally means `for all extensions that support init task externalization`.
Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
////
2+
This guide is maintained in the main Quarkus repository
3+
and pull requests should be submitted there:
4+
https://github.com/quarkusio/quarkus/tree/main/docs/src/main/asciidoc
5+
////
6+
= Initialization tasks
7+
:categories: initialization
8+
:summary: This reference guide explains how to configure initialization tasks
9+
10+
There are often initialization tasks performed by Quarkus extensions that are meant to be run once.
11+
For example, Flyway or Liquibase initialization falls into that category. But what happens when the scaling
12+
needs of an application requires more instances of the application to run? Or what happens when the application
13+
restarts ?
14+
15+
A common environment where both of these cases are pretty common is Kubernetes. To address these challenges,
16+
Quarkus allows externalization of such tasks as Kubernetes https://kubernetes.io/docs/concepts/workloads/controllers/job/[Jobs] and uses https://kubernetes.io/docs/concepts/workloads/pods/init-containers/[init containers] to ensure that an
17+
application instance only starts once the initialization jobs have finished. With this approach even if an
18+
application has multiple replicas, the initialization logic will only run once.
19+
20+
This approach is reflected in the manifests generated by xref:kubernetes.adoc[Kubernetes extension].
21+
22+
== Disabling the feature
23+
24+
The feature can be explictily disabled per task (enabled by default).
25+
The default behavior can change by setting the following property to `false`:
26+
27+
[source,properties]
28+
----
29+
quarkus.kubernetes.init-task-defaults.enabled=false
30+
----
31+
32+
or on Openshift:
33+
34+
[source,properties]
35+
----
36+
quarkus.openshift.init-task-defaults.enabled=false
37+
----
38+
39+
**Note**: All the configuration options in this guide are available on both OpenShift and Kubernetes. The rest of the guide will use Kubernetes(`quarkus.kubernetes` prefix)
40+
configuration prefix, but all the configuration options are also available for OpenShift(`quarkus.openshift` prefix) too.
41+
42+
In the case where we need to disable a particular task, we can use the following property:
43+
44+
[source,properties]
45+
----
46+
quarkus.kubernetes.init-tasks."<task name>".enabled=false
47+
----
48+
49+
The task name is the name of the extension that performs the initialization.
50+
Examples:
51+
52+
For Flyway:
53+
54+
[source,properties]
55+
----
56+
quarkus.kubernetes.init-tasks.flyway.enabled=false
57+
----
58+
59+
For Liquibase:
60+
61+
[source,properties]
62+
----
63+
quarkus.kubernets.init-tasks.liquibase.enabled=false
64+
----
65+
66+
For Liquibase Mongodb:
67+
68+
[source,properties]
69+
----
70+
quarkus.kubernetes.init-tasks.liquibase-mongodb.enabled=false
71+
----
72+
73+
74+
== Controlling the generated job
75+
76+
The job container is pretty similar to the application container, and the only thing that changes is the configured environment variables.
77+
More specifically, the following environment variable is added, to tell the job to exit right after initialization.
78+
79+
[source,properties]
80+
----
81+
QUARKUS_INIT_AND_EXIT=true
82+
----
83+
84+
The image, image pull policy, service account, volumes, mounts and additional environment variables are inherited/copied from the deployment resource.
85+
Any customization to the original deployment resource (via configuration or extension) will also be reflected in the job.
86+
87+
== Controlling the generated init container
88+
89+
The name of the generated init container is `wait-for-${task name}` by default.
90+
Given that the init container is part of the same pod as the actual application it will get the same service account (and therefore permissions) and volumes as the application.
91+
Further customization to the container can be done using using the configuration options for init containers (see `quarkus.kubernetes.init-containers` or `quarkus.openshift.init-containers`).
92+
93+
Examples:
94+
95+
To set the imagePullPolicy to `IfNotPresent` on the init container that waits for the `flyway` job:
96+
97+
[source,properties]
98+
----
99+
quarkus.kubernetes.init-containers.wait-for-flyway.image-pull-policy=IfNotPresent
100+
----
101+
102+
To set custom command (say `custom-wait-for`) on the init container that waits for the `flyway` job:
103+
104+
[source,properties]
105+
----
106+
quarkus.kubernetes.init-containers.wait-for-flyway.command=custom-wait-for
107+
----
108+
109+
110+
== Orchestration of the initialization tasks
111+
112+
The deployment resource should not start until the job has been completed. The typical pattern that is used among Kubernetes users is the
113+
use of init containers to achieve this. An init container that `wait for` the job to complete is enough to enforce that requirement.
114+
115+
=== Using a custom wait-for container image
116+
117+
To change the `wait-for` image which by default is `groundnuty/k8s-wait-for:no-root-v1.7` you can use:
118+
119+
[source,properties]
120+
----
121+
quarkus.kubernetes.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
122+
----
123+
124+
To change the `wait-for` image for a particular init container (e.g. `wait-for-flway`) you can use:
125+
126+
[source,properties]
127+
----
128+
quarkus.kubernetes.init-containers.wait-for-flyway=my/wait-for-image:1.0
129+
----
130+
131+
=== Configuring permissions
132+
133+
For an init container to be able to perform the `wait for job` it needs to be able to perform `get` operations on the job resource.
134+
This is done automatically and the generated manifests include the required `Role` and `RoleBinding` resources.
135+
136+
If for any reason additional permissions are required either by the init container or the job, they can be configured with through the xref:deploying-to-kuberentes.adoc#generating-rbac-resources[Kubernetes RBAC configuration].
137+
138+
**Note**: The application, the init container and the job use the same `ServiceAccount` and therefore, share the same permissions.
139+
140+
== Extension providing Initialization Tasks
141+
142+
Currently, this feature is used by the following extensions:
143+
- xref:flyway.adoc[Flyway]
144+
- xref:liquibase.adoc[Liquibase]
145+
- xref:liquibase-mongodb.adoc[Liquibase MongoDB]

docs/src/main/asciidoc/liquibase-mongodb.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -181,15 +181,15 @@ To change the `wait-for` image which by default is `groundnuty/k8s-wait-for:no-r
181181

182182
[source,properties]
183183
----
184-
quarkus.kubernetes.init-task-defaults.wait-for-image=my/wait-for-image:1.0
184+
quarkus.kubernetes.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
185185
----
186186

187187
or on Openshift:
188188

189189

190190
[source,properties]
191191
----
192-
quarkus.openshift.init-task-defaults.wait-for-image=my/wait-for-image:1.0
192+
quarkus.openshift.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
193193
----
194194

195195
**Note**: In this context globally means `for all extensions that support init task externalization`.

docs/src/main/asciidoc/liquibase.adoc

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -259,19 +259,17 @@ To change the `wait-for` image which by default is `groundnuty/k8s-wait-for:no-r
259259

260260
[source,properties]
261261
----
262-
quarkus.kubernetes.init-task-defaults.wait-for-image=my/wait-for-image:1.0
262+
quarkus.kubernetes.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
263263
----
264264

265265
or on Openshift:
266266

267267

268268
[source,properties]
269269
----
270-
quarkus.openshift.init-task-defaults.wait-for-image=my/wait-for-image:1.0
270+
quarkus.openshift.init-task-defaults.wait-for-container.image=my/wait-for-image:1.0
271271
----
272272

273-
274-
275273
**Note**: In this context globally means `for all extensions that support init task externalization`.
276274

277275
== Configuration Reference

extensions/kubernetes/kind/deployment/src/main/java/io/quarkus/kind/deployment/KindProcessor.java

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
import io.quarkus.deployment.annotations.BuildProducer;
1919
import io.quarkus.deployment.annotations.BuildStep;
2020
import io.quarkus.deployment.builditem.ApplicationInfoBuildItem;
21+
import io.quarkus.deployment.builditem.InitTaskBuildItem;
2122
import io.quarkus.deployment.metrics.MetricsCapabilityBuildItem;
2223
import io.quarkus.deployment.pkg.PackageConfig;
2324
import io.quarkus.deployment.pkg.builditem.ArtifactResultBuildItem;
@@ -26,6 +27,7 @@
2627
import io.quarkus.kubernetes.client.spi.KubernetesClientCapabilityBuildItem;
2728
import io.quarkus.kubernetes.deployment.AddPortToKubernetesConfig;
2829
import io.quarkus.kubernetes.deployment.DevClusterHelper;
30+
import io.quarkus.kubernetes.deployment.InitTaskProcessor;
2931
import io.quarkus.kubernetes.deployment.KubernetesCommonHelper;
3032
import io.quarkus.kubernetes.deployment.KubernetesConfig;
3133
import io.quarkus.kubernetes.deployment.ResourceNameUtil;
@@ -135,4 +137,25 @@ public void postBuild(ContainerImageInfoBuildItem image, List<ContainerImageBuil
135137
//So, we now always perform this step
136138
ExecUtil.exec("kind", "load", "docker-image", image.getImage());
137139
}
138-
}
140+
141+
@BuildStep
142+
void externalizeInitTasks(
143+
ApplicationInfoBuildItem applicationInfo,
144+
KubernetesConfig config,
145+
ContainerImageInfoBuildItem image,
146+
List<InitTaskBuildItem> initTasks,
147+
BuildProducer<KubernetesJobBuildItem> jobs,
148+
BuildProducer<KubernetesInitContainerBuildItem> initContainers,
149+
BuildProducer<KubernetesEnvBuildItem> env,
150+
BuildProducer<KubernetesRoleBuildItem> roles,
151+
BuildProducer<KubernetesRoleBindingBuildItem> roleBindings,
152+
BuildProducer<KubernetesServiceAccountBuildItem> serviceAccount,
153+
154+
BuildProducer<DecoratorBuildItem> decorators) {
155+
final String name = ResourceNameUtil.getResourceName(config, applicationInfo);
156+
if (config.isExternalizeInit()) {
157+
InitTaskProcessor.process(KIND, name, image, initTasks, config.getInitTaskDefaults(), config.getInitTasks(),
158+
jobs, initContainers, env, roles, roleBindings, serviceAccount, decorators);
159+
}
160+
}
161+
}

extensions/kubernetes/minikube/deployment/src/main/java/io/quarkus/minikube/deployment/MinikubeProcessor.java

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,14 @@
1717
import io.quarkus.deployment.annotations.BuildProducer;
1818
import io.quarkus.deployment.annotations.BuildStep;
1919
import io.quarkus.deployment.builditem.ApplicationInfoBuildItem;
20+
import io.quarkus.deployment.builditem.InitTaskBuildItem;
2021
import io.quarkus.deployment.metrics.MetricsCapabilityBuildItem;
2122
import io.quarkus.deployment.pkg.PackageConfig;
2223
import io.quarkus.deployment.pkg.builditem.OutputTargetBuildItem;
2324
import io.quarkus.kubernetes.client.spi.KubernetesClientCapabilityBuildItem;
2425
import io.quarkus.kubernetes.deployment.AddPortToKubernetesConfig;
2526
import io.quarkus.kubernetes.deployment.DevClusterHelper;
27+
import io.quarkus.kubernetes.deployment.InitTaskProcessor;
2628
import io.quarkus.kubernetes.deployment.KubernetesCommonHelper;
2729
import io.quarkus.kubernetes.deployment.KubernetesConfig;
2830
import io.quarkus.kubernetes.deployment.ResourceNameUtil;
@@ -122,4 +124,25 @@ public List<DecoratorBuildItem> createDecorators(ApplicationInfoBuildItem applic
122124
livenessPath, readinessPath, startupPath,
123125
roles, clusterRoles, serviceAccounts, roleBindings, customProjectRoot);
124126
}
125-
}
127+
128+
@BuildStep
129+
void externalizeInitTasks(
130+
ApplicationInfoBuildItem applicationInfo,
131+
KubernetesConfig config,
132+
ContainerImageInfoBuildItem image,
133+
List<InitTaskBuildItem> initTasks,
134+
BuildProducer<KubernetesJobBuildItem> jobs,
135+
BuildProducer<KubernetesInitContainerBuildItem> initContainers,
136+
BuildProducer<KubernetesEnvBuildItem> env,
137+
BuildProducer<KubernetesRoleBuildItem> roles,
138+
BuildProducer<KubernetesRoleBindingBuildItem> roleBindings,
139+
BuildProducer<KubernetesServiceAccountBuildItem> serviceAccount,
140+
141+
BuildProducer<DecoratorBuildItem> decorators) {
142+
final String name = ResourceNameUtil.getResourceName(config, applicationInfo);
143+
if (config.isExternalizeInit()) {
144+
InitTaskProcessor.process(MINIKUBE, name, image, initTasks, config.getInitTaskDefaults(), config.getInitTasks(),
145+
jobs, initContainers, env, roles, roleBindings, serviceAccount, decorators);
146+
}
147+
}
148+
}

extensions/kubernetes/vanilla/deployment/src/main/java/io/quarkus/kubernetes/deployment/InitTaskConfig.java

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,15 +15,15 @@ public class InitTaskConfig {
1515

1616
/**
1717
* The init task image to use by the init-container.
18+
* Deprecated, use waitForContainer.image instead.
1819
*/
1920
@Deprecated
2021
@ConfigItem
2122
public Optional<String> image;
2223

2324
/**
24-
* The init task image to use by the init-container.
25+
* The configuration of the `wait for` container.
2526
*/
26-
@ConfigItem(defaultValue = "groundnuty/k8s-wait-for:no-root-v1.7")
27-
public String waitForImage;
28-
27+
@ConfigItem
28+
public InitTaskContainerConfig waitForContainer;
2929
}
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
package io.quarkus.kubernetes.deployment;
2+
3+
import io.quarkus.runtime.annotations.ConfigGroup;
4+
import io.quarkus.runtime.annotations.ConfigItem;
5+
6+
@ConfigGroup
7+
public class InitTaskContainerConfig {
8+
9+
/**
10+
* The init task image to use by the init-container.
11+
*/
12+
@ConfigItem(defaultValue = "groundnuty/k8s-wait-for:no-root-v1.7")
13+
public String image;
14+
15+
}

extensions/kubernetes/vanilla/deployment/src/main/java/io/quarkus/kubernetes/deployment/InitTaskProcessor.java

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import java.util.Collections;
44
import java.util.List;
55
import java.util.Map;
6+
import java.util.regex.Pattern;
67

78
import io.dekorate.kubernetes.config.EnvBuilder;
89
import io.dekorate.kubernetes.decorator.AddEnvVarDecorator;
@@ -21,9 +22,9 @@
2122

2223
public class InitTaskProcessor {
2324

24-
private static final String INIT_CONTAINER_WAITER_NAME = "init";
25+
private static final String INIT_CONTAINER_WAITER_NAME = "wait-for-";
2526

26-
static void process(
27+
public static void process(
2728
String target, // kubernetes, openshift, etc.
2829
String name,
2930
ContainerImageInfoBuildItem image,
@@ -40,7 +41,12 @@ static void process(
4041

4142
boolean generateRoleForJobs = false;
4243
for (InitTaskBuildItem task : initTasks) {
43-
InitTaskConfig config = initTasksConfig.getOrDefault(task.getName(), initTaskDefaults);
44+
String taskName = task.getName()
45+
//Strip appplication.name prefix and init suffix (for compatibility with previous versions)
46+
.replaceAll("^" + Pattern.quote(name + "-"), "")
47+
.replaceAll(Pattern.quote("-init") + "$", "");
48+
String jobName = name + "-" + taskName + "-init";
49+
InitTaskConfig config = initTasksConfig.getOrDefault(taskName, initTaskDefaults);
4450
if (config == null || config.enabled) {
4551
generateRoleForJobs = true;
4652
jobs.produce(KubernetesJobBuildItem.create(image.getImage())
@@ -60,10 +66,11 @@ static void process(
6066
.build())));
6167
});
6268

63-
String waitForImage = config.image.orElse(config.waitForImage);
64-
initContainers.produce(KubernetesInitContainerBuildItem.create(INIT_CONTAINER_WAITER_NAME, waitForImage)
65-
.withTarget(target)
66-
.withArguments(List.of("job", task.getName())));
69+
String waitForImage = config.image.orElse(config.waitForContainer.image);
70+
initContainers
71+
.produce(KubernetesInitContainerBuildItem.create(INIT_CONTAINER_WAITER_NAME + taskName, waitForImage)
72+
.withTarget(target)
73+
.withArguments(List.of("job", jobName)));
6774
}
6875
}
6976

0 commit comments

Comments
 (0)