You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
:description: Learn how to specify product images for Stackable Data Platform deployments, including using default, custom, or mirrored registries.
5
5
6
-
To run any product on the Stackable Data Platform, you need to specify which version you want to run in the resource definition (i.e. a SparkApplication or DruidCluster).
7
-
The simplest way to do this is this:
6
+
To run any product on the Stackable Data Platform, you need to specify which version you want to run in the resource definition (e.g., a `SparkApplication` or `DruidCluster`).
7
+
The simplest way to do this is as follows:
8
8
9
9
[source,yaml]
10
10
----
11
11
spec:
12
12
image:
13
13
productVersion: 1.2.3 <.>
14
-
# stackableVersion: 24.3.0 optional <.>
14
+
# stackableVersion: 25.7.0 # optional <.>
15
15
----
16
-
<.> The version of your product - consult the product operator documentation to find out about supported product versions.
17
-
<.> The version of the Stackable Data Platform. If you omit it, the operator will use it's own version together with the product version to select the product image. It is recommended to omit this field, so that you are always using the correct, compatible version of the product image.
18
-
19
-
This page explains the different ways of specifying product images and the components that are involved,
20
-
starting from the simple way of using the default images to explaining how to use custom or mirrored registries as well as custom images.
16
+
<.> The version of your product. Consult the product operator documentation to find out about supported product versions.
17
+
<.> The version of the Stackable Data Platform. If you omit it, the operator will use its own version together with the product version to select the product image. It is recommended to omit this field to ensure you are always using the correct, compatible version of the product image.
21
18
22
19
== Product and Stackable version explained
23
20
24
-
All products of the Stackable Data Platform run on Kubernetes and are managed by Stackable Operators. One operator is responsible for exactly one product, for example Apache Spark, but can manage multiple instances (with different versions) of the product at the same time.
25
-
The operators transform a Stacklet definition (for example a SparkApplication object) into Kubernetes native objects. Some of these objects are Pods and they require _container images_ to run.
26
-
These images contain different tools for initialization jobs and/or the actual product itself and the images are version and architecture specific.
27
-
28
-
Stackable uses two separate versions to describe the images that are provided as part of the platform:
21
+
All products of the Stackable Data Platform run on Kubernetes and are managed by Stackable Operators.
22
+
Each operator is responsible for one product, such as Apache Spark, but can manage multiple instances (with different versions) of the product simultaneously.
23
+
The operators transform a Stacklet definition (e.g., a `SparkApplication` object) into Kubernetes-native objects.
24
+
Some of these objects are Pods, which require _container images_ to run.
25
+
These images contain various tools for initialization jobs and/or the actual product itself and are version and architecture-specific.
29
26
27
+
Stackable uses two separate versions to describe the images provided as part of the platform:
30
28
31
29
**Product version** +
32
-
This is the version of the product which this image provides, so this could for example be Kafka 3.3.1.
30
+
This is the version of the product that the image provides, such as Kafka 3.9.0.
33
31
34
32
TIP: You can find all products and their supported versions in the xref:operators:supported_versions.adoc[supported versions overview].
35
-
You can also find the supported versions per product on each operator page, for example for xref:kafka:index.adoc#_supported_versions[Apache Kafka].
36
-
New versions, deprecations and removals are announced in the xref:ROOT:release-notes.adoc[release notes].
33
+
You can also find the supported versions per product on each operator page, for example, for xref:kafka:index.adoc#_supported_versions[Apache Kafka].
34
+
New versions, deprecations, and removals are announced in the xref:ROOT:release-notes.adoc[release notes].
37
35
38
36
**Stackable version** +
39
-
This version is used to track changes to the structure of the image containing the product (in the version specified by _product version_).
40
-
Stackable operators expect to find a very specific structure in the images they use to roll out the products.
41
-
This can be things like startup scripts being present, parameters these startup scripts expect, presence or location of extra libraries and similar things.
42
-
In order for our operators to work as intended every operator has to be used with images from the same release line as this operator.
43
-
44
-
What this means is, that for example the Stackable Operator for Apache HBase will by default try to use images with the same Stackable version, the following table shows a few examples to make this clearer:
37
+
This version tracks changes to the structure of the image containing the product (in the version specified by the _product version_).
38
+
Stackable operators expect to find a specific file and folder structure in the images they use to roll out the products.
39
+
This can include things like startup scripts, parameters these scripts expect, the presence or location of extra libraries, and similar aspects.
40
+
For our operators to work as intended, every operator must be used with images from the same release line as the operator.
45
41
42
+
This means, for example, that the Stackable Operator for Apache HBase will, by default, try to use images with the same Stackable version.
43
+
The following table provides a few examples to clarify this:
46
44
47
45
|===
48
-
|Operator version |HBase version |Image
49
-
50
-
|23.4.0
51
-
|3.3.0
52
-
|hbase:3.3.0-stackable23.4.0
53
-
54
-
|23.4.0
55
-
|3.3.1
56
-
|hbase:3.3.1-stackable23.4.0
57
-
58
-
|23.7.0
59
-
|3.3.0
60
-
|hbase:3.3.0-stackable23.7.0
46
+
| Operator version | HBase version | Image
61
47
62
-
|23.7.0
63
-
|3.3.1
64
-
|hbase:3.3.1-stackable23.7.0
48
+
| 23.4.0 | 3.3.0 | hbase:3.3.0-stackable23.4.0
49
+
| 23.4.0 | 3.3.1 | hbase:3.3.1-stackable23.4.0
50
+
| 23.7.0 | 3.3.0 | hbase:3.3.0-stackable23.7.0
51
+
| 23.7.0 | 3.3.1 | hbase:3.3.1-stackable23.7.0
65
52
|===
66
53
67
-
68
-
However, since the last digit of the Stackable version is considered to be a patch level indicator, operators will be compatible with all images from the same release line.
69
-
So an operator of version _23.4.x_ will be compatible with all images of version _23.4.y_.
70
-
This is intended to allow shorter update cycles for users, when new image versions are released that may contain security fixes.
54
+
However, since the last digit of the Stackable version is considered a patch level indicator, operators will be compatible with all images from the same release line.
55
+
So, an operator of version _25.7.x_ will be compatible with all images of version _25.7.y_ release line.
56
+
This allows for shorter update cycles for users when new image versions are released that may contain security fixes.
71
57
72
58
The following paragraphs explain the available settings and how they work.
73
59
74
-
At the bottom of this page in the <<_common_scenarios, common scenarios>> section some common update scenarios are explained as examples.
60
+
At the bottom of this page, in the <<_common_scenarios, common scenarios>> section, some common update scenarios are explained as examples.
75
61
76
62
== Stackable provided images
77
63
78
-
If your Kubernetes cluster has internet access, the easiest way is to use the publicly available images from the https://oci.stackable.tech/[Image registry hosted by Stackable] (as has been noted at the beginning of the page):
64
+
If your Kubernetes cluster has internet access, the easiest way is to use the publicly available images from the https://oci.stackable.tech/[Image registry hosted by Stackable] (as noted at the beginning of the page):
65
+
66
+
TIP: All our images are also mirrored to our https://quay.io/organization/stackable[quay.io organization].
79
67
80
68
[source,yaml]
81
69
----
82
70
spec:
83
71
image:
84
-
productVersion: 3.3.1 <.>
85
-
# stackableVersion: 23.7.0 # optional <.>
72
+
productVersion: 3.9.0 <.>
73
+
# stackableVersion: 25.7.0 # optional <.>
86
74
----
87
-
<.> The version of your product - consult the product operator documentation to find out about supported product versions.
88
-
<.> The version of the Stackable Data Platform - simply omit this to use the operator version.
75
+
<.> The version of your product. Consult the product operator documentation to find out about supported product versions.
76
+
<.> The version of the Stackable Data Platform. Simply omit this to use the operator version.
89
77
90
78
NOTE: If the Kubernetes cluster does not have internet access, a xref:_custom_docker_registry[] or xref:_custom_images[] can be used.
91
79
92
-
You only need to specify the product version but _can_ also specify an explicit Stackable version.
93
-
the product version can be found on the xref:operators:supported_versions.adoc[list of supported product versions] or on the product operator documentation page.
80
+
You only need to specify the product version, but you _can_ also specify an explicit Stackable version.
81
+
The product version can be found on the xref:operators:supported_versions.adoc[list of supported product versions] or on the product operator documentation page.
94
82
95
-
As images should be updated from time to time (e.g. new base image, security updates), a Stackable version can be provided.
96
-
An image with the Stackable version `23.7.0` is fixed and will never change.
97
-
Security updates within a release line will result in patch version bumps in the Stackable version to e.g. `23.7.1`.
83
+
As images should be updated from time to time (e.g., new base image, security updates), a Stackable version can be provided.
84
+
An image with the Stackable version `25.7.0` is fixed and will never change.
85
+
Security updates within a release line will result in patch version bumps in the Stackable version, for example, to `25.7.1`.
98
86
99
-
If you don't specify the Stackable version, the operator will use its own version, e.g. `23.7.0`.
87
+
If you don't specify the Stackable version, the operator will use its own version, e.g., `25.7.0`.
100
88
When using a nightly operator or a `pr` version, it will use the nightly `0.0.0-dev` image.
101
89
102
-
All the available images (with their product and stackable version) can be found in our https://oci.stackable.tech/api/v2.0/projects/sdp[OCI registry]. Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here]).
90
+
All the available images (with their product and Stackable versions) can be found in our https://oci.stackable.tech/api/v2.0/projects/sdp[OCI registry].
91
+
Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here].
103
92
104
93
== Custom docker registry
105
94
106
-
Custom docker registries can be used to fetch the image from a local image registry rather than from the internet.
107
-
The perquisite is that you mirror all the required images the same way (with the same name and tag) as the images provided by Stackable.
95
+
Custom Docker registries can be used to fetch the image from a local image registry rather than from the internet.
96
+
The prerequisite is that you mirror all the required images in the same way (with the same name and tag) as the images provided by Stackable.
108
97
109
-
Afterwards you can use the following snippet to configure your custom docker repo:
98
+
Afterward, you can use the following snippet to configure your custom Docker repo:
110
99
111
100
[source,yaml]
112
101
----
113
102
spec:
114
103
image:
115
104
productVersion: 3.3.1
116
-
stackableVersion: 23.7.0
117
-
repo: my.corp/myteam/stackable
105
+
stackableVersion: 25.7.0 # Optional
106
+
repo: my.corp/myteam/stackable <.>
118
107
----
108
+
<.> We recommend not including a slash at the end while we plan on https://github.com/stackabletech/operator-rs/issues/1020[improving the situation]
109
+
119
110
120
111
This will change the image from the default Stackable repository `oci.stackable.tech/sdp/kafka:3.3.1-stackable23.7.0` to `my.corp/myteam/stackable/kafka:3.3.1-stackable23.7.0`.
121
112
122
113
== [[customimages]] Custom images
123
114
124
115
Custom images can be used to fetch arbitrary images from local or public registries.
125
-
In comparison to the xref:_custom_docker_registry[], this allows to provide self-hosted or user-created images (e.g. userextended Stackable images).
126
-
If your image has other tags or names than the ones provided by Stackable you need to use this option.
116
+
Unlike the xref:_custom_docker_registry[], this allows you to provide self-hosted or user-created images (e.g., user-extended Stackable images).
117
+
If your image has different tags or names than those provided by Stackable, you need to use this option.
127
118
128
119
[source,yaml]
129
120
----
@@ -134,32 +125,33 @@ spec:
134
125
----
135
126
136
127
Even though the product version is not used anymore for image selection, you still need to provide it, as the operators configure their respective products based on the product version.
137
-
This affects e.g. configuration properties or available product features.
138
-
Only when the correct product version is given to the operator, the product will work correctly, so you need to provide the product version that you have used in your custom image.
128
+
This affects configuration properties or available product features.
129
+
Only when the correct product version is given to the operator will the product work correctly, so you need to provide the product version that you have used in your custom image.
139
130
140
131
Using custom images has a few limitations that users should be aware of:
141
132
142
-
* The images will *have* to have the same structures that Stackable operators expect.
133
+
* The images will _have_ to have the same structures that Stackable operators expect.
143
134
This should usually be ensured by specifying a Stackable image in the `FROM` clause of the Dockerfile (all the available images can be found in our https://oci.stackable.tech/api/v2.0/projects/sdp[OCI registry] - the schema is typically: `oci.stackable.tech/sdp/<product>:<product-version>-stackable<stackable-version>`. Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here]).
144
135
145
136
* Images will have to be upgraded for every new Stackable release to follow structural changes that Stackable may have made to their images.
146
-
When deriving images from official Stackable images this will mean updating the version of the image in the `FROM` clause to the correct Stackable release.
137
+
When deriving images from official Stackable images, this will mean updating the version of the image in the `FROM` clause to the correct Stackable release.
147
138
148
139
* It is not possible to update the Stackable Platform to a new version without changing the deployed cluster definitions when using custom images.
149
140
The recommended process here is:
150
141
151
142
** Set `reconciliationPaused` to `true` in your product cluster (see xref:operations/cluster_operations.adoc[])
152
-
** Update Stackable platform
143
+
** Update the Stackable platform
153
144
** Change custom images in cluster specifications
154
-
** Set `reconciliationPaused` to `false` again to start reconciliation again
145
+
** Set `reconciliationPaused` to `false` again to start reconciliation
155
146
156
-
## [[common_scenarios]] Common Scenarios
147
+
== [[common_scenarios]] Common Scenarios
157
148
158
-
### Planned platform updates
159
-
This is probably the most common scenario, users do not specify a Stackable version, and thus the operators always pick the image from their exact release.
160
-
Updates happen by updating Stackable Operators, which will in turn restart the products with the new images.
149
+
=== Planned platform updates
161
150
162
-
#### Config
151
+
This is probably the most common scenario, where users do not specify a Stackable version, and thus the operators always pick the image from their exact release.
152
+
Updates happen by updating Stackable Operators, which will, in turn, restart the products with the new images.
153
+
154
+
==== Config
163
155
164
156
[source,yaml]
165
157
----
@@ -168,13 +160,14 @@ spec:
168
160
productVersion: 3.3.1
169
161
----
170
162
171
-
### Custom images / pinned images
172
-
When a setup requires the utmost stability, and it is preferable for things to break, rather than run with a different image version that for example has not been certified.
173
-
Or when a user requires custom libraries / code in the images they run and build their own images derived from official Stackable images, this is the only possible way to do this.
163
+
=== Custom images / pinned images
164
+
165
+
When a setup requires the utmost stability, and it is preferable for things to break rather than run with a different image version that, for example, has not been certified, or when a user requires custom libraries or code in the images they run and builds their own images derived from official Stackable images, this is the only possible way to do this.
166
+
167
+
Please see the warnings in the <<customimages, custom images section>> above for how to upgrade in this scenario.
174
168
175
-
Please see the warnings in <<customimages, custom images section>> above for how to upgrade in this scenario.
0 commit comments