Skip to content

Commit 27f69cf

Browse files
AkrogAkrog
committed
Docs: External dependencies
This patch introduces a couple of guides on how to do dependencies in the cinder-operator to explain simple and circular references for both running the operator locally and building the operator.
1 parent 8a69aae commit 27f69cf

File tree

3 files changed

+385
-0
lines changed

3 files changed

+385
-0
lines changed

‎CONTRIBUTING.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,13 @@ section](README.md#getting-started):
7070

7171
- [Run operator locally](docs/dev/local.md)
7272
- [Run operator in OpenShift using a custom image](docs/dev/custom-image.md)
73+
- [Running locally with external dependencies](docs/dev/local-dependencies.md)
74+
- [Running in OpenShift with external
75+
dependencies](docs/dev/custom-image-dependencies.md)
76+
77+
There is a script called `hack/checkout_pr.sh` that is helpful when we want to
78+
test an existing PR that has dependencies. Check the [Testing PR section in the
79+
hack documentation](hack/README.md#testing-prs) for additional information.
7380

7481
### Pull Requests
7582

‎docs/dev/custom-image-dependencies.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Running the operator in OpenShift with external dependencies
2+
3+
**NOTE**: This article [makes some assumptions](assumptions.md), make sure they
4+
are correct or adapt the steps accordingly.
5+
6+
If you've already [run the operator in OpenShift](custom-image.md) before,
7+
you'll be familiar with the process of running custom cinder-operator code
8+
by building and deploying a custom image.
9+
10+
Let's see what we have to do to run our operator in OpenShift when we need to
11+
compile and build containers with external dependencies.
12+
13+
This short guide builds on the knowledge provided in [guide to running the
14+
operator in OpenShift](custom-image.md) to explain external dependencies usage.
15+
16+
The steps will be the same, the only difference is that some additional steps
17+
are necessary between the [Preparation](custom-image.md#preparation) and the
18+
[Image build](custom-image.md#image-build) steps to indicate the new
19+
references.
20+
21+
To avoid repeating things again, this text assumes familiarity with the
22+
concepts presented in the [running locally with external
23+
dependencies](local-dependencies.md) article.
24+
25+
All the cases described in running the operator locally with external
26+
dependencies where the dependency was expressed as github repositories will
27+
work fine when building the containers, so we don't need to do anything
28+
special.
29+
30+
The difference comes when using local paths to reference the external
31+
dependencies, because those are outside of the root directory of the
32+
`Dockerfile` and therefore won't be present in the building container.
33+
34+
The solution is to use bind mounts.
35+
36+
### Unmerged local simple reference
37+
38+
Assuming the same example as running things locally we would do:
39+
40+
```
41+
cd ~/cinder-operator
42+
mkdir -p tmp/lib-common
43+
sudo mount --bind `realpath ../lib-common` `realpath tmp/lib-common`
44+
45+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/common=tmp/lib-common/modules/common
46+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/database=tmp/lib-common/modules/database
47+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/storage=tmp/lib-common/modules/storage
48+
```
49+
50+
Alternatively we can replace the last 3 commands with a single command:
51+
52+
```
53+
hack/setdeps.py lib-common=tmp/lib-common
54+
```
55+
56+
Then we can just continue with [building the
57+
image](custom-image.md#image-build) like we would normally do with the exception
58+
of having to add `GOWORK=` as well:
59+
60+
```
61+
GOWORK= make docker-build
62+
```
63+
64+
### Circular references
65+
66+
In the circular references case when using local paths it's a bit more
67+
complicated than the local run case, because the bind mount created from the
68+
cinder-operator to the openstack-operator is not preserved when we bind mount
69+
the cinder-operator from the openstack-operator, so we need to link it again.
70+
71+
We'll assume we have the dependency between cinder-operator and
72+
openstack-operator and that our local repositories have the code we want to
73+
use.
74+
75+
```
76+
cd ~/cinder-operator
77+
mkdir -p tmp/openstack-operator
78+
sudo mount --bind `realpath ../openstack-operator` `realpath tmp/openstack-operator`
79+
80+
go work edit -replace github.com/openstack-k8s-operators/openstack-operator/apis=tmp/openstack-operator
81+
82+
83+
cd ~/openstack-operator
84+
mkdir -p tmp/cinder-operator
85+
sudo mount --bind `realpath ../cinder-operator` `realpath tmp/cinder-operator`
86+
sudo mount --bind `realpath ./` `realpath tmp/cinder-operator/tmp/openstack-operator`
87+
88+
go work edit -replace github.com/openstack-k8s-operators/cinder-operator/api=tmp/cinder-operator/api
89+
```
90+
91+
Now we can continue with the [Image build steps](custom-image.md#image-build),
92+
again adding `GOWORK=` to the `docker-build`.

‎docs/dev/local-dependencies.md‎

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,286 @@
1+
# Running the operator locally with external dependencies
2+
3+
**NOTE**: This article [makes some assumptions](assumptions.md), make sure they
4+
are correct or adapt the steps accordingly.
5+
6+
If you've already [run the operator locally](local.md) before, you'll be
7+
familiar with the process of removing the cinder podified services and the
8+
cinder-operator pod from the OpenShift cluster, building the operator and
9+
running it locally.
10+
11+
It's very likely that during the development of the cinder-operator you'll find
12+
yourself needing to use a newer version of a dependency or even needing to use
13+
an unmerged code of a dependency.
14+
15+
This short guide builds on the knowledge provided in [guide to locally run the
16+
operator](local.md) to explain those external dependencies usages.
17+
18+
The steps will be the same, the only difference is that some additional steps
19+
are necessary between the [Preparation](local.md#preparation) and the [Build
20+
and Run](local.md#build-and-run) steps to indicate the new references.
21+
22+
There are slight differences between the possible cases, lib-common dependency
23+
update, circular reference with the openstack-operator, the external dependency
24+
code being merged or not, having the external dependency's code locally or not,
25+
etc., so to facilitate its understanding we'll provide separate explanations
26+
for most of them.
27+
28+
### Merged simple reference
29+
30+
If we want to compile the cinder-operator with newer lib-common code that has
31+
already merged in its repository but don't want to wait for dependabot to
32+
propose an update to the project's dependencies and for it to get merged we
33+
can just run the following commands from the `cinder-operator` repository
34+
before [Building and Running the operator](local.md#build-and-run):
35+
36+
```
37+
go get github.com/openstack-k8s-operators/lib-common/modules/common
38+
go get github.com/openstack-k8s-operators/lib-common/modules/database
39+
go get github.com/openstack-k8s-operators/lib-common/modules/storage
40+
```
41+
42+
**NOTE**: Using the ``go get`` command ensures that other `go.mod` adjustments
43+
are made as needed to satisfy constraints imposed by other modules.
44+
45+
After running these commands we can see that the `go.mod` and `go.sum` have
46+
been updated:
47+
48+
```
49+
$ git diff go.*
50+
51+
diff --git a/go.mod b/go.mod
52+
index 599f056..8e7d423 100644
53+
--- a/go.mod
54+
+++ b/go.mod
55+
@@ -9,9 +9,9 @@ require (
56+
github.com/openshift/api v3.9.0+incompatible
57+
github.com/openstack-k8s-operators/cinder-operator/api v0.0.0-20221010180347-a9a8efadf3c3
58+
github.com/openstack-k8s-operators/keystone-operator/api v0.0.0-20220927090553-6b3218c776f7
59+
- github.com/openstack-k8s-operators/lib-common/modules/common v0.0.0-20221103175706-2c39582ce513
60+
- github.com/openstack-k8s-operators/lib-common/modules/database v0.0.0-20220923094431-9fca0c85a9dc
61+
- github.com/openstack-k8s-operators/lib-common/modules/storage v0.0.0-20220923094431-9fca0c85a9dc
62+
+ github.com/openstack-k8s-operators/lib-common/modules/common v0.0.0-20221117092428-c1190ea3bf3d
63+
+ github.com/openstack-k8s-operators/lib-common/modules/database v0.0.0-20221117092428-c1190ea3bf3d
64+
+ github.com/openstack-k8s-operators/lib-common/modules/storage v0.0.0-20221117092428-c1190ea3bf3d
65+
github.com/openstack-k8s-operators/mariadb-operator/api v0.0.0-20221014164348-0a612ae8b391
66+
github.com/openstack-k8s-operators/openstack-operator/apis v0.0.0-20221107090218-8d63dba1ec13
67+
k8s.io/api v0.25.4
68+
diff --git a/go.sum b/go.sum
69+
index c19f296..2130ed6 100644
70+
--- a/go.sum
71+
+++ b/go.sum
72+
@@ -323,12 +323,18 @@ github.com/openstack-k8s-operators/keystone-operator/api v0.0.0-20220927090553-6
73+
github.com/openstack-k8s-operators/keystone-operator/api v0.0.0-20220927090553-6b3218c776f7/go.mod h1:q/owiyXlI2W4uQR4TeHPeeN75AGDfyZgQdNHeKUYN68=
74+
github.com/openstack-k8s-operators/lib-common/modules/common v0.0.0-20221103175706-2c39582ce513 h1:PSXOLFTskoG9R/YR4Pg5AOJYS3CEnFbZ2yVdrk9xOE4=
75+
github.com/openstack-k8s-operators/lib-common/modules/common v0.0.0-20221103175706-2c39582ce513/go.mod h1:KWqK7l2ej+rIYngoNUrxE2YjKGlRAAgJXXM0uU2R6XY=
76+
+github.com/openstack-k8s-operators/lib-common/modules/common v0.0.0-20221117092428-c1190ea3bf3d h1:/1FTHxBQJo4xM0GmJCX5wPCYmyLWTw1uHQKCydGH3mY=
77+
+github.com/openstack-k8s-operators/lib-common/modules/common v0.0.0-20221117092428-c1190ea3bf3d/go.mod h1:KWqK7l2ej+rIYngoNUrxE2YjKGlRAAgJXXM0uU2R6XY=
78+
github.com/openstack-k8s-operators/lib-common/modules/database v0.0.0-20220923094431-9fca0c85a9dc h1:87lUVT3MLRI4Vg0nHpupwPKXtykGX3hZzPl5k6Kcyng=
79+
github.com/openstack-k8s-operators/lib-common/modules/database v0.0.0-20220923094431-9fca0c85a9dc/go.mod h1:umGUqQO4JtgefAaIwZjP+TxfxsLMEEeK/6VNzk8ooaI=
80+
+github.com/openstack-k8s-operators/lib-common/modules/database v0.0.0-20221117092428-c1190ea3bf3d h1:lO5WmV9RjVAIxbr1HPjvqVy6niatdEPG+FyEbL4FMpc=
81+
+github.com/openstack-k8s-operators/lib-common/modules/database v0.0.0-20221117092428-c1190ea3bf3d/go.mod h1:umGUqQO4JtgefAaIwZjP+TxfxsLMEEeK/6VNzk8ooaI=
82+
github.com/openstack-k8s-operators/lib-common/modules/openstack v0.0.0-20220915080953-f73a201a1da6 h1:MVNEHyqD0ZdO9jiyUSKw5M2T9Lc4l4Wx1pdC2/BSJ5Y=
83+
github.com/openstack-k8s-operators/lib-common/modules/openstack v0.0.0-20220915080953-f73a201a1da6/go.mod h1:YsqouRH8DoZAjFaxcIErspk59BcwXtVjPxK/yV17Wrc=
84+
github.com/openstack-k8s-operators/lib-common/modules/storage v0.0.0-20220923094431-9fca0c85a9dc h1:Dud2dr25VhaZF9Av28nqmCeBfNkGWDckZ5TaajEcGFc=
85+
github.com/openstack-k8s-operators/lib-common/modules/storage v0.0.0-20220923094431-9fca0c85a9dc/go.mod h1:fhM62I45VF/5WVpOP1h9OpTfFn+lF2XGrT5jUBKEHVc=
86+
+github.com/openstack-k8s-operators/lib-common/modules/storage v0.0.0-20221117092428-c1190ea3bf3d h1:b2dqfShyX4NO/NMe3rTK1xWRF91ITobjQEfO6ftO6yM=
87+
+github.com/openstack-k8s-operators/lib-common/modules/storage v0.0.0-20221117092428-c1190ea3bf3d/go.mod h1:fhM62I45VF/5WVpOP1h9OpTfFn+lF2XGrT5jUBKEHVc=
88+
github.com/openstack-k8s-operators/mariadb-operator/api v0.0.0-20221014164348-0a612ae8b391 h1:Bd1e4CG/0gQbRoSH1EJLS1tin9XUjPR2s1e+dpBHiUs=
89+
github.com/openstack-k8s-operators/mariadb-operator/api v0.0.0-20221014164348-0a612ae8b391/go.mod h1:HiEKXmDSJ6Gl+pN7kK5CX1sgOjrxybux4Ob5pdUim1M=
90+
github.com/openstack-k8s-operators/openstack-operator/apis v0.0.0-20221107090218-8d63dba1ec13 h1:GkYSRpfdDav5HipJ2oIYqpY8crLaDVRBmhqlUztDTV4=
91+
```
92+
93+
Now we can proceed as usual with the [Building and
94+
Running](local.md#build-and-run) steps.
95+
96+
Once we have confirmed that our code changes work as expected with the new
97+
dependency version we can submit our PR as we normally do but including the
98+
`go.mod` and `go.sum` changes.
99+
100+
If we want to use a specific tag instead of `HEAD` we can provide it in our
101+
calls:
102+
103+
```
104+
go get github.com/openstack-k8s-operators/lib-common/modules/[email protected]
105+
go get github.com/openstack-k8s-operators/lib-common/modules/[email protected]
106+
go get github.com/openstack-k8s-operators/lib-common/modules/[email protected]
107+
```
108+
109+
### Unmerged local simple reference
110+
111+
There will be times when we'll be working on lib-common code in our local
112+
repository while using it in the cinder-operator and we want to test this.
113+
114+
In this case we cannot use the `go get` because our lib-common module is not
115+
within the cinder-operator command so we'll use the `go work edit` command
116+
instead:
117+
118+
```
119+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/common=../lib-common/modules/common
120+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/database=../lib-common/modules/database
121+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/storage=../lib-common/modules/storage
122+
```
123+
124+
Alternatively we can use the `hack/setdeps.py` script to do the same thing:
125+
126+
```
127+
hack/setdeps.py lib-common=../lib-common
128+
```
129+
130+
Now we can proceed as usual with the [Building and
131+
Running](local.md#build-and-run) steps. If we use `make build` then we'll need
132+
to let it know that it should use our workspace changes:
133+
134+
```
135+
GOWORK= make build
136+
```
137+
138+
Once we have confirmed that our code changes work as expected with the new
139+
dependency version we will need to submit the lib-common PR first and then the
140+
one in cinder-operator with the new dependency, as explained in the previous
141+
section.
142+
143+
### Simple reference in PR
144+
145+
Somebody may have submitted a PR to lib-common and we want to start writing
146+
code in the cinder-operator before that PR gets merged.
147+
148+
Here we can either download the PR to our own local repository or just
149+
reference the one from GitHub.
150+
151+
As an example, let's look at the `extraVolumes` effort and the [lib-common
152+
PR#88](https://github.com/openstack-k8s-operators/lib-common/pull/88).
153+
154+
To reference it directly we'll go to GitHub's website and look for the source
155+
of the PR, which in this case it's [fmount's extra_volumes branch](
156+
https://github.com/fmount/lib-common/tree/extra_volumes).
157+
158+
So now we replace our lib-common with that one:
159+
160+
```
161+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/common=github.com/fmount/lib-common/modules/common@extra_volumes
162+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/database=github.com/fmount/lib-common/modules/database@extra_volumes
163+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/storage=github.com/fmount/lib-common/modules/storage@extra_volumes
164+
165+
go mod tidy
166+
```
167+
168+
Alternatively we can use the `hack/setdeps.py` to do all that for us:
169+
170+
```
171+
hack/setdeps.py lib-common=88
172+
```
173+
174+
Or just the last part:
175+
176+
```
177+
go work edit -replace lib-common=fmount/lib-common
178+
```
179+
180+
If we have a local repository and we may want to make changes to the lib-common
181+
code ourselves (in case we find some issue) we can pull the PR and then
182+
reference it:
183+
184+
```
185+
cd ../lib-common
186+
187+
git fetch upstream pull/88/head:extra_volumes
188+
git checkout extra_volumes
189+
190+
cd ../cinder-operator
191+
192+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/common=../lib-common/modules/common
193+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/database=../lib-common/modules/database
194+
go work edit -replace github.com/openstack-k8s-operators/lib-common/modules/storage=../lib-common/modules/storage
195+
```
196+
197+
Alternatively we can replace the last 3 commands with a single command:
198+
199+
```
200+
hack/setdeps.py lib-common=../lib-common
201+
```
202+
203+
Now we can proceed as usual with the [Building and
204+
Running](local.md#build-and-run) steps. If we use `make build` then we'll need
205+
to let it know that it should use our workspace changes:
206+
207+
```
208+
GOWORK= make build
209+
```
210+
211+
### Circular references
212+
213+
There are times where we have circular dependencies where cinder-operator needs
214+
a new structure defined in the openstack-operator but at the same time the
215+
openstack-operator uses the cinder-operator structures to define the
216+
`OpenStackControlPlane`.
217+
218+
An example of this happening is the work on the `TrasportURL` that was
219+
introduced in [openstack-operator
220+
PR#37](https://github.com/openstack-k8s-operators/openstack-operator/pull/27)
221+
and used in the [cinder-operator
222+
PR#62](https://github.com/openstack-k8s-operators/cinder-operator/pull/62).
223+
224+
The solution of how to replace the dependencies is pretty much the same as the
225+
one we've seen in the previous section, the only difference is that we need to
226+
do it in both repositories.
227+
228+
We see in github that the cinder-operator PR source branch is
229+
`use_transport_url_crd` from `abays` and the openstack-operator PR source
230+
branch is `rabbitmq_transporturl` from `dprince`, so we replace their
231+
dependencies:
232+
233+
```
234+
cd ~/cinder-operator
235+
go work edit -replace github.com/openstack-k8s-operators/openstack-operator/apis=github.com/dprince/openstack-operator@rabbitmq_transporturl
236+
237+
cd ~/openstack-operator
238+
go work edit -replace github.com/openstack-k8s-operators/cinder-operator/api=github.com/abays/cinder-operator@use_transport_url_crd
239+
```
240+
241+
**NOTE**: If we are using local directory references we should use absolute
242+
paths instead of relative ones like we did before.
243+
244+
In this case since we have also modified the openstack-operator we'll have to
245+
stop it in our OpenShift cluster and run it locally before we can proceed with
246+
[building and running the cinder-operator](local.md#build-and-run). Remember
247+
to add the `GOWORK=` to the `make build` call.
248+
249+
```sh
250+
CSV=`oc get -l operators.coreos.com/openstack-operator.openstack= -o custom-columns=CSV:.metadata.name --no-headers csv`
251+
252+
oc edit csv $CSV
253+
```
254+
255+
This will drop us in our editor with the contents of CSV YAML manifest where
256+
we'll search for the first instance of `name:
257+
openstack-operator-controller-manager`, and we should see something like:
258+
259+
```
260+
- label:
261+
control-plane: controller-manager
262+
name: openstack-operator-controller-manager
263+
spec:
264+
replicas: 1
265+
selector:
266+
matchLabels:
267+
control-plane: controller-manager
268+
```
269+
270+
Where we see `replicas: 1` change it to `replicas: 0`, save and exit. This
271+
triggers the termination of the openstack-operator pod.
272+
273+
Now we just build and run our local openstack-operator:
274+
275+
```
276+
cd ~/openstack-operator
277+
GOWORK= make run
278+
```
279+
280+
Now we can continue, in another terminal, [running the cinder-operator
281+
locally](local.md#build-and-run) but this time we cannot use `make run`,
282+
because it will fail due to ports :8080 and :8081 already being in use:
283+
284+
```
285+
bin/manager -metrics-bind-address=:8082 -health-probe-bind-address=:8083
286+
```

0 commit comments

Comments
 (0)