|
1 | 1 | **Note**: Kubernetes now manages dependencies using go modules.
|
2 |
| -See [current documentation for working with dependencies](./vendor.md) for master branch development. |
3 |
| -This document only applies to Kubernetes 1.14.x and earlier, |
4 |
| -and should be removed once Kubernetes 1.14.x is no longer supported. |
5 |
| - |
6 |
| -# Using godep to manage dependencies |
7 |
| - |
8 |
| -This document is intended to show a way for managing `vendor/` tree dependencies |
9 |
| -in Kubernetes. If you do not need to manage vendored dependencies, you probably |
10 |
| -do not need to read this. |
11 |
| - |
12 |
| -## Background |
13 |
| - |
14 |
| -As a tool, `godep` leaves much to be desired. It builds on `go get`, and adds |
15 |
| -the ability to pin dependencies to exact git version. The `go get` tool itself |
16 |
| -doesn't have any concept of versions, and tends to blow up if it finds a git |
17 |
| -repo synced to anything but `master`, but that is exactly the state that |
18 |
| -`godep` leaves repos. This is a recipe for frustration when people try to use |
19 |
| -the tools. |
20 |
| - |
21 |
| -This doc will focus on predictability and reproducibility. |
22 |
| - |
23 |
| -## Justifications for an update |
24 |
| - |
25 |
| -Before you update a dependency, take a moment to consider why it should be |
26 |
| -updated. Valid reasons include: |
27 |
| - 1. We need new functionality that is in a later version. |
28 |
| - 2. New or improved APIs in the dependency significantly improve Kubernetes code. |
29 |
| - 3. Bugs were fixed that impact Kubernetes. |
30 |
| - 4. Security issues were fixed even if they don't impact Kubernetes yet. |
31 |
| - 5. Performance, scale, or efficiency was meaningfully improved. |
32 |
| - 6. We need dependency A and there is a transitive dependency B. |
33 |
| - 7. Kubernetes has an older level of a dependency that is precluding being able |
34 |
| -to work with other projects in the ecosystem. |
35 |
| - |
36 |
| -## Theory of operation |
37 |
| - |
38 |
| -The `go` toolchain assumes a global workspace that hosts all of your Go code. |
39 |
| - |
40 |
| -The `godep` tool operates by first "restoring" dependencies into your `$GOPATH`. |
41 |
| -This reads the `Godeps.json` file, downloads all of the dependencies from the |
42 |
| -internet, and syncs them to the specified revisions. You can then make |
43 |
| -changes - sync to different revisions or edit Kubernetes code to use new |
44 |
| -dependencies (and satisfy them with `go get`). When ready, you tell `godep` to |
45 |
| -"save" everything, which it does by walking the Kubernetes code, finding all |
46 |
| -required dependencies, copying them from `$GOPATH` into the `vendor/` directory, |
47 |
| -and rewriting `Godeps.json`. |
48 |
| - |
49 |
| -This does not work well, when combined with a global Go workspace. Instead, we |
50 |
| -will set up a private workspace for this process. |
51 |
| - |
52 |
| -The Kubernetes build process uses this same technique, and offers a tool called |
53 |
| -`run-in-gopath.sh` which sets up and switches to a local, private workspace, |
54 |
| -including setting up `$GOPATH` and `$PATH`. If you wrap commands with this |
55 |
| -tool, they will use the private workspace, which will not conflict with other |
56 |
| -projects and is easily cleaned up and recreated. |
57 |
| - |
58 |
| -To see this in action, you can run an interactive shell in this environment: |
59 |
| - |
60 |
| -```sh |
61 |
| -# Run a shell, but don't run your own shell initializations. |
62 |
| -hack/run-in-gopath.sh bash --norc --noprofile |
63 |
| -``` |
64 |
| - |
65 |
| -## Restoring deps |
66 |
| - |
67 |
| -To extract and download dependencies into `$GOPATH` we provide a script: |
68 |
| -`hack/godep-restore.sh`. If you run this tool, it will restore into your own |
69 |
| -`$GOPATH`. If you wrap it in `run-in-gopath.sh` it will restore into your |
70 |
| -`_output/` directory. |
71 |
| - |
72 |
| -```sh |
73 |
| -hack/run-in-gopath.sh hack/godep-restore.sh |
74 |
| -``` |
75 |
| - |
76 |
| -This script will try to optimize what it needs to download, and if it seems the |
77 |
| -dependencies are all present already, it will return very quickly. |
78 |
| - |
79 |
| -If there's ever any doubt about the correctness of your dependencies, you can |
80 |
| -simply `make clean` or `rm -rf _output`, and run it again. |
81 |
| - |
82 |
| -Now you should have a clean copy of all of the Kubernetes dependencies. |
83 |
| - |
84 |
| -Downloading dependencies might take a while, so if you want to see progress |
85 |
| -information use the `-v` flag: |
86 |
| - |
87 |
| -```sh |
88 |
| -hack/run-in-gopath.sh hack/godep-restore.sh -v |
89 |
| -``` |
90 |
| - |
91 |
| -## Making changes |
92 |
| - |
93 |
| -The most common things people need to do with deps are add and update them. |
94 |
| -These are similar but different. |
95 |
| - |
96 |
| -### Adding a dep |
97 |
| - |
98 |
| -For the sake of examples, consider that we have discovered a wonderful Go |
99 |
| -library at `example.com/go/frob`. The first thing you need to do is get that |
100 |
| -code into your workspace: |
101 |
| - |
102 |
| -```sh |
103 |
| -hack/run-in-gopath.sh go get -d example.com/go/frob |
104 |
| -``` |
105 |
| - |
106 |
| -This will fetch, but not compile (omit the `-d` if you want to compile it now), |
107 |
| -the library into your private `$GOPATH`. It will pull whatever the default |
108 |
| -revision of that library is, typically the `master` branch for git repositories. |
109 |
| -If this is not the revision you need, you can change it, for example to |
110 |
| -`v1.0.0`: |
111 |
| - |
112 |
| -```sh |
113 |
| -hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v1.0.0' |
114 |
| -``` |
115 |
| - |
116 |
| -Now that the code is present, you can start to use it in Kubernetes code. |
117 |
| -Because it is in your private workspace's `$GOPATH`, it might not be part of |
118 |
| -your own `$GOPATH`, so tools like `goimports` might not find it. This is an |
119 |
| -unfortunate side-effect of this process. You can either add the whole private |
120 |
| -workspace to your own `$GOPATH` or you can `go get` the library into your own |
121 |
| -`$GOPATH` until it is properly vendored into Kubernetes. |
122 |
| - |
123 |
| -Another possible complication is a dep that uses `gopdep` itself. In that case, |
124 |
| -you need to restore its dependencies, too: |
125 |
| - |
126 |
| -```sh |
127 |
| -hack/run-in-gopath.sh bash -c 'cd $GOPATH/src/example.com/go/frob && godep restore' |
128 |
| -``` |
129 |
| - |
130 |
| -If the transitive deps collide with Kubernetes deps, you may have to manually |
131 |
| -resolve things. This is where the ability to run a shell in this environment |
132 |
| -comes in handy: |
133 |
| - |
134 |
| -```sh |
135 |
| -hack/run-in-gopath.sh bash --norc --noprofile |
136 |
| -``` |
137 |
| - |
138 |
| -### Updating a dep |
139 |
| - |
140 |
| -Sometimes we already have a dep, but the version of it is wrong. Because of the |
141 |
| -way that `godep` and `go get` interact (badly) it's generally easiest to hit it |
142 |
| -with a big hammer: |
143 |
| - |
144 |
| -```sh |
145 |
| -hack/run-in-gopath.sh bash -c 'rm -rf $GOPATH/src/example.com/go/frob' |
146 |
| -hack/run-in-gopath.sh go get -d example.com/go/frob |
147 |
| -hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v2.0.0' |
148 |
| -``` |
149 |
| - |
150 |
| -This will remove the code, re-fetch it, and sync to your desired version. |
151 |
| - |
152 |
| -### Removing a dep |
153 |
| - |
154 |
| -This happens almost for free. If you edit Kubernetes code and remove the last |
155 |
| -use of a given dependency, you only need to restore and save the deps, and the |
156 |
| -`godep` tool will figure out that you don't need that dep any more: |
157 |
| - |
158 |
| -## Saving deps |
159 |
| - |
160 |
| -Now that you have made your changes - adding, updating, or removing the use of a |
161 |
| -dep - you need to rebuild the dependency database and make changes to the |
162 |
| -`vendor/` directory. |
163 |
| - |
164 |
| -```sh |
165 |
| -hack/run-in-gopath.sh hack/godep-save.sh |
166 |
| -``` |
167 |
| - |
168 |
| -This will run through all of the primary targets for the Kubernetes project, |
169 |
| -calculate which deps are needed, and rebuild the database. It will also |
170 |
| -regenerate other metadata files which the project needs, such as BUILD files and |
171 |
| -the LICENSE database. |
172 |
| - |
173 |
| -Commit the changes before updating deps in staging repos. |
174 |
| - |
175 |
| -## Saving deps in staging repos |
176 |
| - |
177 |
| -Kubernetes stores some code in a directory called `staging` which is handled |
178 |
| -specially, and is not covered by the above. If you modified any code under |
179 |
| -staging, or if you changed a dependency of code under staging (even |
180 |
| -transitively), you'll also need to update deps there: |
181 |
| - |
182 |
| -```sh |
183 |
| -./hack/update-staging-godeps.sh |
184 |
| -``` |
185 |
| - |
186 |
| -Then commit the changes generated by the above script. |
187 |
| - |
188 |
| -## Commit messages |
189 |
| - |
190 |
| -Terse messages like "Update foo.org/bar to 0.42" are problematic |
191 |
| -for maintainability. Please include in your commit message the |
192 |
| -detailed reason why the dependencies were modified. |
193 |
| - |
194 |
| -Too commonly dependency changes have a ripple effect where something |
195 |
| -else breaks unexpectedly. The first instinct during issue triage |
196 |
| -is to revert a change. If the change was made to fix some other |
197 |
| -issue and that issue was not documented, then a revert simply |
198 |
| -continues the ripple by fixing one issue and reintroducing another |
199 |
| -which then needs refixed. This can needlessly span multiple days |
200 |
| -as CI results bubble in and subsequent patches fix and refix and |
201 |
| -rerefix issues. This may be avoided if the original modifications |
202 |
| -recorded artifacts of the change rationale. |
203 |
| - |
204 |
| -## Sanity checking |
205 |
| - |
206 |
| -After all of this is done, `git status` should show you what files have been |
207 |
| -modified and added/removed. Make sure to sanity-check them with `git diff`, and |
208 |
| -to `git add` and `git rm` them, as needed. It is commonly advised to make one |
209 |
| -`git commit` which includes just the dependencies and Godeps files, and |
210 |
| -another `git commit` that includes changes to Kubernetes code to use (or stop |
211 |
| -using) the new/updated/removed dependency. These commits can go into a single |
212 |
| -pull request. |
213 |
| - |
214 |
| -Before sending your PR, it's a good idea to sanity check that your |
215 |
| -Godeps.json file and the contents of `vendor/ `are ok: |
216 |
| - |
217 |
| -```sh |
218 |
| -hack/run-in-gopath.sh hack/verify-godeps.sh |
219 |
| -``` |
220 |
| - |
221 |
| -All this script will do is a restore, followed by a save, and then look for |
222 |
| -changes. If you followed the above instructions, it should be clean. If it is |
223 |
| -not, you get to figure out why. |
224 |
| - |
225 |
| -## Manual updates |
226 |
| - |
227 |
| -It is sometimes expedient to manually fix the `Godeps.json` file to |
228 |
| -minimize the changes. However, without great care this can lead to failures |
229 |
| -with the verifier scripts. The kubernetes codebase does "interesting things" |
230 |
| -with symlinks between `vendor/` and `staging/` to allow multiple Go import |
231 |
| -paths to coexist in the same git repo. |
232 |
| - |
233 |
| -The verifiers, including `hack/verify-godeps.sh` *must* pass for every pull |
234 |
| -request. |
235 |
| - |
236 |
| -## Reviewing and approving dependency changes |
237 |
| - |
238 |
| -Particular attention to detail should be exercised when reviewing and approving |
239 |
| -PRs that add/remove/update dependencies. Importing a new dependency should bring |
240 |
| -a certain degree of value as there is a maintenance overhead for maintaining |
241 |
| -dependencies into the future. |
242 |
| - |
243 |
| -When importing a new dependency, be sure to keep an eye out for the following: |
244 |
| -- Is the dependency maintained? |
245 |
| -- Does the dependency bring value to the project? Could this be done without |
246 |
| - adding a new dependency? |
247 |
| -- Is the target dependency the original source, or a fork? |
248 |
| -- Is there already a dependency in the project that does something similar? |
249 |
| -- Does the dependency have a license that is compatible with the Kubernetes |
250 |
| - project? |
251 |
| - |
252 |
| -Additionally: |
253 |
| -- Look at the godeps file. Check that the only changes are what the PR claims |
254 |
| - them to be. |
255 |
| -- Check if there is a tagged release we can vendor instead of a random hash |
256 |
| -- Scan the imported code for things like init() functions |
257 |
| -- Look at the Kubernetes code changes and make sure they are appropriate |
258 |
| - (e.g. renaming imports or similar). You do not need to do feature code review. |
259 |
| -- If this is all good, approve, but don't LGTM, unless you also do code review |
260 |
| - or unless it is trivial (e.g. moving from k/k/pkg/utils -> k/utils). |
261 |
| - |
262 |
| -All new dependency licenses should be reviewed by @kubernetes/dep-approvers to ensure that they |
263 |
| -are compatible with the Kubernetes project license. It is also important to note |
264 |
| -and flag if a license has changed when updating a dependency, so that these can |
265 |
| -also be reviewed. |
266 |
| - |
267 |
| -For reference, whitelisted licenses as per the CNCF Whitelist Policy are |
268 |
| -mentioned [here](https://git.k8s.io/sig-release/licensing/README.md#licenses-for-dependencies). |
| 2 | +See [current documentation for working with dependencies](./vendor.md) for master branch development. |
0 commit comments