Skip to content

Commit 686b32d

Browse files
AidanDelaneyAidanDelaney
authored
Merge pull request cds-hooks#511 from buildpacks/buildpack-author-phases
Provide buildpack authors with more context on phases
2 parents 0655cfc + 5862b24 commit 686b32d

File tree

5 files changed

+1194
-1
lines changed

5 files changed

+1194
-1
lines changed

‎content/docs/buildpack-author-guide/build-phases-overview.md

Lines changed: 379 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,379 @@
1+
+++
2+
title="Overview of Buildpacks Phases"
3+
weight=15
4+
+++
5+
6+
We work through a full example of building a "hello world" NodeJs web application. In the example we run `pack` on the NodeJS application to produce an application image. We assume that we have a NodeJS buildpack, `registry.fake/buildpacks/nodejs:latest`, that is decomposed into buildpacks that help with the build. We expand each of the buildpacks phases to explain the process. Throughout the example we take a production-level view of their operation. For example, our assumed NodeJS buildpack will be described to create different build, cache and launch layers in a manner similar to how a real NodeJS buildpack would operate.
7+
8+
## NodeJS buildpack
9+
10+
The example NodeJS buildpack is a meta-buildpack. It is composed of
11+
12+
* `node-engine` buildpack that provides the `node` and `npm` binaries,
13+
* `yarn` buildpack that provides the `yarn` binary,
14+
* `yarn-install` and `npm-install` buildpacks that install dependencies using either `yarn` or `npm`,
15+
* `yarn-start` and `npm-start` buildpacks that configure the entrypoint to the application image,
16+
* `procfile` a buildpack that allows developers to provide a [Heroku-style](https://devcenter.heroku.com/articles/procfile#procfile-format) entrypoint for the image.
17+
18+
The `nodejs` buildpack itself is a meta-buildpack which defines two **order groups**. Here we represent the order groups visually:
19+
20+
![nodejs order groups](/images/order-groups.svg)
21+
22+
The order group containing `yarn` logic has higher precedence in the `nodejs` buildpack than the order group containing `npm-install`. In both order groups the `procfile` buildpack is optional. The function of order groups will become more clear as we proceed through our example.
23+
24+
## Running `pack`
25+
26+
Our example NodeJS application is a "hello world" REST-like API. Any request to the `/` URL results in the response `{"message", "Hello world"}`. Our application contains the two source files
27+
28+
```command
29+
$ tree .
30+
.
31+
├── index.js
32+
└── package.json
33+
34+
0 directories, 2 files
35+
```
36+
37+
The core logic is contained in `index.js`, our dependencies are declared in `package.json` and `package.json` also describes how to start our application. The logic in `index.js` listens on a `PORT`, provided as an environment variable:
38+
39+
```js
40+
const express = require('express')
41+
const app = express()
42+
43+
app.get('/', (req, res) => {
44+
res.send({'message': 'Hello World'})
45+
})
46+
47+
var port = process.env.PORT || '8080';
48+
app.listen(port)
49+
```
50+
51+
The dependencies are provided using the mechanism an NodeJS developer expects. In this example we depend upon `express` to provide a framework for our REST-like service.
52+
53+
```json
54+
{
55+
"name": "hello-world",
56+
"version": "1.0.0",
57+
"description": "A hello-world nodejs example",
58+
"main": "index.js",
59+
"scripts": {
60+
"start": "node index.js"
61+
},
62+
"dependencies": {
63+
"express": "^4.18.1"
64+
}
65+
}
66+
```
67+
68+
Finally, we describe the container entrypoint using a script in the `package.json`. The script must be named `start` according to NodeJS convention. Here we see that the entrypoint should run the provided `node index.js` command.
69+
70+
We build our application using the default builder and specify to only use the `nodejs` meta-buildpack in the build. The restriction to use only the `nodejs` meta-buildpack simplifies the explanation as that buildpack provides only two order groups.
71+
72+
<asciinema-player
73+
idle-time-limit="0.5s"
74+
font-size="medium"
75+
poster="data:text/plain,$ pack build example --verbose --buildpack docker://registry.fake/buildpacks/nodejs:latest" src="/images/pack-hello-world-nodejs.cast"></asciinema-player>
76+
77+
Now that we understand the example application we can step through each of the Buildpack phases.
78+
79+
## Phases
80+
81+
There are five phases to a buildpacks build. These logic for each phaze is provided by a binary within [`lifecycle`](https://github.com/buildpacks/lifecycle) and the orchestration of running each of the binaries is the responsibility of the Buildpacks platform. In this case we are using `pack` as our Buildpacks platform.
82+
83+
At a high-level each layer:
84+
85+
* Analyze phase - Reads metadata from any previously built image and ensures we have access to the OCI registry to be able to write the image we will build.
86+
* Detect phase - Chooses buildpacks (via /bin/detect) and produces a build plan.
87+
* Restore phase - Restores layer metadata from the previous image and from the cache, and restores cached layers.
88+
* Build phase - Executes buildpacks (via /bin/build).
89+
* Export phase - Creates an image and caches layers.
90+
91+
We consider each of the buildpacks phases in the context of our invocation of `pack build example --buildpack docker://registry.fake/buildpacks/nodejs:latest`.
92+
93+
### Phase 1: Analyze
94+
95+
The analyze phase checks a registry for previous images called `example`. It resolves the image metadata making it available to the subsequent restore phaze. In addition, analyze verifies that we have write access to the registry to create or update the image called `example`.
96+
97+
In our case `pack` tells us that there is no previous `example` image. It provides the output.
98+
99+
```
100+
Previous image with name "example" not found
101+
Analyzing image "98070ee549c522cbc08d15683d134aa0af1817fcdc56f450b07e6b4a7903f9b
102+
0"
103+
```
104+
105+
The analyze phase writes to disk the metadata it has found. The metadata is used by the restore phase when restoring cached image layers.
106+
107+
### Phase 2: Detect
108+
109+
The detect phase runs the `detect` binary of each buildpack in the order provided in the buildpack metadata.
110+
111+
The invocation of `pack build example --buildpack docker://registry.fake/buildpacks/nodejs:latest` explicitly defines a buildpack order. The command line invocation includes a single `nodejs` buildpack. In our example the detect phase runs the `detect` binary from each buildpack in the first order group. The `yarn-install` `detect` binary will fail as no yarn lock file is present in our source project. As the `detect` binary of a non-optional buildpack has failed, then detection of the entire build group containing `yarn-install` has failed. The detect phase then proceeds to run the `detect` binary of each buildpack in the second order group. As all non-optional buildpacks in this group have passed the detect phase, all the passing buildpacks are added to the build order.
112+
113+
![nodejs order groups](/images/order-groups-detect.svg)
114+
115+
<!-- ALERT-NOTE -->
116+
> The [specification for order resolution](https://github.com/buildpacks/spec/blob/main/buildpack.md#order-resolution) shows each order group as a matrix and the resolution as an operation on matrices.
117+
118+
The above diagram shows that the `detect` binary of each required buildpack in the second order group passes. The detect phase is summarized by pack as
119+
120+
```
121+
===> DETECTING
122+
======== Output: example/[email protected] ========
123+
failed
124+
======== Output: example/[email protected] ========
125+
failed
126+
======== Results ========
127+
pass: example/[email protected]
128+
pass: example/[email protected]
129+
fail: example/[email protected]
130+
fail: example/[email protected]
131+
skip: example/[email protected]
132+
======== Results ========
133+
pass: example/[email protected]
134+
pass: example/[email protected]
135+
pass: example/[email protected]
136+
skip: example/[email protected]
137+
Resolving plan... (try #1)
138+
fail: example/[email protected] requires npm
139+
Resolving plan... (try #2)
140+
3 of 4 buildpacks participating
141+
example/node-engine 0.0.5
142+
example/npm-install 0.0.2
143+
example/npm-start 0.0.2
144+
```
145+
146+
The output of the detect phase includes a **build plan**.
147+
148+
> The build plan is a toml file containing declarations of what each buildpack provides and what it requires.
149+
150+
For example, the `example/node-engine` buildpack will contribute metadata stating that it provides `node`. In addition, it declares a requirement for the subsequent build phase to satisfy the `node` with some specific metadata. That is to say, it requires that the build phase installs a node 14.17.5 runtime.
151+
152+
```toml
153+
[[provides]]
154+
name = "node"
155+
156+
[[requires]]
157+
name = "node"
158+
[requires.metadata]
159+
version = "14.17.5"
160+
version-source = "14.17.5"
161+
```
162+
163+
The `npm-install` `detect` binary contributes more metadata to the build plan. `npm-install` requires that a node runtime available. We will find that the `example/node-engine` buildpack will satisfy that requirement. `npm-install` also requires that the build phase satisfies the `npm` requirement. It is interesting to note that the `npm-install` detect phase attaches additional `build = true` metadata to each of its requirements. We will see that this is interpreted by the build phase as the `node` runtime and `npm` install process contributing **build layers** i.e. as layers that are available to subsequent buildpacks for the purposes of building the application.
164+
165+
```toml
166+
[[provides]]
167+
name = "npm"
168+
169+
[[requires]]
170+
name = "node"
171+
[requires.metadata]
172+
build = true
173+
174+
[requires]
175+
name = "npm"
176+
[entries.requires.metadata]
177+
build = true
178+
```
179+
180+
The `detect` binary of `npm-start` detects the existence of a `start` command within `package.json`. If such a command exists it contributes requirements to the build plan that require `node` and `npm` to be available within a **launch layer** i.e. as layers in the output image.
181+
182+
```toml
183+
[[requires]]
184+
name = "node"
185+
[requires.metadata]
186+
launch = true
187+
188+
[[requires]]
189+
name = "npm"
190+
[requires.metadata]
191+
launch = true
192+
```
193+
194+
The build plans provided by each `detect` binary are resolved as an output of the detect phase. The detect phase can fail if a buildpack requires a dependency that cannot be resolved.
195+
196+
We have provided a complete example of the detect phase for our "hello world" NodeJS application. Given input application code and buildpacks, the output is largely a declarative TOML file passed to the build phase. The detected order is written to a `group.toml` file which is used in the restore phase.
197+
198+
### Phase 3: Restore
199+
200+
The restore phase uses metadata provided by the analyze phase and `group.toml` from the detect phase. It outputs cached layers to `$CNB_LAYERS_DIR/<buildpack-id>` in the application image. If a layer is restored at the restore phase then we skip the build phase for that layer.
201+
202+
In our running example there is no previous build. The analyze phase returned no cached image. Therefore the restore phase is similarly quiet and `pack` outputs the following:
203+
204+
```
205+
===> RESTORING
206+
Reading buildpack directory: /layers/example_node-engine
207+
Not restoring "example/node-engine:node" from cache, marked as launch=true
208+
Reading buildpack directory: /layers/example_npm-install
209+
Not restoring "example/npm-install:modules" from cache, marked as launch=true
210+
Not restoring "example/npm-install:npm-cache" from cache, marked as launch=true
211+
Reading buildpack directory: /layers/example_npm-start
212+
Reading buildpack directory: /layers/example_procfile
213+
Reading buildpack directory: /layers/example_node-engine
214+
Reading buildpack directory: /layers/example_npm-install
215+
Reading buildpack directory: /layers/example_npm-start
216+
Reading buildpack directory: /layers/example_procfile
217+
```
218+
219+
Having resolved the build plan and any cached layers, the build phase can concentrate on creating layers.
220+
221+
### Phase 4: Build
222+
223+
The build phase is provided as input the order in which to run the buildpacks (`group.toml`) and the build plan (`plan.toml`). The build phase runs the `build` binary of each buildpack. The build binary for a buildpack outputs zero or more layers into `$(CNB_LAYERS_DIR)/<buildpack-id>` and writes metadata for each layer as TOML files in that directory. Buildpacks should also provide Software Bill-of-Materials for each layer that they contribute to the build.
224+
225+
In our running NodeJS example the build phase runs the `build` binary from the
226+
227+
* `example/node-engine` buildpack, followed by
228+
* `example/node-install`, followed by
229+
* `example/node-start`, and finally
230+
* `example/procfile`.
231+
232+
In this example, each buildpack contributes a single layer to the output image.
233+
234+
Each invocation of `build` is passed a **buildpack plan** specific to each buildpack. The buildpack plan are those entries from the build plan that reference something provided by that buildpack.
235+
236+
#### `node-engine` build execution
237+
238+
The `example/node-engine` buildpack provides `node`. Therefore all entries in the build plan that require `node` are passed in the buildpack plan for this buildpack. The buildpack plan provided to `example/node-engine` in this example is
239+
240+
```toml
241+
[[entries]]
242+
name = "node"
243+
244+
[entries.metadata]
245+
version = "14.17.5"
246+
version-source = "14.17.5"
247+
248+
[entries.metadata]
249+
build = true
250+
251+
[entries.metadata]
252+
launch = true
253+
```
254+
255+
Given the metadata from the buildpack plan, an archive containing `node` version `14.17.5` is fetched from a network source and expanded as a layer contributed by this buildpack.
256+
257+
```
258+
===> BUILDING
259+
Starting build
260+
Running build for buildpack example/[email protected]
261+
Looking up buildpack
262+
Finding plan
263+
Running build for buildpack node-engine 0.0.5
264+
Creating plan directory
265+
Preparing paths
266+
Running build command
267+
node-engine 0.0.5
268+
Resolving Node Engine version
269+
Candidate version sources (in priority order):
270+
-> ""
271+
<unknown> -> ""
272+
273+
Selected Node Engine version (using ): 14.17.5
274+
275+
Executing build process
276+
Installing Node Engine 14.17.5
277+
Completed in 31.715s
278+
279+
Configuring build environment
280+
NODE_ENV -> "production"
281+
NODE_HOME -> "/layers/example_node-engine/node"
282+
NODE_VERBOSE -> "false"
283+
284+
Configuring launch environment
285+
NODE_ENV -> "production"
286+
NODE_HOME -> "/layers/example_node-engine/node"
287+
NODE_VERBOSE -> "false"
288+
289+
Writing profile.d/0_memory_available.sh
290+
Calculates available memory based on container limits at launch time.
291+
Made available in the MEMORY_AVAILABLE environment variable.
292+
293+
Processing layers
294+
Updating environment
295+
Reading output files
296+
Updating buildpack processes
297+
Updating process list
298+
Finished running build for buildpack example/[email protected]
299+
```
300+
301+
The `node-engine` buildpack contributes a layer containing `bin/node`, the supporting libraries for `bin/node` and sets environment variables that are specific to the node binary.
302+
303+
#### `npm-install` build execution
304+
305+
The `npm-install` buildpack contributes at least two layers. One layer is a cache-only layer and is never exported as part of the application image. The cache layer holds `npm` metadata and the installed `node_modules`. If `npm` is also required to provide a launch layer, as it is in our running example, then the `node_modules` of the cache layer are provided as a layer in the application image. In addition, a script is provided to set up some symlinks. The script executes on container startup ensuring that our `index.js` can resolve JavaScript modules installed in the layer.
306+
307+
```
308+
Running build for buildpack example/[email protected]
309+
Looking up buildpack
310+
Finding plan
311+
Running build for buildpack npm-install 0.0.2
312+
Creating plan directory
313+
Preparing paths
314+
Running build command
315+
npm-install 0.0.2
316+
Resolving installation process
317+
Process inputs:
318+
node_modules -> "Not found"
319+
npm-cache -> "Not found"
320+
package-lock.json -> "Found"
321+
322+
Selected NPM build process: 'npm ci'
323+
324+
Executing build process
325+
Running 'npm ci --unsafe-perm --cache /layers/example_npm-install/npm-cache'
326+
Completed in 11.255s
327+
328+
Configuring launch environment
329+
NPM_CONFIG_LOGLEVEL -> "error"
330+
331+
Configuring environment shared by build and launch
332+
PATH -> "$PATH:/layers/example_npm-install/modules/node_modules/.bin"
333+
334+
335+
Processing layers
336+
Updating environment
337+
Reading output files
338+
Updating buildpack processes
339+
Updating process list
340+
Finished running build for buildpack example/[email protected]
341+
```
342+
343+
In the verbose output of an execution of `npm-install`'s `build` binary we can observe that the `NPM_CONFIG_LOGLEVEL` environment variable is set on the application image and the `PATH` environmental variable is extended so that binaries installed via `npm` can be found.
344+
345+
#### `npm-start` build execution
346+
347+
The `npm-start` build phase creates an entrypoint in the application image.
348+
349+
```
350+
Running build for buildpack example/[email protected]
351+
Looking up buildpack
352+
Finding plan
353+
Running build for buildpack npm-start 0.0.2
354+
Creating plan directory
355+
Preparing paths
356+
Running build command
357+
npm-start 0.0.2
358+
Assigning launch processes
359+
web: node index.js
360+
361+
Processing layers
362+
Updating environment
363+
Reading output files
364+
Updating buildpack processes
365+
Updating process list
366+
Finished running build for buildpack example/[email protected]
367+
```
368+
369+
The output layers contributed by each buildpack are then exported as an OCI image.
370+
371+
### Phase 5: Export
372+
373+
The export phase constructs a new OCI image using all the layers provided in the build phase. In addition the application source is added as a layer to the output image and the image `ENTRYPOINT` is set. The exported image, which we call `example`, contains only launch layers. Cache layers are preserved on the local machine for subsequent builds.
374+
375+
Our NodeJS example image requires an entrypoint called `web`. The `web` entrypoint is implemented on the application image as a symlink to the `launcher` binary. As we have specified a single entrypoint, this then becomes the default entrypoint of the image.
376+
377+
## Summary
378+
379+
We have taken a detailed look at how buildpacks are used to build a sample application. The meta-buildpack contains two order groups and we have seen examples of how an order group is resolved. In addition we have looked at the contributions that a buildpack makes to the build plan and considerd how these are resolved into a buildpack plan to be provided to the build phase of specific buildpacks. Finally, we have briefly considered how the analyze and restore phases can allow advanced caching strategies.

0 commit comments

Comments
 (0)