Skip to content

Commit 7da340c

Browse files
AP-HuntAP-Hunt
authored
Merge pull request #591 from cloudfoundry/rfc_cnbs
RFC to add Cloud Native Buildpacks to CF Deployment
2 parents 0e20b0f + dc568c2 commit 7da340c

File tree

1 file changed

+240
-0
lines changed

1 file changed

+240
-0
lines changed

‎toc/rfc/rfc-draft-add-cnbs.md

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
# Meta
2+
[meta]: #meta
3+
- Name: Add Cloud Native Buildpacks
4+
- Start Date: 2023-05-01
5+
- Author: @gerg
6+
- Contributing Authors: @ryanmoran, @robdimsdale, @dsabeti, @aramprice, @dsboulder
7+
- Status: Draft
8+
- RFC Pull Request: #591
9+
10+
## Summary
11+
12+
[Cloud Native Buildpacks (CNBs)](https://buildpacks.io/), also known as v3
13+
buildpacks, are the current generation of buildpacks and offer some
14+
improvements over the v2 buildpacks that CF Deployment currently uses. The
15+
Cloud Foundry Foundation already has an implementation of Cloud Native
16+
Buildpacks via the [Paketo](https://paketo.io/) project, however these CNBs are
17+
not currently integrated with core CF Deployment.
18+
19+
This RFC proposes adding CNB support to Cloud Foundry and including the Paketo
20+
buildpacks in CF Deployment.
21+
22+
## Problem
23+
24+
The v2 buildpacks are effectively in maintenance mode and do not receive
25+
substantial new features. By not integrating with v3 buildpacks, Cloud Foundry
26+
is missing out on new capabilities like app
27+
[SBOM](https://en.wikipedia.org/wiki/Software_supply_chain) generation, new
28+
buildpacks like the Java Native and Web Servers CNBs, as well as any new
29+
features that are added to the still-actively-developed v3 buildpacks.
30+
31+
## Proposal
32+
33+
### Overview
34+
35+
Cloud Foundry integration with CNBs will be executed over multiple phases.
36+
37+
In the first phase (the focus of this RFC), the Paketo buildpacks will be
38+
"shimmed" to work with Cloud Foundry's existing v2 buildpack interface. These
39+
shimmed Paketo buildpacks will be bundled with select
40+
[CNB lifecycle](https://github.com/buildpacks/lifecycle) binaries and
41+
lightweight orchestration binaries. At build time, the Cloud Foundry
42+
[buildpack lifecycle](https://github.com/cloudfoundry/buildpackapplifecycle)
43+
will invoke the orchestration binaries, which will map the v2 build process to
44+
the v3 build process.
45+
46+
Based on learnings from the first phase of CNB integration and other
47+
factors, there will likely be additional phases of work, however these additional
48+
phases are not covered by this RFC.
49+
50+
Notably, this proposal does NOT include building or running OCI images on Cloud
51+
Foundry. This proposal is to use Paketo buildpacks to build and run Cloud
52+
Foundry droplets. This proposal does not preclude future efforts to integrate
53+
Cloud Foundry with OCI images, container registries, etc.
54+
55+
### Goals
56+
57+
1. Align CF Deployment with the main focus of the Buildpacks team
58+
1. Bring new build capabilities to CF Deployment users, including:
59+
1. [App SBOMs](https://paketo.io/docs/howto/sbom/)
60+
1. [Web Servers Buildpack](https://github.com/paketo-buildpacks/web-servers)
61+
1. [Java Native Buildpack](https://github.com/paketo-buildpacks/java-native-image)
62+
1. Easier buildpack customization/composition
63+
1. Increase cohesion and app portability between CF Deployment and
64+
[Korifi](https://www.cloudfoundry.org/technology/korifi/), via mutual Paketo
65+
integration
66+
1. Increase user base for the CNB lifecycle and Paketo buildpacks
67+
1. Open the door for eventual deprecation of the v2 buildpacks, reducing
68+
maintenance costs (v2 buildpack deprecation is NOT included in this RFC)
69+
70+
### High Level Implementation
71+
72+
Cloud Foundry will produce a series of experimental, shimmed buildpacks that
73+
include:
74+
1. A Paketo CNB
75+
1. Select executables from the CNB lifecycle
76+
1. Orchestrator executables that conform to the v2 buildpack interface
77+
78+
Cloud Foundry users will interact with the shimmed buildpacks the same as they
79+
do for any other buildpack. For example, the following commands will work as
80+
expected:
81+
82+
```
83+
$ cf create-buildpack ruby_cnb ./ruby_cnb_shim.zip
84+
...
85+
$ cf push my-app --buildpack=ruby_cnb
86+
```
87+
88+
At build time, the procedure will be as follows:
89+
1. The Cloud Foundry Buildpack App Lifecycle's Builder will invoke the shim
90+
orchestrator executables, consistent with the v2 buildpack interface.
91+
1. The shim orchestrator will invoke the bundled CNB lifecycle executables.
92+
1. The CNB lifecycle executables will invoke the Paketo buildpack executables,
93+
consistent with the v3 buildpack interface.
94+
1. The Paketo buildpack will build a set of "layer" directories.
95+
1. The shim orchestrator will package the layer directories and
96+
buildpack-generated SBOMs into a Cloud Foundry droplet and return it back to
97+
the Cloud Foundry Lifecycle.
98+
99+
For a proof of concept of shimmed buildpacks, see @ryanmoran's
100+
[cfnb](https://github.com/ryanmoran/cfnb) project. Developing the shim
101+
externally at first will allow for more rapid iteration, without disrupting the
102+
core Cloud Foundry runtime.
103+
104+
105+
### Shim at Build Time
106+
107+
At build time, the shim orchestrator will expose the following v2 buildpack
108+
executables:
109+
110+
| Executable | Function |
111+
| --- | --- |
112+
| `detect` | Invoke the CNB Lifecycle `detector` executable |
113+
| `supply` | Invoke the CNB Lifecycle `builder` executable |
114+
| `finalize` | Add CNB `launcher` executable and runtime configuration files to build layers |
115+
| `release` | Package the generated layer directories into a Cloud Foundry droplet |
116+
117+
These executables will be invoked by the Cloud Foundry Buildpack App Lifecycle's
118+
builder.
119+
120+
### Shim at Run Time
121+
122+
At run time, the Cloud Foundry Buildpack App Lifecycle's Launcher will invoke the
123+
CNB Lifecycle `launcher` executable that was included in the droplet at build
124+
time. In order to support custom start commands for apps using the CNB
125+
buildpack shims, the CNB launcher invocation will be prepended to the app start
126+
command using the [`entrypoint_prefix` hook](https://github.com/cloudfoundry/buildpackapplifecycle/commit/29feb13caeff646f35585eee865c376c818fc2ea)
127+
in the CF Lifecycle launcher.
128+
129+
### Service Bindings
130+
131+
The largest anticipated breaking change from v2 to v3 buildpacks is the change
132+
in how service bindings are consumed by buildpacks. The v3 buildpacks expect
133+
service bindings to follow the [Kubernetes Service Binding specification](https://github.com/servicebinding/spec),
134+
which places service binding credentials on the filesystem. Cloud Foundry and
135+
v2 buildpacks instead use environment variables for service binding
136+
credentials. The majority of the v2 and v3 buildpacks do not consume service
137+
bindings, so solving this problem will not block execution for most buildpacks.
138+
139+
At time of writing, there are two viable solutions to integrate the Paketo
140+
buildpacks with CF Deployment: emulating Kubernetes Service Bindings in Diego
141+
or updating the Paketo buildpacks to consume Cloud Foundry environment
142+
variables.
143+
144+
#### Emulating Kubernetes Service Bindings
145+
146+
In order to securely place credentials on the filesystem, Diego will need the
147+
ability to mount in-memory filesystems (e.g. `tmpfs`) in app containers. Once
148+
an in-memory filesystem is available, select `VCAP_SERVICES` environment
149+
variables can be translated into their Kubernetes Service Binding equivalents
150+
by the shim orchestrator, for consumption by the Paketo buildpacks.
151+
152+
#### Paketo Buildpacks Consume Environment Variables
153+
154+
There is work in progress for CNBs using the `libcnb` package to read Cloud
155+
Foundry environment variables in addition to Kubernetes Service Bindings
156+
(see [buildpacks/libcnb#227](https://github.com/buildpacks/libcnb/pull/227)).
157+
Outside of the Java buildpacks, the Paketo buildpacks do not currently use
158+
`libcnb`, so the remaining Paketo buildpacks that consume service bindings
159+
will need to be updated to use `libcnb` or add equivalent logic. If the Paketo
160+
buildpacks are updated with this change, then no changes will be needed to the
161+
core Cloud Foundry runtime.
162+
163+
#### Recommendation
164+
165+
Updating the Paketo buildpacks to consume Cloud Foundry environment variables
166+
appears to be the easier option. However, given that there are at least two
167+
viable options for buildpack service binding consumption, this RFC proposes
168+
making the final decision at implementation time.
169+
170+
### Governance and Release
171+
172+
A `cnb-shim-builder` repository will be added to the Application Runtime
173+
Interfaces working group. This repository will contain the tooling necessary
174+
for converting existing Paketo buildpacks into shimmed CNBs.
175+
176+
The shimmed buildpacks will be experimental BOSH releases, belonging to the
177+
Application Runtime Interfaces working group. The proposed list of buildpacks
178+
is as follows (in rough priority order):
179+
1. `java_native_cnb_release`
180+
1. `java_cnb_release`
181+
1. `web_servers_cnb_release`
182+
1. `dotnet_core_cnb_release`
183+
1. `nodejs_cnb_release`
184+
1. `python_cnb_release`
185+
1. `ruby_cnb_release`
186+
1. `procfile_cnb_release`
187+
1. `go_cnb_release`
188+
1. `nginx_cnb_release`
189+
1. `php_cnb_release`
190+
1. `rust_cnb_release`
191+
192+
It is possible that not all buildpacks will be shimmed during the first phase
193+
of this project. Depending on the effort to shim buildpacks and other factors,
194+
it may make sense to move to later phases of CNB integration before all Paketo
195+
buildpacks are shimmed. In such an event, follow-on RFCs will describe
196+
additional phases of work.
197+
198+
The CNBs will initially be opt-in for CF Deployment operators via an
199+
experimental ops file. When installed, the Paketo buildpacks will follow a
200+
different naming convention than the v2 buildpacks (e.g. `ruby_cnb` instead
201+
of `ruby_buildpack`) and will be installed with a lower detect order than the
202+
v2 buildpacks. Once the v3 buildpacks have reached a suitable level of maturity
203+
and stability, they will be added to the default set of buildpacks installed as
204+
part of CF Deployment.
205+
206+
### Possible Future Work
207+
208+
The following areas are not directly covered by this RFC, but suggest some of
209+
the possible follow-on work that could be executed in later phases of Cloud
210+
Foundry CNB integration.
211+
212+
#### Support Unmodified Paketo Buildpacks
213+
214+
Unmodified (non-shimmed) Paketo buildpacks could be natively supported by Cloud
215+
Foundry. For instance, this could be implemented by integrating the buildpack
216+
shim logic into a new or existing Cloud Foundry lifecycle. The details of this
217+
integration should be covered by an additional RFC, once the first phase of CNB
218+
integration is complete.
219+
220+
#### Build Apps into OCI Images
221+
222+
Instead of outputting droplets from the staging process, Cloud Foundry could
223+
instead produce OCI images, consistent with other CNB platforms. This change
224+
would have a number of architectural and operator-impacting implications, and
225+
would need to be discussed in detail via another RFC.
226+
227+
#### Add Paketo Stacks
228+
229+
Initially, the shimmed Paketo buildpacks will be compatible with `cflinuxfs4` to
230+
ease development and adoption. Paketo provides
231+
[multiple stacks](https://paketo.io/docs/concepts/stacks/#what-paketo-stacks-are-available)
232+
that are compatible with the Paketo buildpacks. There is an opportunity to
233+
adopt some or all of the Paketo stacks into CF Deployment. The Paketo
234+
buildpacks have greater cohesion with the Paketo stacks than with `cflinuxfs4`,
235+
and the Paketo "base" and "tiny" stacks could offer security-conscious CF
236+
Deployment users stacks with far fewer native packages than are currently
237+
included with `cflinuxfs4`. This RFC does not cover the adoption of additional
238+
stacks into CF Deployment, but it does open the door to add these stacks in the
239+
future.
240+

0 commit comments

Comments
 (0)