|
| 1 | +# Meta |
| 2 | +[meta]: #meta |
| 3 | +- Name: Add Cloud Native Buildpacks |
| 4 | +- Start Date: 2023-05-01 |
| 5 | +- Author: @gerg |
| 6 | +- Contributing Authors: @ryanmoran, @robdimsdale, @dsabeti, @aramprice, @dsboulder |
| 7 | +- Status: Draft |
| 8 | +- RFC Pull Request: #591 |
| 9 | + |
| 10 | +## Summary |
| 11 | + |
| 12 | +[Cloud Native Buildpacks (CNBs)](https://buildpacks.io/), also known as v3 |
| 13 | +buildpacks, are the current generation of buildpacks, and offer some |
| 14 | +improvements over the v2 buildpacks that CF Deployment currently uses. The |
| 15 | +Cloud Foundry Foundation already has an implementation of Cloud Native |
| 16 | +Buildpacks via the [Korifi](https://www.cloudfoundry.org/technology/korifi/) |
| 17 | +project, however these CNBs are not currently integrated with core CF |
| 18 | +Deployment. |
| 19 | + |
| 20 | +This RFC proposes adding CNB support to Cloud Foundry and including the Korifi |
| 21 | +buildpacks in CF Deployment. |
| 22 | + |
| 23 | +## Problem |
| 24 | + |
| 25 | +The v2 buildpacks are effectively in maintenance mode and do not receive |
| 26 | +substantial new features. By not integrating with v3 buildpacks, Cloud Foundry |
| 27 | +is missing out on new capabilities like application |
| 28 | +[SBOM](https://en.wikipedia.org/wiki/Software_supply_chain) generation, new |
| 29 | +buildpacks like the Java Native and Web Servers CNBs, as well as any new |
| 30 | +features that are added to the still-actively-developed v3 buildpacks. |
| 31 | + |
| 32 | +## Proposal |
| 33 | + |
| 34 | +### Overview |
| 35 | + |
| 36 | +Cloud Foundry integration with CNBs will be executed over at least two phases. |
| 37 | + |
| 38 | +In the first phase, the Paketo buildpacks will be "shimmed" to work with Cloud |
| 39 | +Foundry's existing v2 buildpack interface. These shimmed Paketo buildpacks will |
| 40 | +be bundled with select [CNB lifecycle](https://github.com/buildpacks/lifecycle) |
| 41 | +binaries and lightweight orchestration binaries. At build time, the Cloud |
| 42 | +Foundry [buildpack lifecycle](https://github.com/cloudfoundry/buildpackapplifecycle) |
| 43 | +will invoke the orchestration binaries, which will map the v2 build process to |
| 44 | +the v3 build process. |
| 45 | + |
| 46 | +In the second phase, unmodified Paketo buildpacks will be natively supported by |
| 47 | +Cloud Foundry. This will be implemented by integrating the buildpack shim logic |
| 48 | +into a Cloud Foundry lifecycle. |
| 49 | + |
| 50 | +Depending on learnings from the first two phases of CNB integration or other |
| 51 | +factors, there may be additional phases of work, however these additional |
| 52 | +phases are not covered by this RFC. |
| 53 | + |
| 54 | +Notably, this proposal does NOT include building or running OCI images on Cloud |
| 55 | +Foundry. This proposal is to use Paketo buildpacks to build and run Cloud |
| 56 | +Foundry droplets. This proposal does not preclude future efforts to integrate |
| 57 | +Cloud Foundry with OCI images, container registries, etc. |
| 58 | + |
| 59 | +### Goals |
| 60 | + |
| 61 | +1. Align CF Deployment with the main focus of the Buildpacks team |
| 62 | +1. Bring new build capabilities to CF Deployment users, including: |
| 63 | + 1. [Application SBOMs](https://paketo.io/docs/howto/sbom/) |
| 64 | + 1. [Web Servers Buildpack](https://github.com/paketo-buildpacks/web-servers) |
| 65 | + 1. [Java Native Buildpack](https://github.com/paketo-buildpacks/java-native-image) |
| 66 | + 1. Easier buildpack customization/composition |
| 67 | +1. Increase cohesion and application portability between CF Deployment and |
| 68 | + Korifi, via mutual Paketo integration |
| 69 | +1. Increase user base for the CNB lifecycle and Paketo buildpacks |
| 70 | +1. Open the door for eventual deprecation of the v2 buildpacks, reducing |
| 71 | + maintenance costs (v2 buildpack deprecation is NOT included in this RFC) |
| 72 | + |
| 73 | +### High Level Implementation |
| 74 | + |
| 75 | +In phase one, Cloud Foundry will produce a series of experimental shimmed |
| 76 | +buildpacks that include: |
| 77 | +1. A Paketo CNB |
| 78 | +1. Select executables from the CNB lifecycle |
| 79 | +1. Orchestrator executables that conform to the v2 buildpack interface |
| 80 | + |
| 81 | +Cloud Foundry users will interact with the shimmed buildpacks the same as they |
| 82 | +do for any other buildpack. For example, the following commands will work as |
| 83 | +expected: |
| 84 | + |
| 85 | +``` |
| 86 | +$ cf create-buildpack ruby_cnb ./ruby_cnb_shim.zip |
| 87 | +... |
| 88 | +$ cf push my-app --buildpack=ruby_cnb |
| 89 | +``` |
| 90 | + |
| 91 | +At build time, the procedure will be as follows: |
| 92 | +1. The Cloud Foundry Buildpack App Lifecycle's Builder will invoke shim |
| 93 | + orchestrator executables, consistent with the v2 buildpack interface |
| 94 | +1. The shim orchestrator will invoke bundled CNB lifecycle executables |
| 95 | +1. The CNB lifecycle executables will invoke Paketo buildpack executables, |
| 96 | + consistent with the v3 buildpack interface |
| 97 | +1. The Paketo buildpack will build a set of "layer" directories |
| 98 | +1. The shim orchestrator will package the layer directories into a Cloud |
| 99 | + Foundry droplet and return it back to the Cloud Foundry Lifecycle |
| 100 | + |
| 101 | +For a proof of concept of shimmed buildpacks, see @ryanmoran's |
| 102 | +[cfnb](https://github.com/ryanmoran/cfnb) project. Developing the shim |
| 103 | +externally at first will allow for more rapid iteration, without disrupting the |
| 104 | +core Cloud Foundry runtime. |
| 105 | + |
| 106 | +During phase two, the shim logic will be integrated into the Cloud Foundry |
| 107 | +lifecycle itself, allowing for "native" Paketo buildpack support. The details |
| 108 | +of this integration will be decided once phase one is complete. |
| 109 | + |
| 110 | +### Shim at Build Time |
| 111 | + |
| 112 | +At build time, the shim orchestrator will expose the following v2 buildpack |
| 113 | +executables: |
| 114 | + |
| 115 | +| Executable | Function | |
| 116 | +| --- | --- | |
| 117 | +| `detect` | Invoke the CNB Lifecycle `detector` executable | |
| 118 | +| `supply` | Invoke the CNB Lifecycle `builder` executable | |
| 119 | +| `finalize` | Add CNB `launcher` executable and runtime configuration files to build layers | |
| 120 | +| `release` | Package the generated layer directories into a Cloud Foundry droplet | |
| 121 | + |
| 122 | +These executables will be invoked by the Cloud Foundry Buildpack App Lifecycle's |
| 123 | +builder. |
| 124 | + |
| 125 | +### Shim at Run Time |
| 126 | + |
| 127 | +At run time, the Cloud Foundry Buildpack App Lifecycle launcher will invoke the |
| 128 | +CNB Lifecycle `launcher` executable that was included in the droplet at build |
| 129 | +time. In order to support custom start commands for apps using the CNB |
| 130 | +buildpack shims, the CNB launcher invocation will be prepended to the app start |
| 131 | +command using the [`entrypoint_prefix` hook](https://github.com/cloudfoundry/buildpackapplifecycle/commit/29feb13caeff646f35585eee865c376c818fc2ea) |
| 132 | +in the CF Lifecycle launcher. |
| 133 | + |
| 134 | +### Service Bindings |
| 135 | + |
| 136 | +The largest anticipated breaking change from v2 to v3 buildpacks is the change |
| 137 | +in how service bindings are consumed by buildpacks. The v3 buildpacks expect |
| 138 | +service bindings to follow the [Kubernetes Service Binding specification](https://github.com/servicebinding/spec), |
| 139 | +which places service binding credentials on the filesystem. Cloud Foundry and |
| 140 | +v2 buildpacks instead use environment variables for service binding |
| 141 | +credentials. |
| 142 | + |
| 143 | +In order to securely place credentials on the filesystem, Diego will need the |
| 144 | +ability to mount in-memory filesystems (e.g. `tmpfs`) in application |
| 145 | +containers. Once an in-memory filesystem is available, select `VCAP_SERVICES` |
| 146 | +environment variables can be translated into their Kubernetes Service Binding |
| 147 | +equivalents by the shim orchestrator, for consumption by the v3 buildpacks. |
| 148 | + |
| 149 | +### Governance and Release |
| 150 | + |
| 151 | +A `cnb-shim-builder` repository will be added to the Application Runtime |
| 152 | +Interfaces working group. This repository will contain the tooling necessary |
| 153 | +for converting existing Paketo buildpacks into shimmed CNBs. |
| 154 | + |
| 155 | +The shimmed buildpacks will be experimental bosh releases, belonging to the |
| 156 | +Application Runtime Interfaces working group. The proposed list of buildpacks |
| 157 | +is as follows (in rough priority order): |
| 158 | +1. `java_native_cnb_release` |
| 159 | +1. `java_cnb_release` |
| 160 | +1. `web_servers_cnb_release` |
| 161 | +1. `dotnet_core_cnb_release` |
| 162 | +1. `nodejs_cnb_release` |
| 163 | +1. `python_cnb_release` |
| 164 | +1. `ruby_cnb_release` |
| 165 | +1. `procfile_cnb_release` |
| 166 | +1. `go_cnb_release` |
| 167 | +1. `nginx_cnb_release` |
| 168 | +1. `php_cnb_release` |
| 169 | +1. `rust_cnb_release` |
| 170 | + |
| 171 | +It is not required to shim all buildpacks during phase 1 of this project. |
| 172 | +Depending on the effort to shim buildpacks and other factors, it may make sense |
| 173 | +to move to native CNB support before all Paketo buildpacks are shimmed. During |
| 174 | +phase 2, the above bosh release repositories will be converted to unshimmed |
| 175 | +bosh releases of the corresponding Paketo buildpacks. |
| 176 | + |
| 177 | +The CNBs will initially be opt-in for CF Deployment operators via an |
| 178 | +experimental ops file. Once the buildpacks have reached a suitable level of |
| 179 | +maturity and stability, likely after phase 2, they will be added to the default |
| 180 | +set of buildpacks installed as part of CF Deployment. |
| 181 | + |
| 182 | +### Stacks |
| 183 | + |
| 184 | +Initially, the shimmed Paketo buildpacks will be compatible with `cflinuxfs4` to |
| 185 | +ease development and adoption. Paketo provides [multiple stacks](https://paketo.io/docs/concepts/stacks/#what-paketo-stacks-are-available) |
| 186 | +that are compatible with the Paketo buildpacks. There is an opportunity to |
| 187 | +adopt some or all of the Paketo stacks into CF Deployment, in addition to the |
| 188 | +buildpacks. The Paketo buildpacks have greater cohesion with the Paketo |
| 189 | +buildpacks than with `cflinuxfs4`, and the Paketo "base" and "tiny" stacks could |
| 190 | +offer security-conscious CF Deployment users stacks with fewer native |
| 191 | +dependencies than are currently included with `cflinuxfs4`. This RFC does not |
| 192 | +cover the adoption of additional stacks into CF Deployment, but it does open |
| 193 | +the door to add these stacks in the future. |
| 194 | + |
0 commit comments