Generalize Route CRD to be less Cloud Foundry-specific

[tcdowney]

tl;dr

Remove/rename some of the fields in the Route CRD to be less Cloud Foundry-specific.

Background

I want to use this Issue to share some thoughts a few of us have had about the Route CRD and to provide a place for discussion. I don't expect anything concrete to come out of this Issue specifically, but wanted to get some of these thoughts written down somewhere -- especially as y'all begin exploring alternative Ingress/mesh implementations which might require changes to the Route CRD.

The Route CRD (sample Route CR) has several fields that are specific to Cloud Foundry -- especially the app portion of a destination:

...

spec:
  destinations:
  - app:
      guid: be261513-3ccd-4000-b9d8-0023bbb08fbf
      process:
        type: web

...

The spec.destinations[].selector field is what is ultimately used for configuring routing. The spec.destinations[].app field is used for providing a few helpful labels on the created Services and for setting headers that are used for app access logging.

This design has been useful since it closely mirrors the CF v3 Route API. Unfortunately, this design also tightly couples the Route with Cloud Foundry apps and makes it difficult to adapt to other (theoretical) types of routable workloads.

In addition to making it a more flexible interface, this has the added benefit of making the Route CR serve as an example for enhancements we might want to propose upstream.

For example, one day we may want to standardize on some of the next generation Kubernetes Service APIs -- perhaps even use the HTTPRoute/TCPRoute resources directly¹ -- and the CF Route CR could be something we use to drive out enhancements to that.

Or maybe this helps it becomes a more general ApplicationRoute that could be used in other PaaS-like contexts.

¹ - The HTTPRoute/TCPRoute may be too low-level and not something a component like the CF API would want to make directly. I'm mostly using these resources as a straw dog example for a standard we could work toward.

Proposal

App Guid / Process Type

Consider removing the required spec.destinations[].app attribute and replacing it with several optional attributes that more clearly describe the intent of the field.

• spec.destinations[].logging_id - Optional String identifier to use for aggregating logs associated with this route. In Cloud Foundry this would be used to tie access logs for a given route to the app they belong to. For Istio this would ultimately end up being a header that gets added to the request.
• spec.destinations[].headers - Optional map[string]string containing arbitrary headers to add/modify on the request.

Destination Guid

The spec.destinations[].guid attribute is a little strange. It is currently being relied on to give the child Services a unique name. A guid is suitable for this purpose, but it's a little strange that we are relying on the creator of the Route to know this and provide the guid. Perhaps we could rename this field to name (and in Cloud Foundry it's value could remain a guid) and the actual underlying child Services could be named <route-name>-<random-identifier>. Essentially the same as how a Deployment named cf-api-server would create a ReplicaSet named something like cf-api-server-c7d8b8f64.

• spec.destinations[].name - String containing the name of the destination

Domain

The spec.domain attribute is a little interesting. I'm not sure what the internal route UX will end up looking like, but this might be a field we could revisit as well. That said, it might be generic enough as is.

Metadata for Child Resources

The labels we were adding to the child Service created by each destination were more of a nice-to-have, but maybe there can be some form of template.metadata.labels/template.metadata.annotations that could propagate to child resources. I'm not going to include this on the proposed example since it's not strictly necessary, but still something to think about.

Example Route

With these changes a Route CR would look like:

apiVersion: networking.cloudfoundry.org/v1alpha1
kind: Route
metadata:
  name: 7390d59b-f5f1-4c3c-9cb6-c1e2c5c3cf84
  namespace: cf-workloads
  labels:
    app.kubernetes.io/component: cf-networking
    app.kubernetes.io/managed-by: cloudfoundry
    app.kubernetes.io/name: 7390d59b-f5f1-4c3c-9cb6-c1e2c5c3cf84
    app.kubernetes.io/part-of: cloudfoundry
    app.kubernetes.io/version: 0.0.0
    cloudfoundry.org/domain_guid: 23bb47a0-b042-4087-8e55-97ec4b69b43a
    cloudfoundry.org/org_guid: b7ab8526-b63b-4156-90b7-2cacfd686a8b
    cloudfoundry.org/route_guid: 7390d59b-f5f1-4c3c-9cb6-c1e2c5c3cf84
    cloudfoundry.org/space_guid: d4a93829-fed3-497a-bcba-00bb2d454681
spec:
  destinations:
    - name: 9363095c-6be5-4982-a7db-a493e74af2f4
      logging_id: be261513-3ccd-4000-b9d8-0023bbb08fbf
      headers:
        CF-App-Id: be261513-3ccd-4000-b9d8-0023bbb08fbf
        CF-App-Process-Type: web
        CF-Space-Id: d4a93829-fed3-497a-bcba-00bb2d454681
        CF-Organization-Id: b7ab8526-b63b-4156-90b7-2cacfd686a8b
      port: 8080
      selector:
        matchLabels:
          cloudfoundry.org/app_guid: be261513-3ccd-4000-b9d8-0023bbb08fbf
          cloudfoundry.org/process_type: web
  domain:
    internal: false
    name: apps.example.com
  host: catnip
  path: ''
  url: catnip.apps.example.com

Just wanted to start an open discussion here to throw some of these ideas out there. If this is something y'all might actually do I'd be happy to collaborate on the design via Google Doc, PR, or other means.

cc @zrob who has also 🤔 about this and experimented with the Route CR

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174718773

The labels on this github issue will be updated when the story is started.

[mike1808]

@tcdowney I thought Route CR is a part of Cloud Foundry and I don't understand why it should be less CloundFoundry-specific.

[kauana]

Hello @tcdowney thanks for such a beautifully written issue 🎉. Out of curiosity, does the Cloud Controller (CAPI) team have any plans to remove "cf" from their components (e.g cf-api-server, cf-api-worker, etc)?

[mike1808]

However, I agree about changing app objects to what you've suggested as you mentioned we are not using it for app matching.

[tcdowney]

@tcdowney I thought Route CR is a part of Cloud Foundry and I don't understand why it should be less CloundFoundry-specific.

This is definitely a fair point. There's some value in keeping them similar. It makes the code a little easier to reason about and it provides a familiar experience to folks who have been operating Cloud Foundry for a while. But I think the flexibility provided by generalizing the CRD outweighs these benefits.

The close coupling between the Route CR and the CF v3 Route data model limit what consumers are able to do with the CRD. That's fine for the CF API today since it matches perfectly with its database schema, but for future iterations of CF or for folks that are just experimenting with the Route CRD it is pretty unfriendly.

What if I'd like to use some other control plane instead of the CF API (maybe something lighter-weight)? I will have to know what destinations[].app.guid and destinations[].app.process.type refers to and know that I need to just make stuff up here. Similarly for destinations[].guid the onus is on me to come up with a unique identifier -- and why does it need to be a guid? Things like that just make it a bit strange of an interface.

By making the Route CRD more generic in these aspects it should result in a more flexible extension point.

Out of curiosity, does the Cloud Controller (CAPI) team have any plans to remove "cf" from their components (e.g cf-api-server, cf-api-worker, etc)?

No plans to, and I don't think those should be renamed. This isn't a desire to expunge "Cloud Foundry" specifically, but rather to identify extension points that don't have to be Cloud Foundry-specific. The CF API (and friends) enable the traditional CF UX and I think they have to be pretty coupled to CF out of necessity.

The Route CRD though is an inner abstraction that could enable other workflows alongside CF and this proposal would help that.

---

Apart from all that, I think it's cool how some of the Contour folks have been able to use their HTTPProxy CRD to experiment with new ingress features and propose them to the upstream Ingress v2 / next generation K8s Services APIs work. The Route CRD could be an opportunity to help bring Cloud Foundry concepts/features to the broader Kubernetes community. Generalizing the CRD by removing a few CF-isms will make it easier to understand and digest.

[cwlbraa]

Route CR is a part of Cloud Foundry

remove "cf"

a desire to expunge "Cloud Foundry"

I hope I'm not derailing, but I think I'm detecting an opportunity for better nomenclature here. In this thread people are using "CF" to refer both to our broad organization (the thing the Route CR is a part of) AND the "CF API" (the thing that the Route CR ought nought to be specific to. Sometimes it's used twice in the same sentence but could be read to refer to 2 different things. If we draw the distinction between these 2 things, I think maybe we'd have an easier time communicating about them.

Out of curiosity, does the Cloud Controller (CAPI) team have any plans to remove "cf-[API]" from their components (e.g cf-api-server, cf-api-worker, etc)?

identify extension points that don't have to be Cloud Foundry [API]-specific

coupled to [the] CF [API] out of necessity.

I think the analogous thing here is that there's a tiny bit of code starting to form in cf-api-controllers that we might decouple from the API. Most of that component is coupled tightly to the CF API, like Tim says, but there's a TINY bit of glue that watches for kpack stack updates and applies their resulting image changes to Eirini's StatefulSets - an operation that could be useful 100% independent of the involvement of the CF API.

One day we may want to extract that thing to be part of an cf-app-controllers that we might deploy without a CF API or the rest of cf-api-controllers.

Tim's suggested changes to Route echo this intent: in the Route CRD, what could we decouple from the specifics of the CF API?

[rosenhouse]

First off, thanks for opening this, I really like this direction.

We haven't implemented TCP Routing in cf-for-k8s yet, but when we do, the ability to ask for "random port" would be necessary, and a meaningful enhancement atop anything I see in the rest of the kubernetes ecosystem.

there's good tenancy stuff (e.g. route ownership based on domains, first-come-first serve deconfliction of route names) that is well done in CAPI but could be extended into k8s, if we had something like CF Space being 1:1 with K8s Namespace.