Support Bosh trusted certs

[haydonryan]

This has come up a lot of times for our customers. When using tooling with concourse (eg platform automation) for certain kinds of Companies they must do everything securely, including checking certificates are valid. Most of these larger customer are using bosh deployed concourse.

Currently concourse does not automatically include the bosh trusted certs.

I would love to see a feature (behind a feature flag to start with but eventually be the default:
On concourse worker task / job creation

• copy either contents of the VM's /etc/ssl/certs/ directory, or only /etc/ssl/certs/bosh-trusted-cert-*.pem into the container.
• and rebuild the ca bundle.

This would allow users to define once their company internal CAs in bosh (or opsmanager) and have it flow through. In my experience customesrs want to blanket trust those CAs.

Currently the workarounds I've heard about are:

• Building a custom image for every container with the certs pre-injected. (This requires management, and a pipeline to get upstream containers and add on their own certs.)
• Manually downloading and injecting the certs as part of a script.

[taylorsilva]

I am hesitant to add a feature that's so platform-specific or at least presents as being platform-specific. Trusting custom CA cert's seems like a general problem that all large/medium enterprises would have, and plenty of those folks also don't use bosh. There is also a significant number of people running Concourse on k8s with the helm chart. They probably have custom CA cert issues too.

As long as it's not platform specific, I'm open to PR's implementing a feature that solves this problem.

I personally wouldn't be in a rush to implement this myself. The workarounds are likely good enough for most people and I'm in no need for such a solution.

[haydonryan]

To be clear, I wasn't thinking the solution would be specific to bosh deployed concourse, even though I specifically mentioned bosh trusted certs. It's really a feature request to pull from the host's (eg VM, bare metal, etc) trust store into the container's.

[kcbimonte]

This is definitely something we struggle with as well. I had to create small task at the beginning of each job to compile all my trusted certs so I could install them in every subsequent job.

Right now, it just works for get, put, and check steps (which is called out somewhere Resource Certs) but I'd love for it to just work in the tasks as well.

[haydonryan]

I spent a LOT of time yesterday looking into past issues on this topic (across garden, cf, concourse etc.)

Specifically:
#1027 <- I didn't recall even posting in this issue but apparently I did..
cloudfoundry/guardian#83
#1938
#1856

The reason for this isn't already in concourse is that that concourse uses a "bring your own image" approach to containers. To insert certs into a container won't work as unlike /etc/resolv.conf (which is a defacto standard). Certificate trust stores are somewhat different based on the OS. It also doesn't address the issue if someone built a docker image that had custom certs in it. Forcing host certs to overwrite container certs at the concourse level would clobber those existing certs, potentially breaking the task. What rules do we put in?

I agree with Taylor's point about trying to be platform agnostic - it would break the abstraction of BYO task images. Having concourse detect and then manage the injection of certificates into a container is also fraught with problems, lots of edge cases "concourse keeps screwing up my certs" - that kind of thing.

What I think might be a good middle ground is to mount the host certificates to a non standard directory (/var/concourse/host-certs/, or /mnt/host-certs as some ideas?) and let the job / task author / end user then pull in. That way I could (as an example), copy bosh-trusted-ca-*.pem to the cert store directory and run the command to compile them into the library. It allows the user to have full control, but also provides the certs for use so they don't have to be downloaded / built into the image.

Ultimately Concourse shouldn't have to know about all the flavors of OS that could be run on it, but looking now at task configs we already have to specify the platform, so maybe os isn't that bad. So in my opinion it comes down to - do we:

1. Keep it as it is.
2. Take an intermediate approach eg bind host certs to a separate location (but not over /etc/ssl/certs/, and leave it to the task to copy and manage certs.
3. In addition to 2, provide a mechanism inside concourse to copy and rebuild the trust store that's user controlled.

We would also need to add a manifest entry to specify where in the host OS the cert files are stored, otherwise it would be OS specific for the host.

For 3 After a lot of thinking, I think adding a field to the task-config is the right way forward. It would be less verbose to add to the job but you want to support the running of different tasks with different OSes, and still having the correct cert injection.

eg from platform automation:

---
platform: linux

### proposal:
host:
  certs: 
    overwrite-container-certs: true        #default is false
    container-certs-directory:   "/etc/ssl/certs"
    container-certs-update-command:  "update ca-certificates"
    certs-to-copy:
    - "bosh-trusted-cert*.pem"
    - "company.xyz-nonprod.pem"
###

inputs:
- name: platform-automation-tasks
- name: config # contains the director configuration file
- name: env # contains the env file with target OpsMan Information
- name: vars # variable files to be made available
  optional: true
- name: secrets
   optional: true
- name: ops-files # operations files to custom configure the product
  optional: true

params:
  VARS_FILES:
  OPS_FILES:
  ENV_FILE: env.yml

  DIRECTOR_CONFIG_FILE: director.yml
 
run:
  path: platform-automation-tasks/tasks/configure-director.sh

In the idea above - only certs that match the globs would be copied from the mount point to the container's (Ubuntu) trust store (/etc/ssl/certs/), and then run the appropriate command (update-ca-certificates).

This would work for other OSes too - the task author would just need to change parameters.

I would love to have it somewhere else to be less repetitive, eg the resource, so I'm very open to other ideas too....

[taylorsilva]

As an aside, reading

What I think might be a good middle ground is to mount the host certificates to a non standard directory

Made me think of https://xkcd.com/927/ 😅

---

Thanks for digging into the history of this issue and sharing a brief summary.

As noted in your summary, a lot of tools and OS's do different things when it comes to loading certs (Go has this handy list). The idea of loading certs in some Concourse-specific directory (/var/concourse/host-certs) sounds fine. Users would get access to the host certs and can configure their tools to use the certs at that location. Openssl can be configured with SSL_CERT_DIR and most other tools usually have some way of telling the app where the CA certs are stored. GitHub Actions takes a similar approach it seems, though docs are a bit sparse on details (docs). I'd be comfortable with a setup like this because it doesn't change too many things and isn't likely to break anyone's existing pipelines and tasks.

Regarding your proposal, the host.cert field you suggest adding to the task config feels like a lot of settings to throw at a user. The proposal feels like you're trying to solve various problems at once with these new fields. I'd prefer taking a more incremental approach here to avoid overwhelming ourselves.