Fix BYOC TLS

[monrax]

This closes #34

This PR attempts to address the following issue:

• The cert mounted at GRAYLOG_HTTP_TLS_CERT_FILE is presented to both clients outside the cluster for external access, as well as internal clients for traffic between nodes. This means a cert with only a CN field that does not match any of the node names will prevent inter-node secure communication.
• In addition, "None of the TrustManagers trust this certificate chain" pops up when using a self-signed cert.
• Thus, when using a self-signed cert, both:
  • Graylog nodes acting as servers does not advertise a certificate with the correct node names.
  • Graylog nodes acting as clients do not even trust the certificate that is presented to them.
• This causes inputs to not work when BYOC TLS is enabled.

The fix:

• The cert to be mounted should have both a CN field, and a SAN field with at least two entries: the CN again, but also the internal node names as a wild card like so: DNS:*.graylog-svc.graylog.svc.cluster.local (adjusted to the current service name and namespace)
• The mounted cert, and the root CA included inside it if that's the case, should be added to the Java Key Store as indicated here (in the "Java Key Store" section).

Implementation details:

• values.yaml: added updateKeyStore (which defaults to true) and keyStorePass (which defaults to Java's own default "changeit") to graylog.config.tls.byoc to control when to enable updating the Java Key Store.
• _helpers.tpl: the full Java options passed to the containers will also include "-Djavax.net.ssl.trustStore=/usr/share/graylog/data/cacerts/graylog.jks" (as well as the corresponding password option) whenever both graylog.config.tls.byoc.enabled and graylog.config.tls.byoc.updateKeyStore are set to true.
• init-script.sh: this is the main bit. In here we perform one check to see if the wildcard *.graylog-svc.graylog.svc.cluster.local is present as a SAN in the cert mounted using the provided TLS k8s secret when graylog.config.tls.byoc.enabled is true . If this is not the case the Graylog application pods won't start up until a valid certificate is provided (valid meaning the cert has the wildcard SAN field). Secondly, if graylog.config.tls.byoc.updateKeyStore is also true the certificate (and any root CA in it) will be added to the Java Key Store automatically.

The part I'm less comfortable with is the one where the application doesn't start up if a cert is given without the DNS:*.graylog-svc.graylog.svc.cluster.local SAN (which are going to be most certificates out there), even if the init container logs tell the user what's going on. I would rather have Graylog use two certs: one for inter-node communication (which we would generate internally automatically) and leave GRAYLOG_HTTP_TLS_CERT_FILE for external access, or something similar (perhaps, even allow an option to skip hostname verification altogether?). However, that would require internal changes to the application, and not an option right now.

Notes for Reviewers

• The commit history must be preserved - please use the rebase-merge or standard merge option instead of squash-merge
• Sync up with the author before merging

[monrax]

Turning into draft until feature/fix/patch has been documented.

[monrax]

Validation steps:

0. Checkout this branch:

git checkout fix/jks
git pull

1. Install chart:

helm install graylog ./graylog -n graylog --set provider="aws-managed-sc"

2. Set the service type to LoadBalancer:

helm upgrade graylog ./graylog -n graylog --set graylog.custom.service.type="LoadBalancer" --reuse-values

3. Get the service hostname/IP address and update the DNS entry corresponding to your domain:

SERVICE_NAME=graylog-svc; kubectl get svc $SERVICE_NAME -n graylog

4. If you don't already have a certificate, create one with the additional DNS:*.graylog-svc.graylog.svc.cluster.local SAN entry:

openssl req -x509 -sha256 -newkey rsa:2048 -days 365 -keyout server.key -out server.crt -nodes -subj "/CN=<your domain here>" -addext "subjectAltName=DNS:<your domain here>,DNS:*.graylog-svc.graylog.svc.cluster.local"

5. Create a corresponding k8s secret

kubectl create secret tls my-cert --cert=public.crt --key=private.key -n graylog

6. Enable TLS

helm upgrade graylog ./graylog -n graylog --reuse-values --set graylog.config.tls.enabled=true --set  graylog.config.tls.secretName="my-cert" --set graylog.config.tls.updateKeyStore=true

7. Check that you can configure and run inputs:

• Wait for all pods to restart
• Go to http://<your domain>:9000/ and verify the certificate corresponds to the one configured above
• Log in with your credentials, set up an input, and start the input
• Check the container logs to make sure nothing unexpected shows up

Done!

[williamtrelawny]

It works!

The only minor thing I'd add is in NOTES.txt - when a user has graylog.config.tls.enabled=true, change the URL on line 28 to say https instead of the default http.

If I was better at Go, I'd go ahead and do this for you, I'm sorry 😢

[monrax]

It works!

The only minor thing I'd add is in NOTES.txt - when a user has graylog.config.tls.enabled=true, change the URL on line 28 to say https instead of the default http.

Patched! Please re-validate when convenient.

If I was better at Go, I'd go ahead and do this for you, I'm sorry 😢

No worries -- reviewing that the code works is more important than the actual code 💪

[williamtrelawny]

LGTM! 🚀 Awesome work @monrax !!

[williamtrelawny]

I already resolved the merge conflict but it's still telling me there's another conflict, so I'm going to leave it to you @monrax to resolve, sorry... Hope I didn't mess anything up when resolving the first conflict.