K3D not ready for cert-manager

[sdc395]

Hi all

I'm still working with K3D for integration testing (see my discussion re. cluster start-up). Having been forced (perhaps it was just a matter of time) by another requirement to introduce cert-manager to my tests, I've run into what seems to be a timing issue.

My tests are initiated using a script. If I perform the steps manually, and wait for the initial cluster network activity to die down, the cert-manager resource definitions are accepted by the cluster. But running my script results in...

INFO[0000] Prep: Network                                
INFO[0000] Re-using existing network 'k3d-test-network' (713a882bc31f6f3a08f073706ad795c539d88bc2ed6b951a40b3ad0437086e92) 
INFO[0000] Created volume 'k3d-tests-54672-images' 
INFO[0000] Container 'k3d-image-registry' is already connected to 'k3d-test-network' 
INFO[0001] Creating node 'k3d-tests-54672-server-0' 
INFO[0001] Creating LoadBalancer 'k3d-tests-54672-serverlb' 
INFO[0001] Starting cluster 'tests-54672'      
INFO[0001] Starting servers...                          
INFO[0001] Starting Node 'k3d-tests-54672-server-0' 
INFO[0005] Starting agents...                           
INFO[0005] Starting helpers...                          
INFO[0005] Starting Node 'k3d-tests-54672-serverlb' 
INFO[0006] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
INFO[0009] Successfully added host record to /etc/hosts in 2/2 nodes and to the CoreDNS ConfigMap 
INFO[0010] Cluster 'tests-54672' created successfully! 
INFO[0010] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false 
INFO[0010] You can now use it like this:                
kubectl config use-context k3d-tests-54672
kubectl cluster-info
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
namespace/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
configmap/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io condition met
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io condition met
Error from server (InternalError): error when creating "/home/node/tests/base/cert-manager-config.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.43.78.148:443: connect: connection refused
Error from server (InternalError): error when creating "/home/node/tests/base/cert-manager-config.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.43.78.148:443: connect: connection refused
Error from server (InternalError): error when creating "/home/node/tests/base/cert-manager-config.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.43.78.148:443: connect: connection refused
Error from server (InternalError): error when creating "/home/node/tests/base/cert-manager-config.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.43.78.148:443: connect: connection refused
Error from server (InternalError): error when creating "/home/node/tests/base/cert-manager-config.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.43.78.148:443: connect: connection refused
Error from server (InternalError): error when creating "/home/node/tests/base/cert-manager-config.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.43.78.148:443: connect: connection refused

Can anyone tell me what it is I need to be waiting for before trying to create cert-manager resources, and how I go about waiting?

Thanks
Simon

[iwilltry42]

Hi @sdc395 , thanks for starting this discussion!
Can you maybe share your script or at least the mentioned steps here, so it's easier to reproduce?
I guess the cert-manager folks know best what are the pre-requisites for this to work, but we can try to help 👍

[sdc395]

Hi @iwilltry42

I think you're right that this is a cert-manager issue exacerbated by K3D's long start-up time. The solution seems to be waiting for all cert-manager deployments to be "Available".

Here is a simplified version of my script. There's not much to it.

#! /bin/bash
set -euo pipefail

SCRIPT_DIR=$(cd -P -- "$(dirname -- "$0")" && pwd -P)

function get_port
{
    local -n __OUT__="$1"

    LOW_BOUND=49152
    RANGE=16384

    while true; do
        CANDIDATE=$[$LOW_BOUND + ($RANDOM % $RANGE)]
        set +e
        (echo "" >/dev/tcp/127.0.0.1/${CANDIDATE}) >/dev/null 2>&1
        RESULT=$?
        set -e
        if [ $RESULT -ne 0 ]; then
            __OUT__=$CANDIDATE
            break
        fi
    done
}

get_port API_PORT
API_HOSTNAME=localhost
CLUSTER_NAME="tests-$API_PORT"

# Create a cluster for testing.
k3d cluster create "$CLUSTER_NAME" --api-port "$API_HOSTNAME:$API_PORT"

# Apply cert-manager and wait.
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml
kubectl wait --timeout=300s --for=condition=Available --all -n cert-manager deployments

# Apply manifests.
kubectl apply -f "$SCRIPT_DIR/cert-manager-config.yaml"

I don't think this problem is unique to K3D. Hopefully the kubectl wait line will prove to be a reliable solution.

Thanks

[iwilltry42]

I guess I should've read the full log lines there..
I didn't know that you were already deploying cert-manager resources right after the cert-manager deployment, but it makes sense that you'll have to wait for cert-manager to be fully up and running first, as it needs the cert-manager itself to validate your manifests.

So I believe you're right that it's not a problem with k3d itself but just with general startup time of cert-manager in conjunction with really quick deployment of resources requiring it.
I guess the kubectl wait will do the trick for your use-case 👍

Thanks for sharing 👍