cert-manager not able to finish challenge with nginx ingress controller, calico

[grohmio]

I used nginx ingress controller, calico network plugin and activated cert manager by default. If i add an issuer + a certificate to the cluster, cert-manager is not able to fulfill the challenge.

kubectl describe challenge grohmio-app-....

Output:

...
Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://your-domain.de/.well-known/acme-challenge/MbDgz7cbOGdrVx5pNkHLxnxZlHTDJ1ipgBPE-QmEAyE': Get "http://your-domain.de/.well-known/acme-challenge/MbDgz7cbOGdrVx5pNkHLxnxZlHTDJ1ipgBPE-QmEAyE": EOF
  State:       pending
Events:
  Type    Reason     Age    From                     Message
  ----    ------     ----   ----                     -------
  Normal  Started    7m20s  cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  7m19s  cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

The acme challenge link is accessible from outside the cluster but not from inside the cluster.
Creating a curl pod and running curl
kubectl run mycurlpod --image=curlimages/curl -i --tty -- sh
results in:

/ $ curl -v http://your-domain.de/.well-known/acme-challenge/MbDgz7cbOGdrVx5pNkHLxnxZlHTDJ1ipgBPE-QmEAyE
*   Trying 142.132.246.185:80...
* Connected to your-domain.de (142.132.246.185) port 80 (#0)
> GET /.well-known/acme-challenge/MbDgz7cbOGdrVx5pNkHLxnxZlHTDJ1ipgBPE-QmEAyE HTTP/1.1
> Host: your-domain.de
> User-Agent: curl/7.87.0-DEV
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

my kube.tf:

locals {
  hcloud_token = var.my_token
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = local.hcloud_token
  source = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key = file("~/.ssh/hcloud_k3s_cluster.pub")
  ssh_private_key = file("~/.ssh/hcloud_k3s_cluster")
  network_region = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane-nbg1",
      server_type = "cpx11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "agent-large",
      server_type = "cpx11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "storage",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [
        "node.kubernetes.io/server-usage=storage"
      ],
      taints      = [],
      count       = 1
    }
  ]
  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"
  ingress_controller = "nginx"
  ingress_replica_count = 1
  cni_plugin = "calico"
}

provider "hcloud" {
  token = local.hcloud_token
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.35.2"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

my issuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cluster-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ....
    privateKeySecretRef:
      name: letsencrypt-cluster-issuer-key
    solvers:
      - http01:
          ingress:
            class: nginx

my certificate:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: grohmio-app
  namespace: default
spec:
  dnsNames:
    - your-domain.de
  secretName: grohmio-tls-secret
  issuerRef:
    name: letsencrypt-cluster-issuer
    kind: ClusterIssuer

Any idea how to fix that?

[grohmio]

I finally found a solution, it is similar to this one #413 and here hetznercloud/hcloud-cloud-controller-manager#58 (comment)

steps to fix the issue:

• get the ingress controller definition

kubectl get svc nginx-ingress-nginx-controller -n nginx -o yaml > ingress_controller_nginx.yaml

• add annotation load-balancer.hetzner.cloud/hostname: your-domain.de (replace your-domain.de with your domain)

• apply changes

kubectl apply -f ingress_controller_nginx.yaml -n nginx

after these steps, you can apply an issuer and a certificate and cert-manager passes successful.

[primeXchange]

Thank you! This has been driving me nuts as I could not figure it out.