LoadBalancer setup for ports 9090, 9093, and 3000?

[pyrodex]

I have a k8s cluster deployed with Canal and MetalLB working without issues when I publish other services via LoadBalancer (e.g. MySQL). However when I try with the vanilla setup of kube-prometheus using the example.jsonnet file I can't get the services to respond.

Services before LoadBalancer:

$ kubectl get svc -n monitoring -o wide
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE     SELECTOR
alertmanager-main       ClusterIP   10.98.207.250    <none>        9093/TCP,8080/TCP            3m10s   app.kubernetes.io/component=alert-router,app.kubernetes.io/instance=main,app.kubernetes.io/name=alertmanager,app.kubernetes.io/part-of=kube-prometheus
alertmanager-operated   ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   3m9s    app.kubernetes.io/name=alertmanager
blackbox-exporter       ClusterIP   10.102.58.194    <none>        9115/TCP,19115/TCP           3m10s   app.kubernetes.io/component=exporter,app.kubernetes.io/name=blackbox-exporter,app.kubernetes.io/part-of=kube-prometheus
grafana                 ClusterIP   10.104.109.137   <none>        3000/TCP                     3m6s    app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus
kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP            3m6s    app.kubernetes.io/component=exporter,app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/part-of=kube-prometheus
node-exporter           ClusterIP   None             <none>        9100/TCP                     3m5s    app.kubernetes.io/component=exporter,app.kubernetes.io/name=node-exporter,app.kubernetes.io/part-of=kube-prometheus
prometheus-adapter      ClusterIP   10.98.164.138    <none>        443/TCP                      3m2s    app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus
prometheus-k8s          ClusterIP   10.102.10.142    <none>        9090/TCP,8080/TCP            3m      app.kubernetes.io/component=prometheus,app.kubernetes.io/instance=k8s,app.kubernetes.io/name=prometheus,app.kubernetes.io/part-of=kube-prometheus
prometheus-operated     ClusterIP   None             <none>        9090/TCP                     2m58s   app.kubernetes.io/name=prometheus
prometheus-operator     ClusterIP   None             <none>        8443/TCP                     3m13s   app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus
$

Services after LoadBalancer:

$ kubectl get svc -n monitoring -o wide
NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE     SELECTOR
alertmanager-main       ClusterIP      10.98.207.250    <none>           9093/TCP,8080/TCP            3m54s   app.kubernetes.io/component=alert-router,app.kubernetes.io/instance=main,app.kubernetes.io/name=alertmanager,app.kubernetes.io/part-of=kube-prometheus
alertmanager-operated   ClusterIP      None             <none>           9093/TCP,9094/TCP,9094/UDP   3m53s   app.kubernetes.io/name=alertmanager
alertmanager-ui         LoadBalancer   10.106.135.110   192.168.15.122   9093:31126/TCP               6s      app.kubernetes.io/component=alert-router,app.kubernetes.io/instance=main,app.kubernetes.io/name=alertmanager,app.kubernetes.io/part-of=kube-prometheus
blackbox-exporter       ClusterIP      10.102.58.194    <none>           9115/TCP,19115/TCP           3m54s   app.kubernetes.io/component=exporter,app.kubernetes.io/name=blackbox-exporter,app.kubernetes.io/part-of=kube-prometheus
grafana                 ClusterIP      10.104.109.137   <none>           3000/TCP                     3m50s   app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus
grafana-ui              LoadBalancer   10.96.186.72     192.168.15.122   3000:31902/TCP               5s      app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus
kube-state-metrics      ClusterIP      None             <none>           8443/TCP,9443/TCP            3m50s   app.kubernetes.io/component=exporter,app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/part-of=kube-prometheus
node-exporter           ClusterIP      None             <none>           9100/TCP                     3m49s   app.kubernetes.io/component=exporter,app.kubernetes.io/name=node-exporter,app.kubernetes.io/part-of=kube-prometheus
prometheus-adapter      ClusterIP      10.98.164.138    <none>           443/TCP                      3m46s   app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus
prometheus-k8s          ClusterIP      10.102.10.142    <none>           9090/TCP,8080/TCP            3m44s   app.kubernetes.io/component=prometheus,app.kubernetes.io/instance=k8s,app.kubernetes.io/name=prometheus,app.kubernetes.io/part-of=kube-prometheus
prometheus-operated     ClusterIP      None             <none>           9090/TCP                     3m42s   app.kubernetes.io/name=prometheus
prometheus-operator     ClusterIP      None             <none>           8443/TCP                     3m57s   app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus
prometheus-ui           LoadBalancer   10.99.210.214    192.168.15.122   9090:30034/TCP               6s      app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus
$

CURL showing inaccessible access to the ports on the assigned internal IP.

$ for i in 3000 9090 9093; do echo $i; nc -v -z -n -w 3 192.168.15.122 $i; done
3000
(UNKNOWN) [192.168.15.122] 3000 (?) : Connection timed out
9090
(UNKNOWN) [192.168.15.122] 9090 (?) : Connection timed out
9093
(UNKNOWN) [192.168.15.122] 9093 (?) : Connection timed out
$

So what am I doing wrong here or is there some type of configuration I am missing? I am learning k8s after having experience in Docker and have containerized my current prometheus and various exporters.

Thanks!

[pyrodex]

Figured it out, was networkpolicy....

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: grafana-allow-external
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: grafana
      app.kubernetes.io/name: grafana
      app.kubernetes.io/part-of: kube-prometheus
  ingress:
  - ports:
    - port: 3000
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: alertmanager-allow-external
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: alert-router
      app.kubernetes.io/name: alertmanager
      app.kubernetes.io/part-of: kube-prometheus
  ingress:
  - ports:
    - port: 9093
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: prometheus-allow-external
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
  ingress:
  - ports:
    - port: 9090

[marcusnh]

Thank you for this. it worked for me as well, but i don't understand why we should need this. Tought this was taken care of in the orignal NetworkPolicy for the altertmanager, prometheus and grafana. Any idea why?