502 bad gateway when exposing Grafana through ingress on 0.11

[PaulSonOfLars]

Hi all!

I've recently upgraded from 0.10 to 0.11.

On 0.10, I had created an ingress to make Grafana externally available. This worked great, and I had no issues.
However, on 0.11, this ingress fails with a 502 bad gateway. Some access logs show in the traefik logs, but no reason for the 502, and no access logs in grafana at all - the 502 simply seems to be coming out of thin air.

If I change the IngressRoute to use a different service, which still targets the same pod, 502.
If I change the labelSelector on the service to target different labels (but still the same pods), 502.
BUT, if I change the service to target different pods, with different labels, 502 vanishes.

Whatever I do, each service seems to be finding the pods fine; they show up in the Endpoints as expected.

Some debugging later, I have discovered that the 502 will only happen on services which target pods that have all three of the following labels:

 app.kubernetes.io/component: grafana
 app.kubernetes.io/name: grafana
 app.kubernetes.io/part-of: kube-prometheus

Editing just one of those labels will remove the 502, and adding those three labels to any test pods will also immediately cause a 502. This makes me think that I am either misunderstanding kube-prometheus in some way, or missing a piece of knowledge in kubernetes - both entirely possible, but I'd like to have some external opinions.

So - is there something I'm missing?
Thanks in advance!

Versions:

k3s 1.24.4+k3s1
kube-prometheus release-0.11

Repro instructions

To ensure I wasn't going crazy, I've recreated a basic k3s cluster to test these changes in isolation.

I ran:

kubectl apply --server-side --force-conflicts -k 'github.com/prometheus-operator/kube-prometheus?ref=release-0.11'
kubectl apply -f ingress.yaml # See below for ingress file

I've tried this with both 0.10, and 0.11. It works on 0.10, not on 0.11.

ingress.yaml

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: https-redirect
  namespace: monitoring
spec:
  redirectScheme:
    scheme: https
    permanent: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroute-grafana
  namespace: monitoring

spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`grafana.example.org`)
      kind: Rule
      services:
        - kind: Service
          name: grafana
          port: http

# Accepted that TLS would be bad for this, and didnt want to install certmanager for a test.
#  tls:
#    secretName: grafana-prod-tls
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroute-grafana-http
  namespace: monitoring

spec:
  entryPoints:
    - web
  routes:
    - match: Host(`grafana.example.org`)
      kind: Rule
      services:
        - kind: Service
          name: grafana
          port: http
      middlewares:
        - name: https-redirect

[PaulSonOfLars]

Ah, I figured it out.

kube-prometheus sets up a NetworkPolicy which stops the ingress from working.
Editing the policy to remove this restriction fixed my issue.