external endpoint disappear after view hours

[m6f13]

Hello,

i have deployed prometheus on k8s and added an external endpoint.
The issue is: that after view hours, the endpoint disappears in prometheus and when i check the targets, it is listed with the address but 0/0, so the config is missing.
when i then apply the files again, only on the endpoint file I can see: "configured" as response and so the endpoint is working again... until view hours, where it gets removed again.

it would be great if someone could give me a hint, what is wrong or missing in the config.

here are the files for my endpoint config:

secret-file.yaml
###################

apiVersion: v1
data:
  password: xxxxxxxxxxx
  username: xxxxxxxxxxx
kind: Secret
metadata:
  name: basic-auth-node-exporter
  namespace: monitoring-dev
type: Opaque

###################

endpoint.yaml
###################

apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app: node-exporter-vm-agent
    jobLabel: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
subsets:
- addresses:
  - ip: 192.168.1.71
    nodeName: 192.168.1.71
  ports:
  - name: metrics
    port: 9100
    protocol: TCP

###################

service-monitor.yaml
###################

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: prom00
    meta.helm.sh/release-namespace: monitoring-dev
  labels:
    app: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
spec:
  endpoints:
  - basicAuth:
      password:
        key: password
        name: basic-auth-node-exporter
      username:
        key: username
        name: basic-auth-node-exporter
    port: metrics
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: jobLabel
  selector:
    matchLabels:
      app: node-exporter-vm-agent
      release: prom00

###################

service.yaml
###################

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: prom00
    meta.helm.sh/release-namespace: monitoring-dev
    prometheus.io/scrape: 'true'
  labels:
    app: node-exporter-vm-agent
    jobLabel: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
spec:
  externalName: 192.168.1.71
  ports:
  - name: metrics
    port: 9100
    protocol: TCP
    targetPort: 9100
  selector:
    app: vm-agent
    release: prom00
  type: ExternalName

###################

[paulfantom]

Seems like an issue with kubernetes or your deployment. None of the components provided by kube-prometheus is managing Endpoint objects and thus it couldn't remove it. It looks to me like an external factor is removing the object from a cluster.

[m6f13]

thanks for your feedback. do i have a chance to log it?

[ghost]

some issue here, did not find log from prometheus