Prometheus agent not discovering servicemonitors

[ajayanaj]

Hi, I'm experimenting prometheus agent to reduce load on the prometheus. But one bug I noticed is it's not automatically picking up the servicemonitors, and it requires a rollout restart to pick up the servicemonitors.
I have installed kube-prometheus-stack using helm chart and prometheus agent via CRD file, like below.
apiVersion: monitoring.coreos.com/v1alpha1
kind: PrometheusAgent
metadata:
name: kube-prometheus-stack-prometheus-agent-1
namespace: monitoring
spec:
externalLabels:
cluster: cluster-name
image: quay.io/prometheus/prometheus:v2.43.1
containers:
- name: prometheus
args:
- '--enable-feature=agent'
- '--config.file=/etc/prometheus/config_out/prometheus.env.yaml'
- '--storage.agent.path=/prometheus'
- '--storage.agent.retention.max-time=15m'
resources:
requests:
cpu: 500m
memory: 500M
limits:
cpu: 1
memory: 1Gi
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
matchLabels:
release: kube-prometheus-stack-agent1
shards: 1

I'm not sure why prometheusagent is not discovering the servicemonitors, am I missing any configurations?
Please help me here.

[ajayanaj]

The issue was with this error.
msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 403 Forbidden; have you set --web.enable-lifecycle Prometheus flag?"
I was able to fix the same by passing "--web.enable-lifecycle" argument to prometheus agent.