Creating scrape jobs vs. adding annotations to Kubernetes

[christian-kreuzberger-dtx]

Right now prometheus-service works by creating scrape jobs for every stage and every possible candidate a service could have, e.g.:

prometheus-service/eventhandling/configureEvent.go

Lines 250 to 259 in 44488ab

	for _, stage := range shipyard.Spec.Stages {
	var scrapeConfig *prometheusconfig.ScrapeConfig
	// (a) if a scrape config with the same name is available, update that one
	// <service>-primary.<project>-<stage>
	createScrapeJobConfig(scrapeConfig, config, eventData.Project, stage.Name, eventData.Service, false, true)
	// <service>-canary.<project>-<stage>
	createScrapeJobConfig(scrapeConfig, config, eventData.Project, stage.Name, eventData.Service, true, false)
	// <service>.<project>-<stage>
	createScrapeJobConfig(scrapeConfig, config, eventData.Project, stage.Name, eventData.Service, false, false)

I think this behaviour could be improved such that every service that should be monitored needs to have the correct annotations set, e.g.:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: /metrics
    prometheus.io/port: "8080"

This would allow us to remove the code for creating scrape jobs in prometheus-service.

I'm not sure which behaviour would be considered best practice, so please let me know via answering or via voting 👍 if you favor this change, or 👎 if you are against this change.