[discourse] Prometheus as source for analytics

[markphelps]

Originally posted by member on Discourse

Hello team!

Now I’m trying to connect a new feature, but I ran into a problem. With the Prometheus version 3.0.1 (latest) everything works fine, but with version 2.55.1 the graph is not displayed in the UI.

Screenshot 2024-12-01 1537521420×580 11 KB

For the test, I ran docker-compose from the example with image: prom/prometheus:v2.55.1 and got the same effect. Can I somehow fix this myself or will only update to Prometheus 3.0.1?

[dosubot[bot]]

There aren't any Flipt-specific configurations or code that change UI behavior based on the Prometheus version—Flipt uses the Prometheus Go client library and interacts with Prometheus via its API in a version-agnostic way. The example Docker Compose setup uses the latest Prometheus image, which works, but switching to v2.55.1 causes the graph to disappear, as you've seen.

This suggests the issue is likely due to a change or incompatibility in Prometheus itself, not Flipt. It's possible that Prometheus 2.55.1 responds differently to certain queries or has a bug that was fixed in 3.0.1. The Flipt backend expects a specific response format (a matrix type) and handles errors if the response is unexpected, which could explain the missing graph if 2.55.1 returns something different or incomplete for the same query (source).

If you need this working with 2.55.1, you could try running the same Prometheus query manually in the Prometheus UI for both versions and compare the results. If 2.55.1 returns a different structure or no data, that's likely the root cause. Otherwise, upgrading to 3.0.1 is the most straightforward fix, since that's the version Flipt's examples and tests are known to work with.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[markphelps]

Reply from mark on Discourse:

Hey!

We recently merged a fix that should work with older versions of prometheus. We haven’t yet released it as a new version of Flipt, but will create a new release this coming week.

In the mean time, can you try to use the nightly version of Flipt? I updated the docker-compose file to this:

version: "3"

services:
  prometheus:
    image: prom/prometheus:v2.55.1
    ports:
      - "9090:9090"
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
    networks:
      - flipt_network

  flipt:
    image: flipt/flipt:nightly
    command: ["./flipt", "--force-migrate"]
    depends_on:
      - prometheus
    ports:
      - "8080:8080"
    networks:
      - flipt_network
    environment:
      - FLIPT_LOG_LEVEL=debug
      - FLIPT_CACHE_ENABLED=true
      - FLIPT_CACHE_BACKEND=memory
      - FLIPT_CACHE_TTL=1m
      - FLIPT_META_TELEMETRY_ENABLED=false
      
  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat < /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Prometheus
          type: prometheus
          access: proxy 
          orgId: 1
          url: http://prometheus:9090
          uid: prometheus
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - flipt_network

networks:
  flipt_network:

And it seemed to work using prometheus v2.55.1 and flipt nightly

CleanShot 2024-12-01 at 11.56.541621×1304 127 KB

[markphelps]

Reply from member on Discourse:

Hey @mark!

Great, I’ll be waiting for the new release.

Checked now with night build:

Version: v1.52.1
Commit: 284e5e6
Build Date: 2024-12-01T16:53:54Z
Go Version: go1.23.3
OS/Arch: linux/amd64

There is still no analytics in my case (the result is the same as in my screenshot above).

[markphelps]

Reply from erka on Discourse:

There is no problem with Prometheus v2.55.1. It’s a UI bug. As workaround switch Timezone UTC on/off in Settings. The chart will not be broken after that.

[markphelps]

Reply from member on Discourse:

I tried it and it worked with the https://github.com/flipt-io/flipt/blob/main/examples/analytics/prometheus/docker-compose.yml

image: prom/prometheus:v2.55.1,

Screenshot 2024-12-031412×789 23.3 KB

but it still doesn’t work for me.

Screenshot 2024-12-031438×745 13.9 KB

Maybe the problem is on my side, I’ll try to find it

P.S in Grafana everything is ok

[markphelps]

Reply from andi4000 on Discourse:

Thanks for this! I was scratching my head why is analytics on my v1.52.2 instance not working. Do you have ETA for release?

[markphelps]

Reply from mark on Discourse:

@andi4000 we just released v1.53.0 that should contain this fix

  <a href="https://github.com/flipt-io/flipt/releases/tag/v1.53.0" target="_blank" rel="noopener">GitHub</a>

Release v1.53.0 · flipt-io/flipt

Added

ui: new table view for flags and segments (#3664)

Changed

ui: stop using /meta/config endpoint (#3684)
ui: use black as black, white as white (#3687)
github auth: pass org and teams to met...

Please let us know if it resolves the issue!

[markphelps]

Reply from member on Discourse:

Thanks for the update.

For me it still doesn’t work and from this version the graph in Grafana v11.3.1 stopped displaying. In Prometheus itself the numerical metrics are displayed correctly.

[markphelps]

Reply from member on Discourse:

upd: In Grafana everything is ok, I just checked it again.

It is not displayed only in Flipt analytics.

[markphelps]

Reply from andi4000 on Discourse:

Thanks for the update! It works on docker compose (examples/analytics/prometheus/docker-compose.yml, edited prometheus to v2.55.1 and flipt to v1.53.1).

But for some reason it’s not yet working on our k8s cluster. I can see data in prometheus, but not at flipt and grafana (with this dashboard)

[markphelps]

Reply from andi4000 on Discourse:

I noticed this prometheus query here:

  <a href="https://github.com/flipt-io/flipt/blob/df4bd3894e214cc4741a6cbd06404c577b75055a/internal/server/analytics/prometheus/client.go#L47" target="_blank" rel="noopener nofollow ugc">github.com</a>

flipt-io/flipt/blob/df4bd3894e214cc4741a6cbd06404c577b75055a/internal/server/analytics/prometheus/client.go#L47

<pre class="onebox"><code class="lang-go">
  <ol class="start lines" start="37" style="counter-reset: li-counter 36 ;">
      <li>	})</li>
      <li>	if err != nil {</li>
      <li>		return nil, err</li>
      <li>	}</li>
      <li>	promClient := promapi.NewAPI(apiClient)</li>
      <li>	return &client{promClient: promClient, logger: logger}, nil</li>
      <li>}</li>
      <li></li>
      <li>func (c *client) GetFlagEvaluationsCount(ctx context.Context, req *panalytics.FlagEvaluationsCountRequest) ([]string, []float32, error) {</li>
      <li>	query := fmt.Sprintf(</li>
      <li class="selected">		`sum(increase(flipt_evaluations_requests_total{namespace="%s", flag="%s"}[%dm])) or vector(0)`,</li>
      <li>		req.NamespaceKey,</li>
      <li>		req.FlagKey,</li>
      <li>		req.StepMinutes,</li>
      <li>	)</li>
      <li>	r := promapi.Range{</li>
      <li>		Start: req.From.UTC(),</li>
      <li>		End:   req.To.Add(time.Duration(req.StepMinutes) * time.Minute).UTC(),</li>
      <li>		Step:  time.Duration(req.StepMinutes) * time.Minute,</li>
      <li>	}</li>
      <li>	data, warnings, err := c.promClient.QueryRange(ctx, query, r)</li>
  </ol>
</code></pre>

But when I check the data points in prometheus, it looks like this:

flipt_evaluations_requests_total{container="flipt", endpoint="http", exported_namespace="team01", flag="foo-my-flag", instance="10.x.x.x:8080", job="flipt", namespace="flipt", otel_scope_name="github.com/flipt-io/flipt", pod="flipt-XXXX-XXXX", service="flipt"}

If I understand this correctly, flipt retrieves zero data because the query is wrong. The namespace key refers to the k8s application namespace, not flipt namespace.

Whereas on local compose setup, the data point looks like this:

flipt_evaluations_requests_total{flag="foo", instance="flipt:8080", job="flipt-metrics", namespace="default", otel_scope_name="github.com/flipt-io/flipt"}

Which makes sense why it works on docker compose, but not in k8s.

Is there something I can do on my end to make this work? Sorry, I’m not very well versed in golang or prometheus setup.

[markphelps]

Reply from erka on Discourse:

It looks there is no easy way out… Per Prometheus docs

# honor_labels controls how Prometheus handles conflicts between labels that are
# already present in scraped data and labels that Prometheus would attach
# server-side ("job" and "instance" labels, manually configured target
# labels, and labels generated by service discovery implementations).
#
# If honor_labels is set to "true", label conflicts are resolved by keeping label
# values from the scraped data and ignoring the conflicting server-side labels.
#
# If honor_labels is set to "false", label conflicts are resolved by renaming
# conflicting labels in the scraped data to "exported_" (for
# example "exported_instance", "exported_job") and then attaching server-side
# labels.
#
# Setting honor_labels to "true" is useful for use cases such as federation and
# scraping the Pushgateway, where all labels specified in the target should be
# preserved.
#
# Note that any globally configured "external_labels" are unaffected by this
# setting. In communication with external systems, they are always applied only
# when a time series does not have a given label yet and are ignored otherwise.
[ honor_labels:  | default = false ]

What is the best way out for your setup?

[markphelps]

Reply from andi4000 on Discourse:

Thanks for the hint!

I consulted with a colleague and yea that could work. But we’re going to need to create our own ServiceMonitor manifest instead of using the template from flipt’s helm chart, because it’s not exposed there. I’m going to file a PR to expose this in the template → PR link.

Reference for future reader: ServiceMonitor [monitoring.coreos.com/v1] - Monitoring APIs | API reference | OpenShift Container Platform 4.10, look for honorLabels.

But in the long term, isn’t it better for flipt to use other label e.g. flipt_namespace to prevent collision in k8s deployments?

Edit: add PR and docs reference link

[markphelps]

Reply from andi4000 on Discourse:

Just wanted to confirm that creating own ServiceMonitor with the honorLabels flag works in our k8s setup

Edit: with PR above merged, the solution now is just to override the honorLabels to true flag in flipt’s official helm chart.

[markphelps]

Reply from member on Discourse:

Thanks for the suggested solution. It worked with this fix.

[markphelps]

Reply from mark on Discourse:

Hey @andi4000 ! Thanks for the contributions to the helm chart re: ServiceMonitor and honor labels. (And for coming back to update this post!)

I agree that we should probably deprecate namespace and move to more differentiated attributes.

This is currently what we have in code:

// Attributes used in evaluation metrics
AttributeMatch     = attribute.Key("match")
AttributeFlag      = attribute.Key("flag")
AttributeSegment   = attribute.Key("segment")
AttributeReason    = attribute.Key("reason")
AttributeValue     = attribute.Key("value")
AttributeNamespace = attribute.Key("namespace")
AttributeType      = attribute.Key("type")

We should likely add new attributes like:

// Attributes used in evaluation metrics
AttributeMatch     = attribute.Key("match")
AttributeNamespace = attribute.Key("flag_namespace")
AttributeFlag      = attribute.Key("flag_key")
AttributeFlagType  = attribute.Key("flag_type")
AttributeSegment   = attribute.Key("segment_key")
AttributeReason    = attribute.Key("reason")
AttributeValue     = attribute.Key("value")