Counters increase return no value

[bmgante]

Hi everyone,

I have a metric counter which represents the number of calls, and for some labels combination, it generates a single sample data only from time to time.

The problem is that in the dashboard with the following promql these samples are not considered. I know this is a known architecture characteristic for rate/increase functions... but is there some workaround to consider these metrics that do not have the first value as zero? I am using vector to create this metrics from a logs stream and the combination of labels is not predictable.

Promql:
sum(increase(number_of_calls{job="$job",foo="$foo"}[$__interval]))

Also, as soon as the metric is generated it expires after 60s (flush_period_secs) and if the same metric has value again after 5minutes for instance, the increase will be zero again, so I am wondering to increase the flush_period_secs to a very high value, or is there some way to disable it (set it to 0 disable?!) ? Unfortunately even this way, the first value of each counter will be always discarded when applying a increase/rate function.

I really appreciate some guidance on this.

Thanks

[jszwedko]

I think increasing flush_period_secs to greater than the interval between metric emission would help. It doesn't look like flushing can be disabled in the prometheus_exporter sink at the moment, but you could set a very high number that would have the same effect.

[bmgante]

that's exactly what i did, increased it to 24h. now, the only problem is the first value of the counter which is never taken in account when applying the rate/increase.

[jszwedko]

Are the values in the logs you are converting actually incremental? Or are they absolute? Can you share the log_to_metric configuration you are using?

[bmgante]

yes they are incremental... it is a metric to count the number of calls.

    metrics_trunk_calls:
      type: log_to_metric
      inputs:
        - idr_remap
      metrics:
        - field: .variable_direction
          name: trunk_calls
          type: counter
          tags:
            direction: "{{`{{.variable_direction}}`}}"
            account_id: "{{`{{.variable_account_id}}`}}"
            trunk: "{{`{{.call.trunk.name}}`}}"
            cluster: "{{`{{.cluster}}`}}"
            transport: "{{`{{.sip_transport}}`}}"
            codec: "{{`{{.variable_rtp_use_codec_name}}`}}"
            freeswitch: "{{`{{.freeswitch}}`}}"
            fs_instance: "{{`{{.fs_instance}}`}}"
            user_agent: "{{`{{.variable_sip_user_agent}}`}}"