How to record and query total requests (flask_http_request_total) PER endpoint/url_rule etc?

[dempstert]

Problem

As per the title, I don't know how to see total requests per endpoint.

Additional Note about uncertainty

I am not sure if the problem is my prometheus queries, or my configuration of flask prometheus itself. I believe the issue is with the configuration.

Question

How to configure prometheus_flask_exporter so that I can record flask_http_request_total per url_rule?

flask_http_request_total seems to group by response code only, but I want to record it on a url_rule bases

flask_http_request_duration_seconds_bucket and flask_http_request_duration_seconds_count and flask_http_request_duration_seconds_sum is grouped by url_rule by default.

What I've Tried

Below is my configuration. I have added default_labels but it doesn't help.

    metrics = GunicornPrometheusMetrics.for_app_factory(
        group_by="url_rule",
        excluded_paths=[
            "/js/*",
            "/css/*",
            "/images/*",
            "/fonts/*",
        ],
        default_labels={"url_rule": lambda: request.url_rule},
    )
    metrics.init_app(app)

[rycus86]

Some ideas that come to mind that you could try:

• aggregate from flask_http_request_duration_seconds_count on Prometheus if these are already tagged with url_rule
• add a default metric as per the example in the README and see if that helps:

metrics.register_default(
    metrics.counter(
        'by_url_rule_counter', 'Request count by URL rule',
        labels={'url_rule': lambda: request.url_rule}
    )
)

[dempstert]

Thanks @rycus86 but I'm afraid I'm not having any luck with that.

Seems you can't call make_default when using an app factory pattern.

Also, I believe that calling init_app calls the register_default method.

What I tried

    metrics = GunicornPrometheusMetrics.for_app_factory(
        group_by="url_rule",
        excluded_paths=[
            "/js/*",
            "/css/*",
            "/images/*",
            "/fonts/*",
        ],
        # default_labels={"url_rule": lambda: request.url_rule},
    )
    counter = metrics.counter(
        "by_url_rule_counter", "Request count by URL rule", labels={"url_rule": lambda: request.url_rule}
    )
    metrics.register_default(counter)

Traceback

  File "/app/prosebit/app.py", line 86, in create_celery_app
    app = app or create_app()
                 ^^^^^^^^^^^^
  File "/app/prosebit/app.py", line 245, in create_app
    metrics.register_default(counter)
  File "/usr/local/lib/python3.11/site-packages/prometheus_flask_exporter/__init__.py", line 563, in register_default
    for endpoint, view_func in app.view_functions.items():
                               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/werkzeug/local.py", line 316, in __get__
    obj = instance._get_current_object()  # type: ignore[misc]
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/werkzeug/local.py", line 509, in _get_current_object
    raise RuntimeError(unbound_message) from None
RuntimeError: Working outside of application context.

[rycus86]

Would you be able to set up a full minimal example we could test this on?
Based on that we should be able to figure out how to support what you're after.

[AndreyNautilus]

I experience the same issue.

Minimal example

I use app-factory approach and I want to make the metrics work locally first before dropping everything into k8s.

__init__.py file

from flask import Flask

from blueprint import backend_bp
from metrics import metrics


def create_app():
    app = Flask(__name__)

    app.register_blueprint(backend_bp)

    # FAILURE: this casues "RuntimeError: Working outside of application context." exception on startup
    # metrics.register_default(
    #     metrics.counter(
    #         'by_path_counter', 'Request count by request paths',
    #         labels={'path': lambda: request.path}
    #     )
    # )
    metrics.init_app(app)

    return app

blueprint.py file:

from flask import Blueprint
from metrics import metrics

backend_bp = Blueprint("backend", __name__)

by_path_counter = metrics.counter(
    'by_path_counter', 'Request count by request paths',
    labels={'path': lambda: request.path}
)


@by_path_counter
@backend_bp.route("/", methods=("GET",))
def root():
    return "root"


@by_path_counter
@backend_bp.route("/info", methods=("GET",))
def info():
    return "info"


@by_path_counter
@backend_bp.route("/custom", methods=("GET",))
def custom():
    return "custom"

metrics.py file:

from prometheus_flask_exporter import PrometheusMetrics

metrics = PrometheusMetrics.for_app_factory()

Python 3.12.1 and dependencies versions (I'm on Windows):

Flask                     3.0.3
prometheus_client         0.21.1
prometheus_flask_exporter 0.23.1

The command: python -m flask --app=. run --port=8001.

If I just run this code, then all endpoints work (localhost:8001/, localhost:8001/info, localhost:8001/custom and localhost:8001/metrics). I make a few requests and inspect the metrics:

• by_path_counter_total appears in comments only:

  ...
  # HELP by_path_counter_total Request count by request paths
  # TYPE by_path_counter_total counter
  # HELP flask_exporter_info Information about the Prometheus Flask exporter
  # TYPE flask_exporter_info gauge
  flask_exporter_info{version="0.23.1"} 1.0
  

• flask_http_request_duration_seconds_bucket appears with all requested path labels:

  flask_http_request_duration_seconds_bucket{le="0.005",method="GET",path="/custom",status="200"} 3.0
  ...
  flask_http_request_duration_seconds_bucket{le="0.005",method="GET",path="/info",status="200"} 4.0
  

If I try to use register_default approach (which I would prefer) and uncomment the code block in __init__.py file and drop manually added by_path_counter from backend.py file, then the app fails to start with the following error:

  File "C:\Projects\learn-k8s\app\.venv\Lib\site-packages\prometheus_flask_exporter\__init__.py", line 563, in register_default
    for endpoint, view_func in app.view_functions.items():
                               ^^^^^^^^^^^^^^^^^^
  File "C:\Projects\learn-k8s\app\.venv\Lib\site-packages\werkzeug\local.py", line 318, in __get__
    obj = instance._get_current_object()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Projects\learn-k8s\app\.venv\Lib\site-packages\werkzeug\local.py", line 519, in _get_current_object
    raise RuntimeError(unbound_message) from None
RuntimeError: Working outside of application context.

Counting number of requests per path is pretty simple metrics that first came to my mind. I expect it to be simple to implement, but it turned out to be not-so-simple :) Did I do something wrong?

[rycus86]

Thanks a lot for the minimal example @AndreyNautilus !
I was able to reproduce the issue, and the problem seems to be that register_default expects you either already have an app or the current_app is set up Flask already, which is not the case here.

You can simply pass the app instance to the register_default function and that seems to work ok, so in your code:

def create_app():
    app = Flask(__name__)

    app.register_blueprint(backend_bp)

    metrics.register_default(
        metrics.counter(
            'by_path_counter', 'Request count by request paths',
            labels={'path': lambda: request.path}
        ),
        app=app  # this makes it work
    )
    metrics.init_app(app)

    return app

I've also seen that if you use with app.app_context(): around the metrics registration code you get the same effect, but I think the former is clearer.

[AndreyNautilus]

Thanks @rycus86 ! This helps and now everything works.

Hey @rycus86 , how do we fight paths that have some item ID as a parameter? It's okay when it represents some finite list of statuses/enums, etc, but when it is an OrderID in production systems, cardinality grows so much and each route gets its own metric, causing the /metrics response to have dozens of thousands lines of code, putting pressure to the container resources.

[rycus86]

Hello @adnantrumo have you tried the configuration options in https://github.com/rycus86/prometheus_flask_exporter?tab=readme-ov-file#configuration ? You could try either the endpoint or url_rule grouping perhaps instead of the default path ?

Hello @adnantrumo have you tried the configuration options in https://github.com/rycus86/prometheus_flask_exporter?tab=readme-ov-file#configuration ? You could try either the endpoint or url_rule grouping perhaps instead of the default path ?

Yup, it works! Sorry for bothering!

[ktimothy]

By the way, what is the main reason for not having path as a label for flask_http_request_total and flask_http_request_exceptions_total? Seems like it would be great to have it out of the box.

[rycus86]

I think it's a valid setup either way, sometimes you just want to know how many requests/exceptions the app can see in total. path can have a bit of a cardinality issue in some setups, but when appropriate you can always configure it manually (see in some of the comments above).

We could perhaps make it configurable by default, but I'm not sure if it's needed vs where people want it they can just register an additional default metric.