Skip to content

Tornado instrumentation collecting partial/incorrect telemetry when URL excluded #3677

@devmonkey22

Description

@devmonkey22

Describe your environment

OS: Ubuntu
Python version: Python 3.10
SDK version: 1.31.1
API version: 0.54b0

What happened?

When a URL is being excluded by the TORNADO_EXCLUDED_URLS env var, the "prepare" metrics/trace spans are skipped, however, when on_finish runs, it always records metrics and (tries to) finish the trace span. The trace span will have no context (ctx), so those get skipped fine. However, metrics don't have such an escape. The server histograms including active requests are updated, causing incorrect values. It was most noticeable with active requests, when active requests went very negative (-1847 connections when I stopped it lol). I had been using Docker healthchecks to HTTP GET to an excluded URL, so the metrics just kept skewing and going more negative the longer the server was up.

Steps to Reproduce

  1. Add a URL to the TORNADO_EXCLUDED_URLS environment variable.
  2. Make a request to that URL repeatedly
  3. Watch the MetricInstruments.HTTP_SERVER_ACTIVE_REQUESTS metric go more negative after each request.

Expected Result

Metrics collection and finish span (traces) are fully skipped with excluded URLs.

Actual Result

The server histograms including active requests are updated, causing incorrect values. It was most noticeable with active requests, when active requests went very negative (-1847 connections when I stopped it lol). I had been using Docker healthchecks to HTTP GET to an excluded URL, so the metrics just kept skewing and going more negative the longer the server was up.

Additional context

No response

Would you like to implement a fix?

Yes

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions