Skip to content

Fix Prometheus query timestamp formatting - #2397

Open
Pybsama wants to merge 1 commit into
cruise-control-for-kafka:mainfrom
Pybsama:codex/plain-prometheus-query-timestamps
Open

Fix Prometheus query timestamp formatting#2397
Pybsama wants to merge 1 commit into
cruise-control-for-kafka:mainfrom
Pybsama:codex/plain-prometheus-query-timestamps

Conversation

@Pybsama

@Pybsama Pybsama commented Jul 31, 2026

Copy link
Copy Markdown

Summary

  1. Why: PrometheusAdapter serializes Unix timestamps through Double.toString(), which emits scientific notation for current epoch values. Scientific notation is outside the documented Prometheus range-query timestamp format and is rejected by stricter compatible backends such as VictoriaMetrics.
  2. What: serialize start and end directly from epoch milliseconds with BigDecimal.toPlainString(), and add a regression test that inspects the real URL-encoded POST body.

Expected Behavior

PrometheusMetricSampler sends exact plain-decimal Unix timestamps, such as
1784144612.388, so Prometheus-compatible range-query backends can parse them.

Actual Behavior

The previous implementation sent values such as 1.784144612388E9.
VictoriaMetrics rejects that value with HTTP 422, so sampling fails and no monitored
windows are populated.

Steps to Reproduce

  1. Configure prometheus.server.endpoint to use a VictoriaMetrics endpoint.
  2. Start Cruise Control with PrometheusMetricSampler.
  3. Observe /api/v1/query_range receive an exponent-form start/end value and
    return HTTP 422.

The added regression test reproduces the serialization issue without requiring an
external Prometheus-compatible service.

Known Workarounds

Use a backend that leniently accepts exponent-form timestamps. There is no Cruise
Control configuration that changes the previous timestamp serialization.

Additional evidence

  1. Environment: macOS arm64, Eclipse Temurin JDK 17.0.20.
  2. Before the production change, the new regression test failed deterministically:
    expected 1784144612.388, but received 1.784144612388E9.
  3. After the change, all 29 tests in the Prometheus sampling package pass, including
    a clean run from a fresh clone of the public branch.
  4. The same Gradle verification also completed Checkstyle, SpotBugs, and
    JaCoCo report generation successfully.

Categorization

  • documentation
  • bugfix
  • new feature
  • refactor
  • security/CVE
  • other

This PR resolves #2389.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes Prometheus range-query timestamp serialization so start/end are always sent as plain decimal seconds (not scientific notation), improving compatibility with stricter Prometheus-compatible backends (e.g., VictoriaMetrics).

Changes:

  • Serialize start and end from epoch milliseconds using BigDecimal.toPlainString() to avoid exponent formatting.
  • Add a regression test that captures and parses the actual URL-encoded POST body to assert timestamp formatting.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/monitor/sampling/prometheus/PrometheusAdapter.java Replaces double-based timestamp stringification with BigDecimal plain-decimal serialization for start/end.
cruise-control/src/test/java/com/linkedin/kafka/cruisecontrol/monitor/sampling/prometheus/PrometheusAdapterTest.java Adds a regression test that inspects the real form body to ensure start/end are not in scientific notation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@Pybsama
Pybsama force-pushed the codex/plain-prometheus-query-timestamps branch from a495a4a to a5efc47 Compare August 28, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PrometheusMetricSampler emits query timestamps in scientific notation, breaking strict Prometheus-compatible backends (VictoriaMetrics 422)

2 participants