Skip to content

Add observability dashboards and alerting configuration#117

Merged
shayancoin merged 3 commits intomainfrom
codex/create-and-configure-grafana-dashboards
Oct 18, 2025
Merged

Add observability dashboards and alerting configuration#117
shayancoin merged 3 commits intomainfrom
codex/create-and-configure-grafana-dashboards

Conversation

@shayancoin
Copy link
Owner

@shayancoin shayancoin commented Oct 16, 2025

Summary

  • add Grafana dashboard provisioning for backend, sync pipeline, frontend UX, and tracing explorer views
  • configure Prometheus alert rules and Grafana contact points aligned with alerting conventions
  • document how to supply secrets for the provisioned integrations

Testing

  • for file in ops/grafana/provisioning/dashboards/*.json; do jq empty "$file"; done

https://chatgpt.com/codex/tasks/task_e_68f1350019148330a5c6ad77e445196f

Summary by CodeRabbit

  • New Features

    • Added four new Grafana dashboards: Backend Golden Signals, Frontend UX Metrics, Sync Pipeline KPIs, and Tracing Explorer for comprehensive service monitoring
    • Configured alerting infrastructure with Slack and email notifications
    • Added alert rules for high error rates, slow API performance, and frontend LCP degradation
  • Documentation

    • Added Grafana provisioning documentation

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Oct 16, 2025

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

This PR adds Grafana provisioning configuration for observability infrastructure, including alerting contact points, Prometheus alert rules, and four new dashboards for backend services, frontend UX metrics, sync pipeline KPIs, and tracing exploration. Supporting documentation and dashboard provider configuration are also included.

Changes

Cohort / File(s) Change Summary
Documentation
ops/grafana/README.md
Adds Grafana provisioning usage guide describing alerting setup, secrets management (Slack webhook, email), and deployment steps.
Alerting Configuration
ops/grafana/provisioning/alerting/contact-points.yaml, ops/prometheus/alerts.yml
Defines Slack and Email contact points with notification routing policy; adds three Prometheus alert rules for high error rate, slow API latency, and frontend LCP degradation.
Dashboard Definitions
ops/grafana/provisioning/dashboards/backend_golden_signals.json, frontend_ux_metrics.json, sync_pipeline_kpis.json, tracing_explorer.json
Adds four dashboard configurations with templated variables, PromQL/Tempo/Loki queries, and panels for monitoring backend signals, frontend Web Vitals, pipeline KPIs, and distributed traces.
Dashboard Provider Configuration
ops/grafana/provisioning/dashboards/dashboards.yaml
Configures Grafana provisioning provider for file-based dashboard management with auto-refresh and folder structure support.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Rationale: PR spans 8 configuration files across multiple domains (dashboards, alerts, provisioning). While mostly declarative configuration, the heterogeneity of file types (JSON dashboards with PromQL, YAML alert rules, provisioning configs) and density of metric queries require careful validation of syntax and logic consistency across each file. Limited repetition patterns but varied content types increase per-file reasoning overhead.

Poem

🐰 Through dashboards bright and alerts so keen,
The rabbit hops to monitor the scene—
Golden signals glow, frontend vitals dance,
While traces leap and pipelines prance!
Observability magic, now provisionéd with care

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Description Check ⚠️ Warning The PR description does not follow the repository’s required template: it is missing the “PR Type” section, uses “## Summary” instead of the “# Short Description” heading, and has “## Testing” rather than a “# Tests Added” section, so the structure and required headings are not satisfied. Please update the PR description to match the repository template by adding a “PR Type” header with one of the allowed values, renaming your summary under a “# Short Description” heading, and including a “# Tests Added” section that lists any new or modified tests.
✅ Passed checks (2 passed)
Check name Status Explanation
Title Check ✅ Passed The title “Add observability dashboards and alerting configuration” clearly and concisely summarizes the primary changes in this pull request by indicating that both dashboards and alerting configuration are being added, making it easy for reviewers to understand the intent at a glance.
Docstring Coverage ✅ Passed No functions found in the changes. Docstring coverage check skipped.
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch codex/create-and-configure-grafana-dashboards

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a60cf87 and 20ec327.

📒 Files selected for processing (1)
  • ops/grafana/provisioning/dashboards/dashboards.yaml (1 hunks)
🔇 Additional comments (1)
ops/grafana/provisioning/dashboards/dashboards.yaml (1)

3-13: Provisioning config looks solid.

Provider wiring matches the mounted dashboards directory and uses sane defaults; this should let Grafana pick up the shipped JSON files without surprises.


Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting

Comment on lines 1 to 13
apiVersion: 1

providers:
- name: paform-operations
orgId: 1
folder: Operations
type: file
disableDeletion: true
allowUiUpdates: false
updateIntervalSeconds: 30
options:
path: ops/grafana/provisioning/dashboards
foldersFromFilesStructure: true

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Place dashboards provisioning file where Grafana reads it

The new provisioning file points Grafana to ops/grafana/provisioning/dashboards, but it was added at dashboards/dashboards.yaml, outside the ops/grafana/provisioning tree the README instructs operators to mount into /etc/grafana/provisioning. Because Grafana only scans YAML files inside the mounted provisioning directory, none of the JSON dashboards will be discovered, so the commit’s dashboards are never provisioned. Either relocate this YAML under ops/grafana/provisioning/dashboards or adjust the mount path so Grafana can read it.

Useful? React with 👍 / 👎.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d2a35c4 and a60cf87.

📒 Files selected for processing (8)
  • dashboards/dashboards.yaml (1 hunks)
  • ops/grafana/README.md (1 hunks)
  • ops/grafana/provisioning/alerting/contact-points.yaml (1 hunks)
  • ops/grafana/provisioning/dashboards/backend_golden_signals.json (1 hunks)
  • ops/grafana/provisioning/dashboards/frontend_ux_metrics.json (1 hunks)
  • ops/grafana/provisioning/dashboards/sync_pipeline_kpis.json (1 hunks)
  • ops/grafana/provisioning/dashboards/tracing_explorer.json (1 hunks)
  • ops/prometheus/alerts.yml (1 hunks)
🧰 Additional context used
🪛 LanguageTool
ops/grafana/README.md

[grammar] ~1-~1: Use correct spacing
Context: # Grafana Provisioning This directory contains provisioning def...

(QB_NEW_EN_OTHER_ERROR_IDS_5)


[grammar] ~3-~3: Use correct spacing
Context: ...contact points are loaded automatically. ## Secrets * Slack webhook: set the `S...

(QB_NEW_EN_OTHER_ERROR_IDS_5)


[grammar] ~5-~5: Use correct spacing
Context: ...ts are loaded automatically. ## Secrets * Slack webhook: set the `SLACK_WEBHOOK_...

(QB_NEW_EN_OTHER_ERROR_IDS_5)


[grammar] ~8-~8: Use correct spacing
Context: ...t with the correct on-call mailing list. After adjusting secrets, restart Grafana...

(QB_NEW_EN_OTHER_ERROR_IDS_5)

allowUiUpdates: false
updateIntervalSeconds: 30
options:
path: ops/grafana/provisioning/dashboards
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Fix the dashboards provider path

When you follow the README and mount ops/grafana/provisioning into /etc/grafana/provisioning, Grafana will look for dashboards at /etc/grafana/provisioning/dashboards. The current value ops/grafana/provisioning/dashboards only exists in the repo root and will not resolve inside the container, so Grafana never loads any dashboards. Point the provider at the mounted path instead.

-      path: ops/grafana/provisioning/dashboards
+      path: /etc/grafana/provisioning/dashboards
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
path: ops/grafana/provisioning/dashboards
path: /etc/grafana/provisioning/dashboards
🤖 Prompt for AI Agents
In dashboards/dashboards.yaml around line 12, the provider path points to the
repo-local path "ops/grafana/provisioning/dashboards" which does not exist
inside the container; update the path to the container-mounted directory
"/etc/grafana/provisioning/dashboards" so Grafana can find and load the
dashboards when the provisioning directory is mounted.

@shayancoin shayancoin merged commit 39502ea into main Oct 18, 2025
2 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant