Skip to content

fix(cu): add pending evals metric for alerts #1105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bredamatt wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix(cu): add pending evals metric for alerts #1105

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7dad06f
fix(cu): add pending evals metric for alerts
bredamatt Jan 10, 2025
d093512
fix: increment/decrement pendingEvaluationCounter based on appropriat…
bredamatt Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions servers/cu/src/bootstrap.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,15 @@ export const createApis = async (ctx) => {
labelNames: ['stream_type', 'message_type', 'process_error']
})

const pendingEvaluationCounter = MetricsClient.counterWith({})({
name: 'ao_process_pending_evaluations',
description: 'The total number of pending evaluations on a CU',
labelNames: ['type']
})

pendingEvaluationCounter.set({ type: 'primary' }, primaryWorkQueue.size)
Copy link
Contributor

@jfrain99 jfrain99 Jan 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will set the metric amount when this file is run at startup, then never again. We will need to move the sets to the location that the work queues are added to (evaluateWith, maybe?) and increment it by 1 on addition and decrement by 1 on completion.

set does not currently exist on the grafana counters. Only inc is implemented (see effects/metrics.js, counterWith). This can be implemented with inc / dec or feel free to implement set in metrics.js - up to you.

Copy link
Contributor Author

@bredamatt bredamatt Jan 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, will adjust this accordingly.

Copy link
Contributor Author

@bredamatt bredamatt Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jfrain99 Seems there is a way to listen to events on the PQueue and accordingly inc, dec the pendingEvaluationsCounter given those events take place. I've added this accordingly. Not so familiar with JS, so let me know if I need to change further.

pendingEvaluationCounter.set({ type: 'dry-run' }, dryRunWorkQueue.size)

/**
* TODO: Gas can grow to a huge number. We need to make sure this doesn't crash when that happens
*/
Expand Down
Loading