Skip to content

Azure monitor infrastructure metrics on storage queues more testing #617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mrlockstar merged 13 commits into from
Nov 26, 2025
Merged

Azure monitor infrastructure metrics on storage queues more testing #617

mrlockstar merged 13 commits into from
Nov 26, 2025

Conversation

@mrlockstar
Copy link
Contributor

@mrlockstar mrlockstar commented Oct 28, 2025 •
edited
Loading

Description

We need to setup technical tools and alerting to allow us to monitor our live infrastructure.
This is a ticket to setup and investigate whether Azure monitor infrastructure metrics can meet our needs.

Below is the guide about how to create them via Microsoft
https://learn.microsoft.com/en-us/azure/storage/queues/monitor-queue-storage?source=recommendations&tabs=azure-portal

This PR

  • Moves the creation of RBAC so it's implemented in the Storage account child module as this eliminates potential circle dependency and simplifies the code.
  • Creates a custom opentelemetry Azure metric gauge as a service class.
  • Adds unit tests to the new service class
  • Creates a few new alarms inside the terraform code for both the notifications-message-status-updates and notifications-message-batch-retries storage queues.
image image

Jira link

https://nhsd-jira.digital.nhs.uk/browse/DTOSS-11379

Review notes

@github-actions
Copy link

github-actions bot commented Oct 28, 2025 •
edited
Loading

The review app at this URL has been deleted:
https://pr-617.manage-breast-screening.non-live.screening.nhs.uk

@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch from 0338403 to 88413c6 Compare October 30, 2025 11:05
@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch 2 times, most recently from b03e41b to bf7d56e Compare November 4, 2025 09:37
@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch 2 times, most recently from 92d103d to 56ff15a Compare November 4, 2025 09:43
@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch from 44eb77c to eb75adb Compare November 6, 2025 13:17
@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch from 9d9fec9 to 9e776d6 Compare November 24, 2025 17:37
@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch from 9ff7228 to c4c1264 Compare November 25, 2025 15:02
@mrlockstar
Metric service class now a singleton with an array of gauges per inst…
c397452 
…ance

To avoid the complexity of running multiple instances of the Metric Service,
which would introduce unnecessary computational overhead, we decided it would
be better to use a single instance of the Metric Service class and allow it
to manage an array of gauges instead.

Co-authored-by: Alastair Lock
Co-authored-by: Colin Saliceti
@mrlockstar
update the unit tests
f23728f
@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch from c4c1264 to f23728f Compare November 25, 2025 15:11
@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch from 5e1d48f to eb4c7e7 Compare November 25, 2025 15:21
@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch from 694122d to b2d7aae Compare November 25, 2025 15:31
@mrlockstar mrlockstar force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch from b2d7aae to 69ff47d Compare November 25, 2025 15:38
Harriethw
Harriethw reviewed Nov 26, 2025
View reviewed changes
@steventux steventux force-pushed the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch from 1d996f3 to 3abef93 Compare November 26, 2025 11:26
steventux
steventux approved these changes Nov 26, 2025
View reviewed changes
Copy link
Contributor

@steventux steventux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another reviewer need for my changes but this looks good to me. :shipit:

@mrlockstar mrlockstar merged commit 6ff6934 into main Nov 26, 2025
13 checks passed
@mrlockstar mrlockstar deleted the feat/DTOSS-11379-Azure-monitor-infrastructure-metrics-on-storage-queues-more-testing branch November 26, 2025 12:54
@Harriethw Harriethw mentioned this pull request Jan 2, 2026
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@saliceti saliceti saliceti left review comments

@Harriethw Harriethw Harriethw left review comments

@steventux steventux steventux approved these changes

@josielsouzanordcloud josielsouzanordcloud Awaiting requested review from josielsouzanordcloud

Assignees

No one assigned

Labels

deploy

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mrlockstar @steventux @saliceti @Harriethw