[PRMP-926] Create Log Alarm for PatchReviewDocument Lambda

[steph-torres-nhs]

Overview

Jira ticket: PRMP-926

Description

Creation of a CloudWatch Log Metric Filter and Alarm to trigger on log pattern

Context

I want to be alerted when the PatchReviewDocument Lambda fails to delete a file from S3 So that I can investigate and manually clean up the file, even if the Lambda execution technically succeeded.

Checklist

Tasks for all changes:

• 1. I have linked this PR to its Jira ticket.
• 2. I have run git pre-commits.
• [] 3. I have updated relevant documentation.
• 4. I have considered the cross-team impact (and have PR approval from both Core & Demographics if necessary).
• 5. I have successfully deployed this change to a sandbox and witnessed it build: Workflow run
• 6. I have checked the Terraform Plan from this PR against ndr-dev.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[robg-test]

1. Happy that this has been tested by proxy of the alarm being enabled on sandbox

[github-actions]

Report for environment: ndr-dev

Terraform Initialization ⚙️success

Initialization Output

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...
- access-audit-alarm in modules/lambda_alarms
- access-audit-alarm-topic in modules/sns
- access-audit-gateway in modules/gateway
- access-audit-lambda in modules/lambda
- access_audit_dynamodb_table in modules/dynamo_db
- alarm_state_history_table in modules/dynamo_db
- api_endpoint_url_ssm_parameter in modules/ssm_parameter
- auth_session_dynamodb_table in modules/dynamo_db
- auth_state_dynamodb_table in modules/dynamo_db
- authoriser-alarm in modules/lambda_alarms
- authoriser-alarm-topic in modules/sns
- authoriser-lambda in modules/lambda
- back-channel-logout-gateway in modules/gateway
- back_channel_logout_alarm in modules/lambda_alarms
- back_channel_logout_alarm_topic in modules/sns
- back_channel_logout_lambda in modules/lambda
- bulk-upload-alarm in modules/lambda_alarms
- bulk-upload-alarm-topic in modules/sns
- bulk-upload-lambda in modules/lambda
- bulk-upload-metadata-alarm in modules/lambda_alarms
- bulk-upload-metadata-alarm-topic in modules/sns
- bulk-upload-metadata-lambda in modules/lambda
- bulk-upload-metadata-processor-alarm in modules/lambda_alarms
- bulk-upload-metadata-processor-alarm-topic in modules/sns
- bulk-upload-metadata-processor-lambda in modules/lambda
- bulk-upload-report-alarm in modules/lambda_alarms
- bulk-upload-report-alarm-topic in modules/sns
- bulk-upload-report-lambda in modules/lambda
- bulk_upload_metadata_preprocessor_lambda in modules/lambda
- bulk_upload_report_dynamodb_table in modules/dynamo_db
Downloading registry.terraform.io/cloudstoragesec/cloud-storage-security/aws 1.8.10+css9.03.000 for cloud_storage_security...
- cloud_storage_security in .terraform/modules/cloud_storage_security
- cloudfront_edge_dynamodb_table in modules/dynamo_db
- cloudfront_firewall_waf_v2 in modules/firewall_waf_v2
- core_dynamodb_table in modules/dynamo_db
- create-doc-ref-lambda in modules/lambda
- create-token-gateway in modules/gateway
- create-token-lambda in modules/lambda
- create_doc_alarm in modules/lambda_alarms
- create_doc_alarm_topic in modules/sns
- create_token-alarm in modules/lambda_alarms
- create_token-alarm_topic in modules/sns
- data-collection-alarm in modules/lambda_alarms
- data-collection-alarm-topic in modules/sns
- data-collection-lambda in modules/lambda
- delete-doc-ref-gateway in modules/gateway
- delete-doc-ref-lambda in modules/lambda
- delete-document-object-alarm in modules/lambda_alarms
- delete-document-object-alarm-topic in modules/sns
- delete-document-object-lambda in modules/lambda
- delete_doc_alarm in modules/lambda_alarms
- delete_doc_alarm_topic in modules/sns
- document-manifest-job-gateway in modules/gateway
- document-manifest-job-lambda in modules/lambda
- document-status-check-alarm in modules/lambda_alarms
- document-status-check-alarm-topic in modules/sns
- document-status-check-gateway in modules/gateway
- document-status-check-lambda in modules/lambda
- document_manifest_alarm in modules/lambda_alarms
- document_manifest_alarm_topic in modules/sns
- document_reference_dynamodb_table in modules/dynamo_db
- document_reference_gateway in modules/gateway
- document_reference_id_gateway in modules/gateway
- document_review_dlq_alarm_topic in modules/sns
- document_review_processor_lambda in modules/lambda
- document_review_queue in modules/sqs
- document_upload_check_lambda in modules/lambda
- document_upload_review_dynamodb_table in modules/dynamo_db
- edge-presign-lambda in modules/lambda_edge
- edge_presign_alarm in modules/lambda_alarms
- edge_presign_alarm_topic in modules/sns
- feature-flags-gateway in modules/gateway
- feature-flags-lambda in modules/lambda
- feature_flags_alarm in modules/lambda_alarms
- feature_flags_alarm_topic in modules/sns
- fhir_document_reference_gateway in modules/gateway
- fhir_document_reference_mtls_gateway in modules/gateway
- firewall_waf_v2 in modules/firewall_waf_v2
- firewall_waf_v2_api in modules/firewall_waf_v2
- generate-document-manifest-alarm in modules/lambda_alarms
- generate-document-manifest-alarm-topic in modules/sns
- generate-document-manifest-lambda in modules/lambda
- generate-lloyd-george-stitch-alarm in modules/lambda_alarms
- generate-lloyd-george-stitch-alarm-topic in modules/sns
- generate-lloyd-george-stitch-lambda in modules/lambda
- get-doc-fhir-lambda in modules/lambda
- get-doc-ref-alarm in modules/lambda_alarms
- get-doc-ref-alarm-topic in modules/sns
- get-doc-ref-lambda in modules/lambda
- get-report-by-ods-alarm in modules/lambda_alarms
- get-report-by-ods-alarm-topic in modules/sns
- get-report-by-ods-gateway in modules/gateway
- get-report-by-ods-lambda in modules/lambda
- get_document_review_lambda in modules/lambda
- get_document_review_lambda_alarm in modules/lambda_alarms
- get_document_review_lambda_alarm_topic in modules/sns
- global_sqs_age_alarm_topic in modules/sns
- im-alerting-lambda in modules/lambda
- lambda-layer-alerting in modules/lambda_layers
- lambda-layer-core in modules/lambda_layers
- lambda-layer-data in modules/lambda_layers
- lambda-layer-reports in modules/lambda_layers
- lg-bulk-upload-expedite-metadata-queue in modules/sqs
- lloyd-george-stitch-gateway in modules/gateway
- lloyd-george-stitch-lambda in modules/lambda
- lloyd-george-stitch_alarm in modules/lambda_alarms
- lloyd-george-stitch_topic in modules/sns
- lloyd_george_reference_dynamodb_table in modules/dynamo_db
- login_redirect-alarm_topic in modules/sns
- login_redirect_alarm in modules/lambda_alarms
- login_redirect_lambda in modules/lambda
- logout-gateway in modules/gateway
- logout_alarm in modules/lambda_alarms
- logout_alarm_topic in modules/sns
- logout_lambda in modules/lambda
- manage-nrl-pointer-alarm in modules/lambda_alarms
- manage-nrl-pointer-alarm-topic in modules/sns
- manage-nrl-pointer-lambda in modules/lambda
- migration-dynamodb-lambda in modules/lambda
- migration-dynamodb-segment-lambda in modules/lambda
- migration-dynamodb-segment-store in modules/s3
- migration-failed-items-store in modules/s3
- mns-dlq-alarm-topic in modules/sns
- mns-notification-alarm in modules/lambda_alarms
- mns-notification-alarm-topic in modules/sns
- mns-notification-lambda in modules/lambda
- mns_encryption_key in modules/kms
- mtls_api_endpoint_url_ssm_parameter in modules/ssm_parameter
- ndr-app-config in modules/app_config
- ndr-bulk-staging-store in modules/s3
- ndr-docker-ecr-data-collection in modules/ecr
- ndr-docker-ecr-ui in modules/ecr
- ndr-document-pending-review-store in modules/s3
- ndr-document-store in modules/s3
- ndr-ecs-container-port-ssm-parameter in modules/ssm_parameter
- ndr-ecs-fargate-app in modules/ecs
- ndr-ecs-fargate-data-collection in modules/ecs
- ndr-feedback-mailbox in modules/ses
- ndr-lloyd-george-store in modules/s3
- ndr-truststore in modules/s3
- ndr-vpc-ui in modules/vpc
- ndr-zip-request-store in modules/s3
- nhs-oauth-token-generator-alarm in modules/lambda_alarms
- nhs-oauth-token-generator-alarm-topic in modules/sns
- nhs-oauth-token-generator-lambda in modules/lambda
- nrl-dlq-alarm-topic in modules/sns
- patch_document_review_lambda in modules/lambda
- patch_document_review_lambda_alarm in modules/lambda_alarms
- patch_document_review_lambda_alarm_topic in modules/sns
- pdf-stitching-alarm-topic in modules/sns
- pdf-stitching-lambda in modules/lambda
- pdf-stitching-lambda-alarms in modules/lambda_alarms
- pdm-document-store in modules/s3
- pdm_dynamodb_table in modules/dynamo_db
- pdm_encryption_key in modules/kms
- post-document-references-fhir-lambda in modules/lambda
- post_document_review_lambda in modules/lambda
- post_document_review_lambda_alarm in modules/lambda_alarms
- post_document_review_lambda_alarm_topic in modules/sns
- review-document-status-check-alarm-topic in modules/sns
- review-document-status-check-lambda in modules/lambda
- review-document-status-check-lambda-alarm in modules/lambda_alarms
- review_document_gateway in modules/gateway
- review_document_status_gateway in modules/gateway
- review_document_version_gateway in modules/gateway
- route53_fargate_ui in modules/route53
- search-document-references-fhir-lambda in modules/lambda
- search-document-references-gateway in modules/gateway
- search-document-references-lambda in modules/lambda
- search-patient-details-gateway in modules/gateway
- search-patient-details-lambda in modules/lambda
- search_doc_alarm in modules/lambda_alarms
- search_doc_alarm_topic in modules/sns
- search_document_review_lambda in modules/lambda
- search_document_review_lambda_alarm in modules/lambda_alarms
- search_document_review_lambda_alarm_topic in modules/sns
- search_patient_alarm in modules/lambda_alarms
- search_patient_alarm_topic in modules/sns
- send-feedback-alarm in modules/lambda_alarms
- send-feedback-alarm-topic in modules/sns
- send-feedback-gateway in modules/gateway
- send-feedback-lambda in modules/lambda
- sns_encryption_key in modules/kms
- sqs-lg-bulk-upload-invalid-queue in modules/sqs
- sqs-lg-bulk-upload-metadata-queue in modules/sqs
- sqs-mns-notification-queue in modules/sqs
- sqs-nrl-queue in modules/sqs
- sqs-stitching-queue in modules/sqs
- sqs_alarm_lambda_topic in modules/sns
- ssm_param_external_client_cert in modules/ssm_parameter
- ssm_param_external_client_key in modules/ssm_parameter
- ssm_param_mtls_common_names in modules/ssm_parameter
- ssm_param_unauthorised_client_cert in modules/ssm_parameter
- ssm_param_unauthorised_client_key in modules/ssm_parameter
- statistical-report-alarm in modules/lambda_alarms
- statistical-report-alarm-topic in modules/sns
- statistical-report-lambda in modules/lambda
- statistical-reports-store in modules/s3
- statistics_dynamodb_table in modules/dynamo_db
- stitch_metadata_reference_dynamodb_table in modules/dynamo_db
- stitching-dlq-alarm-topic in modules/sns
- toggle-bulk-upload-lambda in modules/lambda
- transfer-key-manager-alarm in modules/lambda_alarms
- transfer-key-manager-alarm-topic in modules/sns
- transfer-key-manager-lambda in modules/lambda
- transfer_family_kill_switch_lambda in modules/lambda
- unstitched_lloyd_george_reference_dynamodb_table in modules/dynamo_db
- update-doc-ref-alarm in modules/lambda_alarms
- update-doc-ref-alarm-topic in modules/sns
- update-doc-ref-lambda in modules/lambda
- update-upload-state-gateway in modules/gateway
- update-upload-state-lambda in modules/lambda
- update_upload_state_alarm in modules/lambda_alarms
- update_upload_state_alarm_topic in modules/sns
- virus_scan_result_alarm in modules/lambda_alarms
- virus_scan_result_alarm_topic in modules/sns
- virus_scan_result_gateway in modules/gateway
- virus_scan_result_lambda in modules/lambda
- zip_store_reference_dynamodb_table in modules/dynamo_db
Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding latest version of hashicorp/random...
- Finding latest version of hashicorp/time...
- Finding hashicorp/aws versions matching ">= 4.0.0, ~> 5.0"...
- Finding hashicorp/awscc versions matching ">= 0.72.1, ~> 1.0"...
- Finding latest version of hashicorp/archive...
- Installing hashicorp/random v3.7.2...
- Installed hashicorp/random v3.7.2 (signed by HashiCorp)
- Installing hashicorp/time v0.13.1...
- Installed hashicorp/time v0.13.1 (signed by HashiCorp)
- Installing hashicorp/aws v5.100.0...
- Installed hashicorp/aws v5.100.0 (signed by HashiCorp)
- Installing hashicorp/awscc v1.67.0...
- Installed hashicorp/awscc v1.67.0 (signed by HashiCorp)
- Installing hashicorp/archive v2.7.1...
- Installed hashicorp/archive v2.7.1 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Terraform Validation 🤖success

Validation Output

Success! The configuration is valid.

Terraform Plan 📖success

Show Plan (4 to add, 4 to change, 2 to destroy)

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.edge-presign-lambda.aws_lambda_function.lambda has changed
  ~ resource "aws_lambda_function" "lambda" {
        id                             = "ndr-dev_EdgePresignLambda"
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:[REDACTED_AWS_ACCOUNT_ID]:function:ndr-dev_EdgePresignLambda:552" -> "arn:aws:lambda:us-east-1:[REDACTED_AWS_ACCOUNT_ID]:function:ndr-dev_EdgePresignLambda:558"
        tags                           = {}
        # (28 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
+/- create replacement and then destroy

Terraform will perform the following actions:

  # aws_api_gateway_deployment.ndr_api_deploy must be replaced
+/- resource "aws_api_gateway_deployment" "ndr_api_deploy" {
      ~ created_date  = "2026-01-06T15:05:09Z" -> (known after apply)
      ~ execution_arn = "arn:aws:execute-api:eu-west-2:[REDACTED_AWS_ACCOUNT_ID]:ccy0v3rve9/" -> (known after apply)
      ~ id            = "z9mfa9" -> (known after apply)
      ~ invoke_url    = "[REDACTED_API_GATEWAY_URL]" -> (known after apply)
      ~ variables     = {
          - "deployed_at" = "2026-01-06T15:05:08Z"
        } -> (known after apply) # forces replacement
        # (2 unchanged attributes hidden)
    }

  # aws_api_gateway_deployment.ndr_api_deploy_mtls must be replaced
+/- resource "aws_api_gateway_deployment" "ndr_api_deploy_mtls" {
      ~ created_date  = "2026-01-06T15:04:55Z" -> (known after apply)
      ~ execution_arn = "arn:aws:execute-api:eu-west-2:[REDACTED_AWS_ACCOUNT_ID]:p9uuab4oyl/" -> (known after apply)
      ~ id            = "aiwjpp" -> (known after apply)
      ~ invoke_url    = "[REDACTED_API_GATEWAY_URL]" -> (known after apply)
      ~ variables     = {
          - "deployed_at" = "2026-01-06T15:04:54Z"
        } -> (known after apply) # forces replacement
        # (2 unchanged attributes hidden)
    }

  # aws_api_gateway_stage.ndr_api will be updated in-place
  ~ resource "aws_api_gateway_stage" "ndr_api" {
      ~ deployment_id         = "z9mfa9" -> (known after apply)
        id                    = "ags-ccy0v3rve9-dev"
        tags                  = {}
        # (14 unchanged attributes hidden)
    }

  # aws_api_gateway_stage.ndr_api_mtls will be updated in-place
  ~ resource "aws_api_gateway_stage" "ndr_api_mtls" {
      ~ deployment_id         = "aiwjpp" -> (known after apply)
        id                    = "ags-p9uuab4oyl-dev"
        tags                  = {}
        # (14 unchanged attributes hidden)
    }

  # aws_cloudwatch_log_metric_filter.review_patch_failed_to_delete_from_s3[0] will be created
  + resource "aws_cloudwatch_log_metric_filter" "review_patch_failed_to_delete_from_s3" {
      + id             = (known after apply)
      + log_group_name = "/aws/lambda/ndr-dev_PatchDocumentReview"
      + name           = "ReviewPatchFailedToDeleteFromS3"
      + pattern        = "%Unable to delete file%"

      + metric_transformation {
          + name      = "S3DeleteFailures"
          + namespace = "App/Review"
          + unit      = "None"
          + value     = "1"
        }
    }

  # aws_cloudwatch_metric_alarm.review_patch_failed_to_delete_from_s3[0] will be created
  + resource "aws_cloudwatch_metric_alarm" "review_patch_failed_to_delete_from_s3" {
      + actions_enabled                       = true
      + alarm_actions                         = [
          + "arn:aws:sns:eu-west-2:[REDACTED_AWS_ACCOUNT_ID]:ndr-dev-alarms-notification-topic-[REDACTED_AWS_ACCOUNT_ID][REDACTED_AWS_ACCOUNT_ID]14",
        ]
      + alarm_description                     = "Triggers when the PatchDocumentReview fails to delete an object from S3."
      + alarm_name                            = "ndr-dev_PatchDocumentReview_failed_to_delete_from_s3"
      + arn                                   = (known after apply)
      + comparison_operator                   = "GreaterThanThreshold"
      + evaluate_low_sample_count_percentiles = (known after apply)
      + evaluation_periods                    = 1
      + id                                    = (known after apply)
      + metric_name                           = "S3DeleteFailures"
      + namespace                             = "App/Review"
      + ok_actions                            = [
          + "arn:aws:sns:eu-west-2:[REDACTED_AWS_ACCOUNT_ID]:ndr-dev-sns-patch-document-review-lambda-alarm-topic[REDACTED_AWS_ACCOUNT_ID][REDACTED_AWS_ACCOUNT_ID]04",
        ]
      + period                                = 300
      + statistic                             = "Sum"
      + tags_all                              = {
          + "Environment" = "dev"
          + "Owner"       = "nhse/ndr-team"
          + "Workspace"   = "ndr-dev"
        }
      + threshold                             = 0
      + treat_missing_data                    = "missing"
    }

  # module.ndr-ecs-fargate-app.aws_ecs_task_definition.ndr_ecs_task will be updated in-place
  ~ resource "aws_ecs_task_definition" "ndr_ecs_task" {
        id                       = "ndr-dev-task-app-cluster"
        tags                     = {}
      ~ tags_all                 = {
          + "Environment" = "dev"
          + "Owner"       = "nhse/ndr-team"
          + "Workspace"   = "ndr-dev"
        }
        # (16 unchanged attributes hidden)
    }

  # module.ndr-ecs-fargate-data-collection[0].aws_ecs_task_definition.ndr_ecs_task will be updated in-place
  ~ resource "aws_ecs_task_definition" "ndr_ecs_task" {
        id                       = "ndr-dev-task-data-collection"
        tags                     = {}
      ~ tags_all                 = {
          + "Environment" = "dev"
          + "Owner"       = "nhse/ndr-team"
          + "Workspace"   = "ndr-dev"
        }
        # (16 unchanged attributes hidden)
    }

Plan: 4 to add, 4 to change, 2 to destroy.