Merge pull request #12 from dwhswenson/single-sns-topic

Single SNS topic, instead of topic per result type

Author: ethanholz

IDEA.md

@@ -6,36 +6,35 @@ This is intended to be a framework that can be used by client code to define the
 
 ## Components
 
-1. **Scheduled Lambda Functions**: The goal here is that the client can provide their own lambda function (as a container image) and, from that, we will run it on a schedule defined by the client. The lambda function needs to know 1 or more SNS topics to which it will publish messages when it runs; different "types" of messages can go to different SNS topics, which will then be subscribed to by different notification channels. This will include the lambda execution role and the scheduled events.
-2. **SNS Topics**: These will be manually created by the client code, but ARNs might be needed to give the lambda permissions to publish.
-3. **Notification Channels**: We will provide modules for different notification channels (e.g., email via SES, SMS via Twilio, etc.). Each notification module owns the SNS->SQS->Lambda wiring: it provisions the FIFO SQS queue/subscription used for deduplication and triggers its handler; the user should not create that queue manually. Each channel ships its own container image (build or republish) that renders a Jinja2 template and delivers via its notifier.
+1. **Scheduled Lambda Functions**: The goal here is that the client can provide their own lambda function (as a container image) and, from that, we will run it on a schedule defined by the client. The lambda function will publish to a single SNS topic and set a `result_type` (or similar) message attribute; notification channels will filter on one or more types. This will include the lambda execution role and the scheduled events.
+2. **SNS Topic**: A single topic is manually created by the client code; its ARN is needed to give the lambda permissions to publish. Filtering is done via subscription filter policies, not separate topics.
+3. **Notification Channels**: We will provide modules for different notification channels (e.g., email via SES, SMS via Twilio, etc.). Each notification module owns the SNS->SQS->Lambda wiring: it provisions the FIFO SQS queue/subscription used for deduplication and triggers its handler, with an SNS filter policy on the result type(s); the user should not create that queue manually. Each channel ships its own container image (build or republish) that renders a Jinja2 template and delivers via its notifier.
 4. **Lambda Image Utilities**: In addition to republishing an existing Lambda container, we will provide a module to build an image from a local directory containing a Dockerfile and publish it to ECR for use by the scheduled-lambda module.
 5. **Python Runtime Library**: Provide reusable Python code in `src/cloud_cron/` that makes authoring custom scheduled lambdas easy (task base class, SNS dispatch helpers, and ergonomic handler wiring). This includes a template provider abstraction so notification handlers can source templates from env vars, URLs, or S3.
 
 
 The goal is that the user will need to:
 
-1. Write a lambda function that publishes to the desired SNS topics (provided as envionment variables).
+1. Write a lambda function that publishes to the shared SNS topic and sets a `result_type` message attribute (provided as an environment variable or constant).
 2. Write a small terraform module that looks something like:
 
 ```hcl
-resource "aws_sns_topic" "example_topic" {
-  name = "example-topic"
+resource "aws_sns_topic" "results" {
+  name = "cloud-cron-results"
 }
 
 module "my_scheduled_lambda" {
   source = "./modules/scheduled-lambda"
 
   lambda_image_uri = "123456789012.dkr.ecr.us-west-2.amazonaws.com/my-lambda:latest"
   schedule_expression = "rate(5 minutes)"
-  sns_topic_arns = {
-    example = aws_sns_topic.example_topic.arn
-  }
+  sns_topic_arn = aws_sns_topic.results.arn
 }
 
 module "my_email_notification" {
   source = "./modules/email-notification"
-  sns_topic_arn = aws_sns_topic.example_topic.arn
+  sns_topic_arn = aws_sns_topic.results.arn
+  result_types = ["example"]
   template_file = "path/to/email/template.html"
   email_sender = "me@example.com"
   email_recipients = [
@@ -47,6 +46,10 @@ module "my_email_notification" {
 
 (There may be additional parameters needed, but this is the general idea.)
 
+Notes on the single-topic approach:
+- It simplifies wiring and allows channels to subscribe to multiple result types via filter policies.
+- It reduces per-type IAM/topic configuration, but also means topic-level settings (KMS, delivery policy, metrics) are shared.
+
 ## Additional convenience
 
 We should also provide a module that allows us to take an externally-defined lambda and redeploy it in a local environment. The idea is that the lambdas users use are served from their own accounts, and are copies of our official release lambdas.

PLAN.md

@@ -27,12 +27,36 @@
 - [x] Example touchpoint: allow `examples/basic` to build/push a simple placeholder Lambda image from a local Dockerfile as an alternative to the republish module.
 
 ## Phase 2: Build scheduled Lambda module (`modules/scheduled-lambda`)
-- [x] Define inputs: `lambda_image_uri`, `schedule_expression`, `sns_topic_arns` (map topic key->ARN), optional `lambda_env`, `timeout`, `memory_size`, `tags`.
-- [x] Create resources: IAM role/policy (CloudWatch Logs + `sns:Publish` to provided ARNs), Lambda from container image, EventBridge rule/target/permission.
+- [x] Define inputs: `lambda_image_uri`, `schedule_expression`, `sns_topic_arn` (single), optional `lambda_env`, `timeout`, `memory_size`, `tags`.
+- [x] Create resources: IAM role/policy (CloudWatch Logs + `sns:Publish` to provided ARN), Lambda from container image, EventBridge rule/target/permission.
 - [x] Outputs: Lambda ARN, execution role ARN, log group name, schedule rule name.
 - [x] Docs: README with usage matching IDEA example.
 - [x] Example touchpoint: scaffold `examples/basic` with this module + stub SNS topic(s) and the container image outputs from Phase 1; `terraform validate/plan` should pass to prove schedule wiring.
 
+## Phase 2.5: Consolidate result routing into a single SNS topic
+
+Overview: replace per-result-type topics with one shared SNS topic and use message attributes + subscription filter policies to route results to notification channels.
+
+Success criteria:
+
+- Scheduled lambdas publish to one topic and set a `result_type` message attribute.
+- Notification modules accept `result_types` (list) and apply SNS filter policies to their subscriptions.
+- Examples and docs reflect the single-topic pattern.
+
+Decisions and motivations:
+
+- Use SNS filter policies to reduce infrastructure and make multi-type subscriptions easy.
+- Keep result types as message attributes to avoid changing payload shapes or handler code.
+
+To-do:
+
+- [x] Update `modules/scheduled-lambda` to accept `sns_topic_arn` (single) and adjust IAM to `sns:Publish` on that ARN.
+- [x] Update notification plumbing module to accept `result_types` and apply SNS filter policy on the subscription.
+- [x] Update existing per-channel modules to pass through `result_types` and document the attribute name.
+- [x] Update `src/cloud_cron/` helpers to publish with a `result_type` attribute and validate allowed types.
+- [x] Update `examples/basic` to use one topic and a single `result_types` subscription.
+- [x] Update module READMEs and `IDEA.md` usage examples to match the new wiring.
+
 ## Phase 3: Python runtime library for custom lambdas (`src/cloud_cron/`)
 
 Overview: turn the existing Python helpers into a reusable, testable library that makes it easy for users to author scheduled lambdas while keeping SNS wiring and logging consistent. This package will also host shared notification handler code (under `src/cloud_cron/notifications/`), while deployment/container wiring remains in Terraform modules.

examples/basic/main.tf

@@ -58,14 +58,11 @@ locals {
   active_print_image_uri  = module.print_lambda_image_build.image_uri_with_digest
 }
 
-module "sns_topics" {
-  source = "../../modules/sns-topics"
-
-  topic_names = {
-    example = "example-topic.fifo"
-  }
-
-  tags = local.common_tags
+resource "aws_sns_topic" "results" {
+  name                        = "cloud-cron-results.fifo"
+  fifo_topic                  = true
+  content_based_deduplication = true
+  tags                        = local.common_tags
 }
 
 module "scheduled_lambda" {
@@ -74,15 +71,15 @@ module "scheduled_lambda" {
   lambda_image_uri    = local.active_lambda_image_uri
   schedule_expression = var.schedule_expression
   lambda_name         = var.lambda_name
-  sns_topic_arns      = module.sns_topics.topic_arns
+  sns_topic_arn       = aws_sns_topic.results.arn
 
   tags = local.common_tags
 }
 
 module "print_notification" {
   source = "../../modules/print-notification"
 
-  sns_topic_arn    = module.sns_topics.topic_arns.example
+  sns_topic_arn    = aws_sns_topic.results.arn
   fifo_queue_name  = "example-print.fifo"
   lambda_image_uri = local.active_print_image_uri
   template_file    = "${path.module}/templates/print.txt"

modules/notification-plumbing/README.md

@@ -11,13 +11,15 @@ module "notification_plumbing" {
   sns_topic_arn       = aws_sns_topic.example.arn
   lambda_function_arn = aws_lambda_function.print.arn
   fifo_queue_name     = "example-notifications.fifo"
+  result_types        = ["example"]
 }
 ```
 
 ## Inputs
 
 - `sns_topic_arn` (string): SNS topic ARN that feeds the notification queue.
 - `lambda_function_arn` (string): ARN of the Lambda function that processes SQS messages.
+- `result_types` (list(string)): Result types to subscribe to. Empty means all.
 - `fifo_queue_name` (string): Name of the FIFO SQS queue (must end with `.fifo`).
 - `content_based_deduplication` (bool): Enable content-based deduplication. Default `true`.
 - `visibility_timeout_seconds` (number): Visibility timeout for the queue. Default `30`.

modules/notification-plumbing/main.tf

@@ -1,5 +1,8 @@
 locals {
   tags = merge({ managed_by = "cloudcron" }, var.tags)
+  filter_policy = length(var.result_types) > 0 ? jsonencode({
+    result_type = var.result_types
+  }) : null
 }
 
 resource "aws_sqs_queue" "dlq" {
@@ -57,6 +60,8 @@ resource "aws_sns_topic_subscription" "queue" {
   endpoint  = aws_sqs_queue.queue.arn
 
   raw_message_delivery = true
+  filter_policy        = local.filter_policy
+  filter_policy_scope  = length(var.result_types) > 0 ? "MessageAttributes" : null
 }
 
 resource "aws_lambda_event_source_mapping" "sqs" {

modules/notification-plumbing/variables.tf

@@ -12,6 +12,12 @@ variable "lambda_function_arn" {
   type        = string
 }
 
+variable "result_types" {
+  description = "Result types to subscribe to; empty means all."
+  type        = list(string)
+  default     = []
+}
+
 variable "fifo_queue_name" {
   description = "Name of the FIFO SQS queue (must end with .fifo)."
   type        = string

modules/print-notification/README.md

@@ -9,6 +9,7 @@ module "print_notification" {
   source = "./modules/print-notification"
 
   sns_topic_arn   = aws_sns_topic.example.arn
+  result_types    = ["example"]
   fifo_queue_name = "example-print.fifo"
   lambda_image_uri = module.print_image.image_uri_with_digest
 
@@ -19,6 +20,7 @@ module "print_notification" {
 ## Inputs
 
 - `sns_topic_arn` (string): SNS topic ARN that feeds the notification queue.
+- `result_types` (list(string)): Result types to subscribe to. Empty means all.
 - `fifo_queue_name` (string): Name of the FIFO SQS queue (must end with `.fifo`).
 - `lambda_image_uri` (string): URI of the Lambda container image.
 - `lambda_name` (string): Optional name for the Lambda function.

modules/print-notification/main.tf

@@ -27,6 +27,7 @@ module "plumbing" {
 
   sns_topic_arn       = var.sns_topic_arn
   lambda_function_arn = aws_lambda_function.print.arn
+  result_types        = var.result_types
   fifo_queue_name     = var.fifo_queue_name
   batch_size          = var.batch_size
   enabled             = var.enabled

modules/print-notification/variables.tf

@@ -3,6 +3,12 @@ variable "sns_topic_arn" {
   type        = string
 }
 
+variable "result_types" {
+  description = "Result types to subscribe to; empty means all."
+  type        = list(string)
+  default     = []
+}
+
 variable "fifo_queue_name" {
   description = "Name of the FIFO SQS queue (must end with .fifo)."
   type        = string

modules/scheduled-lambda/README.md

@@ -1,22 +1,20 @@
 # Scheduled Lambda Module
 
-Provision a container-based Lambda function that runs on an EventBridge schedule and publishes to SNS topics.
+Provision a container-based Lambda function that runs on an EventBridge schedule and publishes to a shared SNS topic.
 
 ## Usage
 
 ```hcl
-resource "aws_sns_topic" "example_topic" {
-  name = "example-topic"
+resource "aws_sns_topic" "results" {
+  name = "cloud-cron-results"
 }
 
 module "scheduled_lambda" {
   source = "./modules/scheduled-lambda"
 
   lambda_image_uri     = "123456789012.dkr.ecr.us-west-2.amazonaws.com/my-lambda:latest"
   schedule_expression  = "rate(5 minutes)"
-  sns_topic_arns = {
-    task = aws_sns_topic.example_topic.arn
-  }
+  sns_topic_arn         = aws_sns_topic.results.arn
 
   lambda_env = {
     LOG_LEVEL = "info"
@@ -28,7 +26,7 @@ module "scheduled_lambda" {
 
 - `lambda_image_uri` (string): URI of the Lambda container image.
 - `schedule_expression` (string): EventBridge schedule expression.
-- `sns_topic_arns` (map(string)): Logical topic keys to SNS topic ARN mapping.
+- `sns_topic_arn` (string): SNS topic ARN for publishing results.
 - `lambda_env` (map(string)): Additional environment variables for the Lambda.
 - `timeout` (number): Lambda timeout in seconds.
 - `memory_size` (number): Lambda memory size in MB.