Merge pull request #20 from dwhswenson/more-docs

Docs: First batch of explanation, tutorials, and how-tos

Author: ethanholz

.readthedocs.yaml

@@ -0,0 +1,16 @@
+version: 2
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.12"
+
+python:
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - dev
+
+mkdocs:
+  configuration: mkdocs.yml

AGENTS.md

@@ -31,3 +31,16 @@
 - Use structured logging over prints; avoid noisy logs in hot paths.
 - Fail fast with clear errors; prefer explicit exceptions to silent fallbacks.
 - Keep configuration externalized (env vars/config files) and document defaults.
+
+## Documentation (Diataxis + MkDocs)
+- Write docs in Markdown and keep them under `docs/` for MkDocs rendering.
+- Follow Diataxis and keep content intent-specific:
+  - `docs/tutorials/` for guided learning.
+  - `docs/how-to/` for task-oriented procedures.
+  - `docs/reference/` for factual API/module/config details.
+  - `docs/explanation/` for rationale, tradeoffs, and architecture context.
+- Do not mix Diataxis types on a single page; split pages when intent changes.
+- Any PR that changes behavior, inputs/outputs, or operations should update the corresponding docs in the same PR.
+- Prefer runnable, copy/paste-safe examples with prerequisites and expected outcomes.
+- Keep MkDocs navigation explicit in `mkdocs.yml` and use stable page paths/titles to reduce link churn.
+

docs/explanation/AWSArchitecture-ConceptualArchitecture.drawio.svg

@@ -0,0 +1,29 @@
+# Architecture
+
+## Overview
+
+The basic idea is that we have a scheduled lambda function (triggered by EventBridge events) that performs some tasks and publishes results to a shared SNS topic. Then we have notification modules that subscribe to this SNS topic with filter policies to receive only the relevant messages. This allows us to decouple the task execution from the notification handling and makes it easier to add new notification modules in the future without changing the core logic of the scheduled lambda.
+
+![Architecture Diagram](AWSArchitecture-ConceptualArchitecture.drawio.svg)
+
+The overall data flow is that an EventBridge event triggers the scheduled lambda. That scheduled lambda executes its tasks and then publishes results to the SNS topic.The results are a JSON object where the keys represent the type of result (e.g., "success", "failure", "warning") and the values contain additional information that can be useful for notifying the end user. The SQS FIFO queues subscribe to specific notification types based on filter policies, so a given queue might, for example, only receive messages where `result_type` is "failure". The notification handler lambdas are then triggered by messages in the SQS queues, and use Jinja templates to format the notifications before sending them to the end user (e.g., via email).
+
+## Modules
+
+When we decompose the architecture into Terraform modules, the architecture splits differently from the conceptual architecture. The message broker splits into some components being associated with the notification lambdas (the SQS queues) and some parts being associated with the core functionality (the SNS topic).
+
+![Module-based architecture diagram](AWSArchitecture-ModuleArchitecture.drawio.svg)
+
+We provide a `scheduled-lambda` module that handles everything about the EventBridge rules and the Lambda function that performs the scheduled tasks. You'll need to provide a Docker image for the Lambda function, which will do the actual business logic.
+
+The root module wraps the `scheduled-lambda` module with an SNS topic, and handles the wiring of that.
+
+Notification modules are separate, and when you deploy, you'll need to include your own notification modules: the root module does not contain any default notifications. Each notification module manages the SQS queue and the Lambda function to handle messages, as well as the subscription to the SNS topic with the appropriate filter policy. There's a core `notification-plumbing` module, and individual modules for different notification types (e.g., email via SES), are wrappers around that core module, exposing the correct parameters to configure the specific notification type.
+
+
+
+## Notification Templates
+
+One of the big ideas of LambdaCron is that the notifications use Jinja templates to format the messages sent to the end user. This allows for a lot of flexibility in how the notifications are presented, and makes it easy to include dynamic content based on the results of the scheduled tasks.
+
+The key thing is that the template must correspond to the structure of the JSON object published to the SNS topic. This is how we can insert a message broker that knows nothing about the specific use case of the scheduled lambda: it just gives us a message (as a JSON object) and then it tells us how to format that message using the Jinja template.

docs/explanation/AWSArchitecture-LambdaModules.drawio.svg

@@ -0,0 +1,103 @@
+# Data Flow: `_perform_task` to Rendered Notification
+
+This page traces the data flow from a scheduled task's `_perform_task` return value through SNS publication, SQS subscription, notifier parsing, and final template rendering. It illustrates how the shape of the data evolves at each stage and how it interacts with filter policies and message attributes.
+
+## Scenario
+
+Assume one scheduled task publishes into one shared SNS topic, and two notifiers are subscribed with filter policies:
+
+* Print notifier queue: `result_types = ["OK"]`
+* Email notifier queue: `result_types = ["ERROR"]`
+
+## Phase 1: `_perform_task` Return Value
+
+Let's say the lambda runs a few tasks, which either pass (status `OK`) or fail (status `ERROR`). Let's say that the task-specific information to be used in the templates are `taskid` and `name`. Here's an example of what the return value of `_perform_task` might look like:
+
+```json
+{
+  "OK": {
+    "tasks": [{"taskid": 1, "name": "Foo"}, {"taskid": 2, "name": "Bar"}]
+  },
+  "ERROR": {
+    "tasks": [{"taskid": 3, "name": "Baz"}]
+  }
+}
+```
+
+## Phase 2: What Gets Published to SNS
+
+We publish one SNS message per status type, so in this case two messages: one for `OK` and one for `ERROR`. Each message includes the relevant portion of the `_perform_task` return value (JSON-encoded) as the `Message` payload. The `result_type` is included as a `MessageAttribute` to allow for filtering by the SQS subscriptions.
+
+For `OK`, the publish call payload shape is:
+
+```json
+{
+  "TopicArn": "arn:aws:sns:us-east-1:123456789012:lambdacron-results.fifo",
+  "Message": "{\"tasks\": [{\"taskid\": 1, \"name\": \"Foo\"}, {\"taskid\": 2, \"name\": \"Bar\"}]}",
+  "Subject": "Notification for OK",
+  "MessageAttributes": {
+    "result_type": {
+      "DataType": "String",
+      "StringValue": "OK"
+    }
+  },
+  "MessageGroupId": "lambdacron"
+}
+```
+
+For `ERROR`, the shape is identical except:
+
+- `Message` is `"{\"tasks\": [{\"taskid\": 3, \"name\": \"Baz\"}]}"`
+- `Subject` is `"Notification for ERROR"`
+- `MessageAttributes.result_type.StringValue` is `"ERROR"`
+
+## Phase 3: What Arrives in Each SQS Queue
+
+Because subscriptions filter on `MessageAttributes.result_type`, each queue receives only matching messages. From here, we'll just follow the message for `OK`; the message for `ERROR` has the same shape but different content (and a different template rendering result in the end).
+
+The SQS event record (as seen by notifier Lambda) looks like:
+
+```json
+{
+  "Records": [
+    {
+      "messageId": "msg-ok-1",
+      "eventSource": "aws:sqs",
+      "body": "{\"tasks\": [{\"taskid\": 1, \"name\": \"Foo\"}, {\"taskid\": 2, \"name\": \"Bar\"}]}",
+      "messageAttributes": {
+        "result_type": {
+          "stringValue": "OK",
+          "dataType": "String"
+        }
+      }
+    }
+  ]
+}
+```
+
+## Phase 4: Notifier Parse Behavior with This Shape
+
+The notifier parser converts the JSON string back into an object, and, if `result_type` is missing from the payload, injects it from the SQS `messageAttributes`. In this case, the payload doesn't include `result_type`, so it is injected.
+
+The result, which is fed to the notifier's templates, is:
+
+```json
+{
+  "tasks": [{"taskid": 1, "name": "Foo"}, {"taskid": 2, "name": "Bar"}],
+  "result_type": "OK"
+}
+```
+
+## Phase 5: Template Rendering
+
+Let's say the template used by the print notifier for `OK` messages is:
+
+```jinja2
+Result {{ result_type }}: {% for task in tasks %}#{{ task.taskid }} {{ task.name }}{% if not loop.last %}, {% endif %}{% endfor %}
+```
+
+The output for the `OK` message would be:
+
+```text
+Result OK: #1 Foo, #2 Bar
+```

docs/explanation/AWSArchitecture-ModuleArchitecture.drawio.svg

@@ -0,0 +1,17 @@
+# Lambda Container Image Architecture
+
+
+## Overview
+
+LambdaCron uses Docker images to back our Lambda functions. Docker-based Lambdas are easier to develop and test locally, and are actually faster to start up than zip-based Lambdas.
+
+Since your Lambda needs to be backed by an ECR repository in your own region, we require that our lambda modules take an image URI as input. We provide modules that cover 2 approaches to having the user deploy the image into their own ECR repository:
+
+1. `lambda-image-build`: Directly build the code. This takes a local code source and builds the Docker image, then deploys it to the user's account. This is most suitable for development and for users who want to customize the code.
+2. `lambda-image-republish`: Republish a public ECR image. This simply copies a public image into the user's ECR repository. This is for users who want to use the default code, or pin to a specific and well-documented version of the image.
+
+In order to facilitate developers creating their own tools build on LambdaCron, we also provide a `lambda-image-public` module, which builds the image and publishes it to a public ECR repository.
+
+![AWSArchitecture-LambdaModules.drawio.svg](AWSArchitecture-LambdaModules.drawio.svg)
+
+The two approaches are illustrated in the figure. The direct-from-source approach is on the top, and the republish approach is on the bottom. Normally, the core maintainer of the image does the first stage in the republish approach, and users can do the second stage to copy the image into their own account.

docs/explanation/architecture.md

@@ -0,0 +1,45 @@
+# Set Up SES Prerequisites
+
+Set up Amazon SES once per AWS account and region where your notifier Lambda runs. AWS now guides much of this in the SES setup wizard; this page is the short checklist for LambdaCron.
+
+
+## When to Use
+
+* You want to use Amazon SES as the delivery provider for LambdaCron email notifications.
+* You have not yet prepared SES identities or account access in the target AWS region.
+
+## Before You Begin
+
+* Choose the AWS account/region where email will be sent (SES setup is regional).
+* Choose a sender identity:
+  * Email identity for one sender address.
+  * Domain identity if you want to send from multiple addresses in one domain.
+* Make sure you can edit DNS records if you choose a domain identity.
+
+SES requires verified identities for senders and (in sandbox) recipients. That means you must verify the email address or domain you want to send from, and if in sandbox, also verify any recipient addresses. Sandbox mode limits how much you can send, and you'll probably want to request production access if you want to send to more than one recipient.
+
+## Steps
+
+1. Open Amazon SES in the notifier's region and follow the setup flow.
+   * Start with AWS's setup guide and wizard: <https://docs.aws.amazon.com/ses/latest/dg/setting-up.html>.
+2. Create and verify your sender identity.
+   * SES requires verified identities for `From`/`Sender` addresses.
+   * Use the verified identities guide: <https://docs.aws.amazon.com/ses/latest/dg/verify-addresses-and-domains.html>.
+   * If using a domain, publish SES-provided DNS records (including DKIM) and wait for `Verified`.
+3. Account for sandbox restrictions while testing.
+   * New SES accounts are in sandbox mode per region.
+   * In sandbox, you can only send to verified recipients (or the SES mailbox simulator).
+4. Request production access when you are ready to send to unverified recipients.
+   * Submit a production access request in the same region: <https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html>.
+   * Keep your verified sender identity in place after approval.
+5. (Recommended) Publish SPF/DMARC records for your sending domain to improve deliverability.
+6. Send a test email from SES in that region before deploying the notifier.
+   * Use the exact sender address you plan to configure in the `email-notification` module.
+
+## Validation
+
+* SES identity status is `Verified` for your sender.
+* All required tasks are completed on the SES console's "Get set up" page.
+* If still in sandbox, all test recipients are verified.
+* If in production, you can send to non-verified recipients.
+* A console/API test send succeeds in the same region as your notifier Lambda.

docs/explanation/data-flow.md

@@ -0,0 +1,129 @@
+# Use Email Notifications
+
+The LambdaCron root module creates the scheduled Lambda and shared SNS topic, but it does not create notification channels. Use `email-notification` to send selected `result_type` values through Amazon SES.
+
+## When to Use
+* You want SES email notifications for one or more LambdaCron `result_type` values.
+* You already have, or can provide, a notification-handler container image.
+
+## Before You Begin
+
+* Complete SES setup in [Set Up SES Prerequisites](set-up-ses.md).
+* Verify your SES sender identity in the same AWS region as your notifier Lambda.
+* If your account is in SES sandbox mode, verify recipient addresses too.
+* Create your subject/text/html template files.
+
+## Inputs to Provide
+
+* `sns_topic_arn` from your LambdaCron stack output.
+* `lambda_image_uri` for the notification handler container.
+* `fifo_queue_name` ending with `.fifo`.
+* `sender`, `recipients`, and optional `reply_to`.
+* `subject_template_file`, `text_template_file`, and `html_template_file`.
+* Optional runtime/routing settings such as `result_types`, `lambda_name`, `batch_size`, and `tags`.
+
+## Steps
+
+### 1. Create email templates
+
+The email notification module requires 3 Jinja templates: one for the email subject, one for the text-only body, and one for the HTML body.
+
+```jinja
+{# templates/email-subject.txt #}
+[{{ result_type }}] LambdaCron notification
+```
+
+```jinja
+{# templates/email-body.txt #}
+Result type: {{ result_type }}
+Message: {{ message }}
+```
+
+```html
+<!-- templates/email-body.html -->
+<h2>LambdaCron notification</h2>
+<p><strong>Result type:</strong> {{ result_type }}</p>
+<p><strong>Message:</strong> {{ message }}</p>
+```
+
+### 2. Republish the notification image or get the image URI
+
+If you have already republished the notification handler image to your own ECR repository, you can skip this step. Otherwise, use the `lambda-image-republish` module to copy the public image to your account/region.
+
+```hcl
+module "notification_image_republish" {
+  source = "../../modules/lambda-image-republish"
+
+  source_lambda_repo = "public.ecr.aws/i9p4w7k9/lambdacron-notifications"
+  source_lambda_tag  = "latest"
+}
+```
+
+If you do this, you can wire in the lambda image URI from the module output as shown in the next step.
+
+On the other hand, if you have your own notification handler image, you can provide the URI directly. To get that, you can find it in the ECR console or use the AWS CLI:
+
+```bash
+aws ecr describe-repositories --repository-names "your-repo-name" --query "repositories[0].repositoryUri" --output text
+```
+
+
+### 3. Add the `email-notification` module
+
+You'll connect it to the LambdaCron SNS topic and provide the email settings and templates. You'll also select which result types to send email for. If you leave `result_types` empty or omit it, the notification handler will send emails for all result types. **You will get one email for each notification type**, this will not combine multiple notification types into a single email.
+
+```hcl
+module "email_notification" {
+  source = "../../modules/email-notification"
+
+  sns_topic_arn    = module.lambdacron.sns_topic_arn
+  fifo_queue_name  = "lambdacron-email.fifo"
+  lambda_image_uri = module.notification_image_republish.lambda_image_uri_with_digest
+
+  result_types = ["example", "ERROR"]
+
+  sender     = var.email_sender
+  recipients = var.email_recipients
+  reply_to   = var.email_reply_to
+
+  subject_template_file = "${path.module}/templates/email-subject.txt"
+  text_template_file    = "${path.module}/templates/email-body.txt"
+  html_template_file    = "${path.module}/templates/email-body.html"
+
+  tags = local.common_tags
+}
+```
+
+### 4. Plan and apply
+
+Here we use `tofu`, but you could also use `terraform`. This will deploy your infrastructure.
+
+```bash
+tofu plan
+tofu apply
+```
+
+## Validation
+
+There are two relatively easy ways you can trigger the email notification lambda:
+
+1. [Deploy your LambdaCron with `create_test_url`](use-test-url.md) enabled and invoke the test URL.
+2. Publish a test message with a `result_type` value included in `result_types`:
+
+```bash
+aws sns publish \
+  --topic-arn "$(tofu output -raw sns_topic_arn)" \
+  --message '{"message":"Email notification smoke test"}' \
+  --message-attributes '{"result_type":{"DataType":"String","StringValue":"example"}}' \
+  --message-group-id "email-smoke-test"
+```
+
+Note that the test URL won't create an email unless the result creates a message with a `result_type` value included in your notifier's `result_types` list.
+
+If you receive the email, then it was successful!
+
+If not, here are some troubleshooting tips:
+
+* Check the CloudWatch logs for the notifier Lambda for any errors. In particular, it might complain if you aren't allowed to send email (see setup instructions in [Set Up SES Prerequisites](set-up-ses.md)).
+* If using the test URL, check the CloudWatch logs for the scheduled Lambda to confirm that it is working.
+* If you don't see logs in the CloudWatch logs for your lambdas, check the Lambda console to confirm that the Lambda was created successfully and that it has the correct triggers.