padim-aws-infra-anomaly

Anomaly detection for AWS infrastructure metrics — PaDiM-inspired, fully serverless.

CloudWatch time-series and Cost Explorer data are converted into 2-D images via Gramian Angular Field, features are extracted with a frozen EfficientNet-B4, and Mahalanobis distance over per-patch Gaussian distributions produces anomaly scores and visual heatmaps — one per resource, per metric group.

---

How it works

CloudWatch / Cost Explorer
         │
         ▼
  ┌─────────────┐     Parquet     ┌──────────────────┐
  │  Ingestion  │ ─────────────► │   S3 raw bucket  │
  │  (Phase 1)  │                └────────┬─────────┘
  └─────────────┘                         │ Parquet
                                          ▼
                                ┌──────────────────┐
                                │  GAF Transform   │   sliding window
                                │  (Phase 2)       │ ─────────────────►  (N, 3, 64, 64)
                                └──────────────────┘   per metric group
                                          │
                                          ▼
                                ┌──────────────────┐
                                │ EfficientNet-B4  │   frozen backbone
                                │ Feature Extract  │ ──────────────────►  patch embeddings
                                │  (Phase 3)       │
                                └──────────────────┘
                                          │
                                          ▼
                                ┌──────────────────┐
                                │  Mahalanobis     │   per-patch Gaussian
                                │  Inference       │ ──────────────────►  anomaly score
                                │  (Phase 4)       │                       + heatmap
                                └──────────────────┘

---

Stack

Layer	Technology
Data collection	boto3 — CloudWatch, Cost Explorer
Storage	S3 (Parquet via PyArrow)
Signal → image	pyts — Gramian Angular Field
Image resize	scikit-image
Feature extraction	EfficientNet-B4 (efficientnet-pytorch, frozen)
Anomaly model	scikit-learn (Gaussian), numpy (Mahalanobis)
Experiment tracking	MLflow
Training infra	Amazon SageMaker
IaC	Terraform ≥ 1.5
Tests	pytest + unittest.mock

---

Metrics collected (8 total)

Group	Metrics	CloudWatch namespace
0	CPUUtilization, NetworkIn, NetworkOut	AWS/EC2
1	DiskReadOps, DatabaseConnections, ReadLatency	AWS/EC2 + AWS/RDS
2	Duration, Errors	AWS/Lambda

Each group of 3 metrics becomes one RGB-equivalent image tensor (N, 3, 64, 64) fed to EfficientNet.

---

AWS infrastructure

Provisioned via Terraform (terraform/main.tf):

Resource	Details
S3 bucket	padim-infra-anomaly-raw — versioning on, AES-256, lifecycle → IA after 30 days
IAM role	padim-collector-role — CloudWatch read, Cost Explorer read, S3 write

cd terraform
terraform init
terraform apply

---

S3 path conventions

# CloudWatch metrics (Phase 1)
s3://padim-infra-anomaly-raw/
  cloudwatch/account={account_id}/resource={resource_id}/date={YYYY/MM/DD}/data.parquet

# Cost Explorer (Phase 1)
  cost/account={account_id}/date={YYYY/MM/DD}/cost_daily.parquet

# GAF images (Phase 2)
  gaf/account={account_id}/resource={resource_id}/metric={metric}/date={YYYY/MM/DD}/image.npy

---

Getting started

pip install -r requirements.txt

Make sure ~/.aws/config contains a [profile labs] entry pointing to account 810710296000.

Phase 1 — collect data

# CloudWatch metrics for one or more resource IDs
python -m src.ingestion.cloudwatch_collector i-0abc123 i-0def456

# Daily cost breakdown by service
python -m src.ingestion.cost_explorer_collector

Phase 2 — generate GAF images

# Transform a resource for today
python -m src.transform.gaf_transform i-0abc123

# Or for a specific date
python -m src.transform.gaf_transform i-0abc123 2024/01/15

Run all tests

# Standard
pytest tests/ -v

# With uv (no local install required)
uv run --with pytest --with pandas --with pyarrow --with numpy \
       --with pyts --with scikit-image --with matplotlib \
       pytest tests/ -v

---

Project structure

padim-aws-infra-anomaly/
├── src/
│   ├── ingestion/
│   │   ├── cloudwatch_collector.py   # Phase 1 — CloudWatch → S3
│   │   └── cost_explorer_collector.py # Phase 1 — Cost Explorer → S3
│   ├── transform/
│   │   └── gaf_transform.py          # Phase 2 — GAF sliding window transform
│   ├── model/
│   │   └── ...                       # Phase 3 — EfficientNet-B4 + Gaussian fitting
│   └── inference/
│       └── ...                       # Phase 4 — Mahalanobis scoring + heatmaps
├── tests/
│   ├── test_ingestion.py             # 4 tests — Phase 1
│   └── test_gaf.py                   # 15 tests — Phase 2
├── notebooks/
│   ├── 01_eda.ipynb                  # Exploratory analysis
│   └── 02_gaf_visualization.ipynb    # GASF vs GADF, normal vs anomalous windows
├── terraform/
│   └── main.tf                       # S3 bucket + IAM role
├── requirements.txt
└── CLAUDE.md

---

GAFTransformer — interface contract

from src.transform.gaf_transform import GAFConfig, GAFTransformer

cfg = GAFConfig(image_size=64, window_size=24, step_size=1, method="summation")
t   = GAFTransformer(cfg)

# Single metric: 1-D series → (N_windows, 64, 64)
images = t.fit_transform(series)

# Multivariate: DataFrame + metric list → (N_windows, N_groups*3, 64, 64)
out = t.fit_transform_multivariate(df, metric_cols)

# Slice per group before feeding EfficientNet (expects 3 channels)
group_0 = out[:, 0:3, :, :]   # (N_windows, 3, 64, 64)
group_1 = out[:, 3:6, :, :]

If len(metric_cols) is not divisible by 3, the last metric is replicated to fill the final group. Caller controls metric ordering — group semantics are deterministic.

---

Roadmap

Phase	Status	Description
1 — Ingestion	✅	CloudWatch + Cost Explorer → S3 Parquet
2 — GAF Transform	✅	Sliding window + Gramian Angular Field → .npy
3 — Feature Extraction	🔲	EfficientNet-B4 frozen + per-patch Gaussian fitting
4 — Inference	🔲	Mahalanobis distance → anomaly score + heatmap

---

Author

Rafael Santiago Silva — Cloud & MLOps Engineer
github.com/tommmdl