Tech debt: updating the project structure (#102)

* Tech debt changes: project file structure, documentation, getting rid of obsolete files

Author: tmikula-dev

.github/workflows/aquasec_repo_scan.yml

@@ -3,7 +3,7 @@ name: AquaSec Full Repository Scan
 on:
   workflow_dispatch:
   pull_request:
-    types: [ opened, synchronize ]
+    types: [ opened ]
 
 permissions:
   contents: read

.github/workflows/check_python.yml

@@ -31,13 +31,20 @@ jobs:
       - name: Check if Python files changed
         id: changes
         shell: bash
+        env:
+          GH_TOKEN: ${{ github.token }}
         run: |
+          set -euo pipefail
+          
           if [[ "${{ github.event_name }}" == "pull_request" ]]; then
-            RANGE="${{ github.event.pull_request.base.sha }}...${{ github.sha }}"
+            CHANGED_FILES=$(gh api \
+              "repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files" \
+              --jq '.[].filename | select(endswith(".py"))')
           else
-            RANGE="${{ github.sha }}~1...${{ github.sha }}"
+            CHANGED_FILES=$(git diff --name-only "${{ github.sha }}~1" "${{ github.sha }}" -- '*.py')
           fi
-          if git diff --name-only "$RANGE" -- '*.py' | grep -q .; then
+
+          if [[ -n "$CHANGED_FILES" ]]; then
             echo "python_changed=true" >> "$GITHUB_OUTPUT"
           else
             echo "python_changed=false" >> "$GITHUB_OUTPUT"

.github/workflows/check_terraform.yml

@@ -5,10 +5,5 @@ __pycache__
 /dependencies
 /lambda_function.zip
 
-# Terraform files
-/terraform/*.tfvars
-/terraform/*.tfstate*
-/terraform/.terraform*
-
 # Terraform Plan output files
 *.sarif

.gitignore

@@ -73,6 +73,7 @@ RUN \
 COPY $SASL_SSL_ARTIFACTS /opt/sasl_ssl_artifacts/
 COPY src $LAMBDA_TASK_ROOT/src
 COPY conf $LAMBDA_TASK_ROOT/conf
+COPY api.yaml $LAMBDA_TASK_ROOT/api.yaml
 
 # Mark librdkafka to LD_LIBRARY_PATH  
 # Kerberos default CCACHE override due to KEYRING issues

Dockerfile

@@ -19,7 +19,6 @@ Python AWS Lambda that exposes a simple HTTP API (via API Gateway) for validatin
   - [Kafka Writer](#kafka-writer)
   - [EventBridge Writer](#eventbridge-writer)
   - [Postgres Writer](#postgres-writer)
-- [Scripts](#scripts)
 - [Troubleshooting](#troubleshooting)
 - [License](#license)
 <!-- tocstop -->
@@ -34,7 +33,7 @@ EventGate receives JSON payloads for registered topics, authorizes the caller vi
 - Runtime-configurable access rules (local or S3)
 - API-discoverable schema catalogue
 - Pluggable writer initialization via `config.json`
-- Terraform IaC examples for AWS deployment (API Gateway + Lambda)
+- Terraform IaC examples for AWS deployment (API Gateway + Lambda) in `terraform_examples/`
 - Supports both Zip-based and Container Image Lambda packaging (Container path enables custom `librdkafka` / SASL_SSL / Kerberos builds)
 
 ## Architecture
@@ -47,8 +46,9 @@ High-level flow:
 
 Key files:
 - `src/event_gate_lambda.py` – main Lambda handler and routing
-- `conf/*.json` – configuration and topic schemas
-- `conf/api.yaml` – OpenAPI 3 definition served at `/api`
+- `conf/*.json` – configuration files
+- `conf/topic_schemas/` – JSON topic schemas
+- `api.yaml` – OpenAPI 3 definition served at `/api`
 - `writer_*.py` – individual sink implementations
 
 ## API
@@ -58,18 +58,21 @@ All responses are JSON unless otherwise noted. The POST endpoint requires a vali
 |--------|----------|------|-------------|
 | GET | `/api` | none | Returns OpenAPI 3 definition (raw YAML) |
 | GET | `/token` | none | 303 redirect to external token provider |
+| GET | `/health` | none | Returns service health and dependency status |
 | GET | `/topics` | none | Lists available topic names |
 | GET | `/topics/{topicName}` | none | Returns JSON Schema for the topic |
 | POST | `/topics/{topicName}` | JWT | Validates + forwards message to configured sinks |
 | POST | `/terminate` | (internal) | Forces Lambda process exit (used to trigger cold start & config reload) |
 
 Status codes:
+- 200 – Health check pass
 - 202 – Accepted (all writers succeeded)
 - 400 – Schema validation failure
 - 401 – Token missing/invalid
 - 403 – Subject unauthorized for topic
 - 404 – Unknown topic or route
 - 500 – One or more writers failed / internal error
+- 503 – Service degraded (dependency not initialized)
 
 ## Configuration
 All core runtime configuration is driven by JSON files located in `conf/` unless S3 paths are specified.
@@ -100,14 +103,13 @@ Configuration keys:
 
 Supporting configs:
 - `access.json` – map: topicName -> array of authorized subjects (JWT `sub`). May reside locally or at S3 path referenced by `access_config`.
-- `topic_*.json` – each file contains a JSON Schema for a topic. In the current code these are explicitly loaded inside `event_gate_lambda.py`. (Future enhancement: auto-discover or index file.)
-- `api.yaml` – OpenAPI spec served verbatim at runtime.
+- `topic_schemas/*.json` – each file contains a JSON Schema for a topic. In the current code these are explicitly loaded inside `event_gate_lambda.py`. (Future enhancement: auto-discover or index file.)
 
 Environment variables:
 - `LOG_LEVEL` (optional) – defaults to `INFO`.
 
 ## Deployment
-Infrastructure-as-Code examples live in the `terraform/` directory. Variables are supplied via a `*.tfvars` file or CLI.
+Infrastructure-as-Code examples are provided in `terraform_examples/`. These are reference implementations that you can adapt to your environment. Variables are supplied via a `*.tfvars` file or CLI.
 
 ### Zip Lambda Package
 Use when no custom native libraries are needed.
@@ -169,18 +171,14 @@ Publishes events to the configured `event_bus_arn` using put events API.
 ### Postgres Writer
 Example writer (currently a placeholder if no DSN present) demonstrating extensibility pattern.
 
-## Scripts
-- `scripts/prepare.deplyoment.sh` – build Zip artifact for Lambda (typo in name retained for now; may rename later)
-- `scripts/notebook.ipynb` – exploratory invocation cells per endpoint
-- `scripts/get_token.http` – sample HTTP request for tooling (e.g., VSCode REST client)
-
 ## Troubleshooting
 | Symptom | Possible Cause | Action |
 |---------|----------------|--------|
 | 401 Unauthorized | Missing / malformed token header | Ensure `Authorization: Bearer` present |
 | 403 Forbidden | Subject not listed in access map | Update `access.json` and redeploy / reload |
 | 404 Topic not found | Wrong casing or not loaded in code | Verify loaded topics & file names |
 | 500 Writer failure | Downstream (Kafka / EventBridge / DB) unreachable | Check network / VPC endpoints / security groups |
+| 503 Service degraded | Dependency not initialized | Check `/health` response for specific failure |
 | Lambda keeps old config | Warm container | Call `/terminate` (internal) to force cold start |
 
 ## License