BibleBrain Services

A serverless application providing API services for BibleBrain, primarily focused on copyright management and PDF generation.

Overview

This project is built with Go and deployed on AWS Lambda using the Serverless Framework. It provides API endpoints for copyright information retrieval and PDF document generation based on product codes.

Prerequisites

• Go 1.24+
• Node.js and npm
• AWS CLI configured with appropriate permissions
• Docker (for local development)
• MySQL database access

Development Setup

This project uses a development container with all necessary tools pre-installed:

1. Clone the repository:

   git clone <repository-url>
   cd biblebrain-services

2. Configure environment variables:

   cp .devcontainer/.env.template .devcontainer/.env
   # Edit .env file with your configuration

3. Start the development container:

   # Using VS Code
   # Open the project in Visual Studio Code and click "Reopen in Container"
   
   # Or use Docker Compose directly
   docker-compose -f .devcontainer/docker-compose.yml up -d

4. Install dependencies:

   go mod tidy
   npm install

LocalStack Setup

This project uses LocalStack to emulate AWS services (SSM, S3) for local development.

Starting LocalStack

LocalStack is automatically started when you run make offline, but you can manage it manually:

# Start LocalStack but first, check if it is running
make localstack-start

# Check LocalStack status
make localstack-status

# Stop LocalStack
make localstack-stop

Initialize LocalStack Services

Although localstack-start populates the SSM parameters when LocalStack is not running, to manually populate the SSM parameters, run:

# Populate SSM parameters for local development
make ssmlocal

Create S3 Bucket in LocalStack

To create the my-local-bucket for local S3 operations:

# Using awslocal (if installed)
awslocal s3 mb s3://my-local-bucket

# Or using aws CLI with endpoint
aws --endpoint-url=http://localhost:4566 s3 mb s3://my-local-bucket

Update Environment Configuration

Ensure your .devcontainer/.env file points to LocalStack for S3:

# S3 endpoint should point to LocalStack
AWS_SERVERLESS_S3_LOCAL_HOST=http://localhost:4566

# LocalStack endpoint
LOCALSTACK_ENDPOINT=http://localhost:4566

Available Commands

The project uses a Makefile for common operations:

Build Commands

# Build all services
make build-all

# Build individual services
make build-copyright
make build-storage
make build-stitch
make build-stream
make build-service

# Legacy build command (builds all services)
make build

# Clean build artifacts
make clean

Development Commands

# Run serverless offline for local testing
make offline

# Format and standardize code
make standardize

# Run pre-commit checks (formatting and linting)
make precommit

Video Transcoding Commands

# Build the transcoder Docker image
make build-transcoder-image

# Run the transcoder interactively
make execute-transcoder

⚠️ IMPORTANT: Transcoder Script Requirements

Before running /workspace/scripts/run_transcoder.sh or make execute-transcoder, ensure:

1. LocalStack is running: Start with make localstack-start
2. SSM parameters are populated: Run make ssmlocal to populate SSM with database connection strings
3. Environment variable BIBLEBRAIN_DSN_SSM_ID is set in .devcontainer/.env

The script uses the environment file .devcontainer/.env and requires LocalStack services to be accessible.

Deployment Commands

# Deploy all services
make deploy-all

# Deploy individual services
make deploy-copyright
make deploy-storage
make deploy-stitch
make deploy-stream
make deploy-service

# Legacy deploy command (deploys all services)
make deploy

API Endpoints

Status Endpoint

• Path: /api/status
• Method: GET
• Parameters: name (query string)
• Description: Returns a simple status message to verify the service is running

Copyright Creation Endpoint

• Path: /api/copyright/create
• Method: GET
• Query Parameter:
  • productList: A comma-separated list of product codes (e.g. P1PUI/LAN,N2ENG/NIV)
  • format: json or pdf
  • mode: audio, video or text
• Response: PDF document containing copyright information
• Content-Type: application/pdf or application/json

Environment Configuration

The application uses environment variables for configuration:

Variable	Description	Default	Stage
LOG_LEVEL	Logging level (Debug, Error, Info)	Debug	All
BIBLEBRAIN_DSN	Database connection string for local development	-	Local
BIBLEBRAIN_DSN_SSM_ID	SSM parameter ID for database connection in AWS	-	Dev/Prod
CDN_SERVER	CDN server URL for video streaming	'' (empty)	All
CDN_SIGNING_KEY_SSM_ID_KEYPAIRID	CDN signing key pair ID	local-keypair-id	Local
CDN_SIGNING_KEY_SSM_ID_PRIVATEKEY	CDN signing private key path	./cloudfront-private-key.pem	Local
environment	Deployment environment (local, dev, prod)	local	Deployment

Stage-Specific Configuration

The serverless.yml file configures environment variables differently per stage:

• Local: Uses environment variables or defaults
• Dev/Prod: Uses AWS SSM Parameter Store for secrets and database connections

Local Development with Serverless Offline

This project uses serverless-offline with Docker to simulate AWS Lambda locally. All services run on a single port (http://localhost:3009) with API Gateway routing.

Starting Serverless Offline

make offline

This command:

1. Builds all service binaries (build/bin/copyright, build/bin/storage, etc.)
2. Copies each binary to build/local/{service}/bootstrap for Docker mounting
3. Verifies the root bootstrap wrapper script exists and is executable
4. Starts serverless-offline with Docker containers

Bootstrap Wrapper Script

The root bootstrap script is a wrapper for local development only. It's used by serverless-offline to route Lambda invocations to the correct service binary.

How it works:

• AWS Lambda sets AWS_LAMBDA_FUNCTION_NAME environment variable
• The wrapper script reads this variable to determine which function is being invoked
• It maps function names to their corresponding binaries:
  • biblebrain-services-local-service → /var/task/build/local/service/bootstrap
  • biblebrain-services-local-copyright → /var/task/build/local/copyright/bootstrap
  • biblebrain-services-local-storage → /var/task/build/local/storage/bootstrap
  • biblebrain-services-local-stitch → /var/task/build/local/stitch/bootstrap
  • biblebrain-services-local-stream → /var/task/build/local/stream/bootstrap
• Executes the appropriate binary with all arguments passed through

Important: This wrapper script is only used for local development. In AWS Lambda (dev/prod), each function has its own bootstrap binary created by the serverless-go-plugin during deployment.

Available Endpoints (Local)

When running make offline, all endpoints are available at http://0.0.0.0:3009:

• GET /api/status - Service health check
• GET /api/copyright - Copyright information
• GET /api/copyright/dependency/{license_group_id} - Copyright dependencies
• POST /api/storage/parse - Parse storage data
• GET /api/storage/inventory - Storage inventory
• POST /api/media/stitch - Stitch media files
• GET /api/media/stream/playlist/{fileset_id}/{book}/{chapter}/master.m3u8 - Stream playlist
• GET /api/media/stream/playlist/{fileset_id}/{book}/{chapter}/{file_name} - Stream file
• GET /api/media/stream/playlist-auto/{fileset_id}/{book}/{chapter} - Auto playlist

Deployment

The application uses a microservices architecture with independent function deployments:

Deploy Individual Functions

Each function can be deployed independently for faster iteration:

# Set environment variable (required)
export environment=dev  # or prod

# Deploy individual services
make deploy-copyright   # Copyright service only
make deploy-storage     # Storage service only
make deploy-stitch      # Stitch service only
make deploy-stream      # Stream service only
make deploy-service     # Status service only

Benefits of individual deployment:

• Faster deployment times (only builds and deploys one function)
• Reduced risk when making changes to a single service
• Independent release cycles for each microservice

Deploy All Functions

Deploy all services at once:

# Using Makefile
export environment=dev  # or prod
make deploy-all

# Or using Serverless Framework directly
sls deploy --stage dev
sls deploy --stage prod

Deployment Process

Using Makefile (Local/Manual Deployments):

When deploying via Makefile commands, the process is:

1. Cleans previous build artifacts (make clean)
2. Builds the specific Go binary with Lambda tags
3. Runs sls deploy which invokes serverless-go-plugin
4. serverless-go-plugin creates a bootstrap binary
5. Packages and uploads to AWS Lambda
6. Updates API Gateway routes and permissions

Using CI/CD (AWS CodeBuild):

The buildspec files (buildspecs/buildspec-*.yml) use an optimized single-compilation approach:

1. Installs dependencies (Go, Node.js, Serverless Framework)
2. Runs go mod tidy to ensure dependencies are ready
3. Directly runs sls deploy - skips Makefile build
4. serverless-go-plugin automatically compiles the Go binary during deployment
5. Packages and deploys to AWS Lambda

Why CI/CD skips make build-* commands:

• ✅ Avoids duplicate compilation - Previously, make build-{service} would compile, then sls deploy would compile again via serverless-go-plugin
• ✅ Smaller binaries - serverless-go-plugin uses -ldflags="-s -w" to strip debug symbols
• ✅ Simpler pipeline - Let serverless-go-plugin handle what it's designed to do

When Makefile builds ARE used:

• Local development with make offline (needs pre-built binaries for Docker)
• Local testing and validation
• Manual deployments via make deploy-* commands

Testing

Run tests with the standard Go test command:

go test ./...

Integration tests for the copyright service are available in copyright_test.go.

Project Structure

.
├── build/                 # Build artifacts
│   ├── bin/              # Compiled service binaries
│   │   ├── copyright     # Copyright service binary
│   │   ├── storage       # Storage service binary
│   │   ├── stitch        # Stitch service binary
│   │   ├── stream        # Stream service binary
│   │   └── service       # Status service binary
│   └── local/            # Local development binaries
│       ├── copyright/    # Copyright bootstrap for local
│       ├── storage/      # Storage bootstrap for local
│       ├── stitch/       # Stitch bootstrap for local
│       ├── stream/       # Stream bootstrap for local
│       └── service/      # Service bootstrap for local
├── cmd/                   # Command entry points (microservices)
│   ├── copyright/        # Copyright service entry point
│   │   └── main.go
│   ├── storage/          # Storage service entry point
│   │   └── main.go
│   ├── stitch/           # Stitch service entry point
│   │   └── main.go
│   ├── stream/           # Stream service entry point
│   │   └── main.go
│   └── service/          # Status service entry point
│       └── main.go
├── service/               # Business logic services
│   ├── connection/        # Database connection handling
│   ├── copyright/         # Copyright service implementation
│   ├── pdf/               # PDF generation utilities
│   └── sign/              # AWS signature utilities
├── util/                  # Utility functions
├── buildspecs/            # AWS CodeBuild CI/CD configurations
│   ├── buildspec-copyright.yml   # CI/CD for copyright service
│   ├── buildspec-service.yml     # CI/CD for status service
│   ├── buildspec-storage.yml     # CI/CD for storage service
│   ├── buildspec-stitch.yml      # CI/CD for stitch service
│   ├── buildspec-stream.yml      # CI/CD for stream service
│   ├── buildspec-all.yml         # CI/CD for all services + transcoder
│   └── buildspec-transcode.yml   # CI/CD for transcoder Docker image
├── .devcontainer/         # Development container configuration
├── bootstrap              # Wrapper script (local development only)
├── go.mod                 # Go module definition
├── go.sum                 # Go module checksums
├── Makefile               # Build automation
├── package.json           # Node.js dependencies
└── serverless.yml         # Serverless Framework configuration

Microservices Architecture

The project is organized into independent microservices:

• service: Health check and status endpoint (no external dependencies)
• copyright: Copyright management with database access
• storage: Storage parsing and inventory with database + S3 access
• stitch: Media stitching service with S3 access (long timeout)
• stream: Video streaming with database + CDN signing

Each service:

• Has its own entry point in cmd/{service}/main.go
• Can be built independently
• Can be deployed independently
• Has its own AWS Lambda function configuration in serverless.yml

CI/CD Pipeline

The project uses AWS CodeBuild with individual buildspec files for each service:

• buildspec-{service}.yml: Deploys a single microservice (copyright, storage, stitch, stream, service)
• buildspec-all.yml: Deploys all Lambda functions + transcoder Docker image
• buildspec-transcode.yml: Builds and pushes transcoder Docker image to ECR

Key optimization: Buildspec files use serverless-go-plugin for compilation during sls deploy, avoiding duplicate builds.