|
| 1 | +# Pulumi Examples Repository - Development Guide |
| 2 | + |
| 3 | +This repository contains hundreds of Infrastructure as Code (IaC) examples using Pulumi across multiple cloud providers and programming languages. |
| 4 | + |
| 5 | +## Repository Architecture |
| 6 | + |
| 7 | +### High-Level Structure |
| 8 | +This is a monorepo containing **300+ examples** organized by cloud provider and programming language: |
| 9 | + |
| 10 | +``` |
| 11 | +{cloud}-{language}-{example-type}/ |
| 12 | +├── aws-{go|py|ts|js|cs|fs}-{feature}/ |
| 13 | +├── azure-{go|py|ts|js|cs|fs}-{feature}/ |
| 14 | +├── gcp-{go|py|ts|js|cs|fs}-{feature}/ |
| 15 | +├── kubernetes-{go|py|ts|js|cs|fs}-{feature}/ |
| 16 | +└── testing-{integration|unit}-{language}/ |
| 17 | +``` |
| 18 | + |
| 19 | +**Supported Languages:** |
| 20 | +- **TypeScript/JavaScript** (`ts`/`js`) - Most comprehensive coverage |
| 21 | +- **Python** (`py`) - Second-most coverage |
| 22 | +- **Go** (`go`) - Growing coverage, especially for AWS |
| 23 | +- **C#** (`cs`) - .NET examples |
| 24 | +- **F#** (`fs`) - Functional .NET examples |
| 25 | + |
| 26 | +**Cloud Providers:** |
| 27 | +- AWS (largest coverage) |
| 28 | +- Azure (comprehensive) |
| 29 | +- Google Cloud Platform (GCP) |
| 30 | +- Kubernetes |
| 31 | +- DigitalOcean, Equinix Metal, Linode, OVHCloud |
| 32 | + |
| 33 | +### Project Structure Patterns |
| 34 | + |
| 35 | +Each example follows a consistent structure: |
| 36 | +``` |
| 37 | +example-name/ |
| 38 | +├── Pulumi.yaml # Project definition |
| 39 | +├── main.{go|py|ts|js} # Infrastructure code |
| 40 | +├── go.mod # Go modules (Go projects) |
| 41 | +├── package.json # Node dependencies (TS/JS) |
| 42 | +├── requirements.txt # Python dependencies |
| 43 | +├── README.md # Usage documentation |
| 44 | +├── app/ # Application code (if containerized) |
| 45 | +└── .gitignore |
| 46 | +``` |
| 47 | + |
| 48 | +### Core Architectural Patterns |
| 49 | + |
| 50 | +1. **Infrastructure Components**: Reusable modules for common patterns |
| 51 | + - VPC/Network setup with public/private subnets |
| 52 | + - Load balancers with target groups |
| 53 | + - ECS/Fargate containers with ECR repositories |
| 54 | + - IAM roles and policies |
| 55 | + - Security groups with ingress/egress rules |
| 56 | + |
| 57 | +2. **Multi-Tier Applications**: Complete application stacks |
| 58 | + - Web servers with databases |
| 59 | + - Container orchestration (ECS, EKS, AKS, GKE) |
| 60 | + - Serverless functions with API gateways |
| 61 | + - Static websites with CDNs |
| 62 | + |
| 63 | +3. **Testing Patterns**: Comprehensive testing strategies |
| 64 | + - Unit tests with mocking |
| 65 | + - Integration tests with real deployments |
| 66 | + - Policy-as-Code validation |
| 67 | + |
| 68 | +## Development Commands |
| 69 | + |
| 70 | +### Essential Commands for Daily Development |
| 71 | + |
| 72 | +#### Working with Individual Examples |
| 73 | +```bash |
| 74 | +# Navigate to an example and preview changes |
| 75 | +cd aws-typescript-eks && pulumi preview |
| 76 | + |
| 77 | +# Deploy an example |
| 78 | +cd aws-typescript-eks && pulumi up |
| 79 | + |
| 80 | +# Clean up resources |
| 81 | +cd aws-typescript-eks && pulumi destroy |
| 82 | + |
| 83 | +# Check stack outputs |
| 84 | +cd aws-typescript-eks && pulumi stack output |
| 85 | +``` |
| 86 | + |
| 87 | +#### Building and Testing |
| 88 | +```bash |
| 89 | +# Quick validation of code formatting and linting |
| 90 | +make format && make lint |
| 91 | + |
| 92 | +# Run all integration tests (4-hour timeout) |
| 93 | +make only_test |
| 94 | + |
| 95 | +# Run tests for specific cloud/language combination |
| 96 | +make specific_test_set TestSet=AwsGo |
| 97 | + |
| 98 | +# Run tests with specific tags |
| 99 | +make specific_tag_set TagSet=aws TestSet=Go |
| 100 | + |
| 101 | +# Test a single example |
| 102 | +make test_example.TestAccAwsPyS3Folder |
| 103 | + |
| 104 | +# Preview changes in PR mode (tests only changed examples) |
| 105 | +make pr_preview |
| 106 | +``` |
| 107 | + |
| 108 | +#### Language-Specific Linting and Formatting |
| 109 | +```bash |
| 110 | +# Repository-wide linting and formatting |
| 111 | +make lint # Run all linting across languages |
| 112 | +make format # Format all code (Python with Black) |
| 113 | + |
| 114 | +# Python-specific |
| 115 | +make check_python_formatting # Validate Python formatting |
| 116 | +make setup_python # Setup Python virtual environment |
| 117 | + |
| 118 | +# TypeScript linting (repository-wide) |
| 119 | +tslint -c tslint.json **/*.ts |
| 120 | + |
| 121 | +# Go formatting (per-project) |
| 122 | +cd aws-go-fargate && go fmt ./... |
| 123 | +cd aws-go-fargate && go mod tidy |
| 124 | + |
| 125 | +# C#/.NET (per-project) |
| 126 | +cd aws-cs-webserver && dotnet format |
| 127 | +``` |
| 128 | + |
| 129 | +#### Testing Infrastructure |
| 130 | +```bash |
| 131 | +# Setup test dependencies |
| 132 | +make ensure |
| 133 | + |
| 134 | +# Go unit tests (specific project) |
| 135 | +cd testing-unit-go && go test |
| 136 | + |
| 137 | +# Python unit tests |
| 138 | +cd testing-unit-py && python -m pytest |
| 139 | + |
| 140 | +# TypeScript unit tests |
| 141 | +cd testing-unit-ts/mocha && npm test |
| 142 | +``` |
| 143 | + |
| 144 | +### CI/CD Workflows |
| 145 | + |
| 146 | +The repository uses GitHub Actions with several specialized workflows: |
| 147 | + |
| 148 | +#### Main Test Workflow (`test-examples.yml`) |
| 149 | +- **TypeScript Linting**: `tslint` validation across all TS files |
| 150 | +- **Unit Tests**: Language-specific unit test execution |
| 151 | +- **Python Formatting**: Black code formatting validation |
| 152 | +- **Integration Tests**: Matrix testing across cloud providers and languages |
| 153 | + - Platforms: AWS, Azure, GCP, DigitalOcean, Equinix Metal |
| 154 | + - Languages: Go, Python, TypeScript, JavaScript, C#, F# |
| 155 | +- **Kubernetes Tests**: Specialized K8s testing with Minikube |
| 156 | + |
| 157 | +#### PR Preview Workflow |
| 158 | +- Detects changed examples vs base branch |
| 159 | +- Automatically seeds missing configuration with safe placeholders |
| 160 | +- Runs `pulumi preview` on changed examples only |
| 161 | +- Supports multiple runtime detection (Node.js, Python, Go, .NET, YAML) |
| 162 | + |
| 163 | +### Testing Architecture |
| 164 | + |
| 165 | +#### Integration Testing Framework |
| 166 | +Located in `misc/test/`, the testing system: |
| 167 | + |
| 168 | +1. **Test Definitions**: Structured test configuration in `definitions/` |
| 169 | + - Tagged by language and cloud provider |
| 170 | + - Uses Pulumi's `integration.ProgramTestOptions` |
| 171 | + - Supports parallel execution (40 parallel tests) |
| 172 | + |
| 173 | +2. **Test Execution**: |
| 174 | + - Go-based test runner with `gotestfmt` formatting |
| 175 | + - 4-hour timeout for full test suite |
| 176 | + - Comprehensive cloud provider authentication |
| 177 | + |
| 178 | +3. **Test Categories**: |
| 179 | + - **Unit Tests**: Mock-based testing with `pulumi.WithMocks()` |
| 180 | + - **Integration Tests**: Deploy-check-destroy lifecycle |
| 181 | + - **Policy Tests**: Policy-as-Code validation |
| 182 | + |
| 183 | +#### Example Unit Test Pattern (Go) |
| 184 | +```go |
| 185 | +// Mock resource creation |
| 186 | +func (mocks) NewResource(args pulumi.MockResourceArgs) (string, resource.PropertyMap, error) { |
| 187 | + outputs := args.Inputs.Mappable() |
| 188 | + if args.TypeToken == "aws:ec2/instance:Instance" { |
| 189 | + outputs["publicIp"] = "203.0.113.12" |
| 190 | + } |
| 191 | + return args.Name + "_id", resource.NewPropertyMapFromMap(outputs), nil |
| 192 | +} |
| 193 | + |
| 194 | +// Test infrastructure properties |
| 195 | +func TestInfrastructure(t *testing.T) { |
| 196 | + err := pulumi.RunErr(func(ctx *pulumi.Context) error { |
| 197 | + infra, err := createInfrastructure(ctx) |
| 198 | + // Assert expected properties |
| 199 | + pulumi.All(infra.server.Tags).ApplyT(func(tags interface{}) error { |
| 200 | + assert.Contains(t, tags, "Name") |
| 201 | + return nil |
| 202 | + }) |
| 203 | + return err |
| 204 | + }, pulumi.WithMocks("project", "stack", mocks(0))) |
| 205 | + assert.NoError(t, err) |
| 206 | +} |
| 207 | +``` |
| 208 | + |
| 209 | +### Development Workflow |
| 210 | + |
| 211 | +#### Before Working on Examples |
| 212 | +1. **Prerequisites Setup**: |
| 213 | + - Configure cloud provider credentials (AWS CLI, Azure CLI, gcloud) |
| 214 | + - Install language-specific tools (Node.js, Python, Go, .NET) |
| 215 | + - Run `pulumi login` to authenticate with Pulumi |
| 216 | + |
| 217 | +2. **Working with Existing Examples**: |
| 218 | + - Navigate to example directory: `cd aws-typescript-eks` |
| 219 | + - Install dependencies: `npm install` (or language equivalent) |
| 220 | + - Configure stack: `pulumi config set aws:region us-west-2` |
| 221 | + - Preview: `pulumi preview` |
| 222 | + - Deploy: `pulumi up` |
| 223 | + - Clean up: `pulumi destroy` |
| 224 | + |
| 225 | +#### Adding New Examples |
| 226 | +1. **Naming and Structure**: |
| 227 | + - Follow `{cloud}-{language}-{feature}` naming convention |
| 228 | + - Create directory with standard structure (see Project Structure Patterns above) |
| 229 | + - Include comprehensive README with deployment steps |
| 230 | + |
| 231 | +2. **Testing Integration**: |
| 232 | + - Add test definitions to `misc/test/definitions/` |
| 233 | + - Tag appropriately for CI matrix execution |
| 234 | + - Test locally before submitting PR |
| 235 | + |
| 236 | +3. **Code Quality**: |
| 237 | + - Run language-specific linting and formatting |
| 238 | + - Ensure example works with `pulumi preview` and `pulumi up` |
| 239 | + - Include proper resource tagging and cleanup |
| 240 | + |
| 241 | +#### PR Workflow |
| 242 | +- Run `make pr_preview` to test only changed examples |
| 243 | +- Automatic linting and formatting validation |
| 244 | +- Integration tests run on maintainer approval |
| 245 | +- Use `make format && make lint` before committing |
| 246 | + |
| 247 | +## Key Dependencies and Tools |
| 248 | + |
| 249 | +### Core Infrastructure |
| 250 | +- **Pulumi SDK**: v3.x across all languages |
| 251 | +- **Cloud Provider SDKs**: AWS SDK v7, Azure SDK, GCP SDK |
| 252 | +- **Docker**: For containerized examples |
| 253 | + |
| 254 | +### Testing and CI |
| 255 | +- **gotestfmt**: Go test output formatting |
| 256 | +- **pytest**: Python testing framework |
| 257 | +- **Mocha**: TypeScript/JavaScript testing |
| 258 | +- **tslint**: TypeScript linting |
| 259 | +- **Black**: Python code formatting |
| 260 | + |
| 261 | +### Development Tools |
| 262 | +- **Node.js 20**: TypeScript/JavaScript runtime |
| 263 | +- **Python 3.9+**: Python runtime |
| 264 | +- **Go 1.21+**: Go runtime |
| 265 | +- **Helm**: Kubernetes package management |
| 266 | +- **kubectl**: Kubernetes CLI |
| 267 | + |
| 268 | +## Common Development Issues |
| 269 | + |
| 270 | +### Authentication Problems |
| 271 | +- **Cloud Provider Credentials**: Ensure AWS CLI, Azure CLI, or gcloud is configured |
| 272 | +- **Pulumi Backend**: Run `pulumi login` to authenticate with Pulumi service |
| 273 | +- **Permission Errors**: Check IAM roles have sufficient permissions for resource creation |
| 274 | + |
| 275 | +### Resource Conflicts |
| 276 | +- **Stack State**: Use unique stack names to avoid conflicts: `pulumi stack select dev-yourname` |
| 277 | +- **Resource Names**: Many examples use randomized names to avoid conflicts |
| 278 | +- **Region Conflicts**: Configure appropriate regions: `pulumi config set aws:region us-west-2` |
| 279 | + |
| 280 | +### Testing Issues |
| 281 | +- **Long Test Times**: Full integration tests take up to 4 hours - use `make pr_preview` for quick validation |
| 282 | +- **Parallel Test Failures**: Tests run 40 in parallel - occasional timeouts are expected |
| 283 | +- **Resource Cleanup**: Always run `pulumi destroy` after testing to avoid costs |
| 284 | + |
| 285 | +## Architecture Insights |
| 286 | + |
| 287 | +This repository demonstrates several key IaC architectural patterns: |
| 288 | + |
| 289 | +1. **Multi-Cloud Abstraction**: Examples show similar patterns across cloud providers |
| 290 | +2. **Language Polyglot**: Same infrastructure patterns implemented across multiple programming languages |
| 291 | +3. **Component Reusability**: Higher-level components built from cloud primitives |
| 292 | +4. **Testing Strategy**: Comprehensive unit, integration, and policy testing |
| 293 | +5. **CI/CD Integration**: Automated testing and validation at scale |
| 294 | + |
| 295 | +The testing infrastructure alone processes hundreds of examples across multiple cloud providers, languages, and test types, making it one of the most comprehensive IaC testing frameworks available. |
0 commit comments