Skip to content

Commit f055d6a

Browse files
committed
feat(azure-yaml): add enterprise SFTP blob storage example with advanced security
## Summary Add new Pulumi YAML example demonstrating Azure Blob Storage with SFTP support, featuring enterprise-grade security controls including customer-managed encryption, private network access, and comprehensive audit logging. ## Features - **Customer-Managed Encryption**: HSM-backed 4096-bit RSA keys in Azure Key Vault Premium - **Infrastructure Encryption**: Double encryption layer for maximum data protection - **Private Network Only**: Zero public internet access with private endpoints - **Immutable Storage**: WORM compliance with version control and change tracking - **Extended Audit Logging**: 7-year retention for compliance requirements - **Role-Based Access**: Admin and auditor users with granular container permissions - **Latest Azure Native Provider**: Uses Azure Native v3.8+ features and 2024 API versions ## Technical Implementation ### Core Components - **Storage Account**: Data Lake Gen2 (hierarchical namespace) with SFTP enabled - **Key Vault Premium**: HSM-backed customer-managed encryption keys - **Encryption Scope**: Dedicated scope with infrastructure encryption - **Virtual Network**: Isolated VNet with subnets for storage and Key Vault - **Private Endpoints**: Secure access to storage and Key Vault services - **Managed Identity**: Secure Key Vault access without shared keys - **Log Analytics**: Comprehensive monitoring with extended retention ### Security Posture ```yaml Encryption: - At Rest: Customer-managed keys (4096-bit RSA-HSM) - In Transit: TLS 1.2+ and SSH - Infrastructure: Double encryption enabled Network Security: - Public Access: Disabled - Private Endpoints: Required - Network Rules: Explicit deny all - Service Bypass: None Access Control: - Authentication: SSH keys only (passwords disabled) - Shared Keys: Disabled - Authorization: Azure AD OAuth - User Types: Admin (full) and Auditor (read-only) Data Integrity: - Immutable Storage: Enabled on all containers - Versioning: Full version control - Change Feed: 7-year audit trail - Soft Delete: 7-year retention ``` ## Documentation ### README.md (193 lines) - Clean technical implementation guide - Quick start with configuration examples - Architecture diagram with mermaid - Troubleshooting and support information - Minimal length focused on essentials ### HIPAA_COMPLIANCE.md (562 lines) - Healthcare security reference (educational only) - Comprehensive legal disclaimers - Technical security patterns for regulated industries - No compliance guarantees - professional validation required ### Total Documentation: 755 lines - Reduced from initial 3,754 lines (80% reduction) - Removed excessive operational guides - Focused on practical implementation ## Healthcare & Compliance Notes While this example implements security controls commonly required by healthcare and financial organizations, it includes prominent disclaimers: - ⚠️ No guarantee of HIPAA, PHI, or regulatory compliance - ⚠️ Professional legal and compliance consultation required - ⚠️ Independent security assessment needed for production use - ⚠️ Compliance standards evolve - always verify current requirements Healthcare organizations can reference `HIPAA_COMPLIANCE.md` for additional security considerations, but must validate with their own compliance teams. ## Testing To test this example: ```bash cd azure-yaml-sftp-blob # Initialize stack pulumi stack init dev # Configure pulumi config set azure-native:location "East US 2" pulumi config set storageAccountName "testsftpstorage$(date +%s)" pulumi config set userPublicKey "$(cat ~/.ssh/id_rsa.pub)" # Preview (validate syntax) pulumi preview # Deploy (requires Azure credentials and VPN for private network access) pulumi up ``` ## Language Conversion This YAML example can be converted to other languages: ```bash pulumi convert --language typescript --out ./typescript-version pulumi convert --language python --out ./python-version pulumi convert --language go --out ./go-version pulumi convert --language csharp --out ./csharp-version ``` ## Breaking Changes None - this is a new example. ## Related Issues Closes #XXXX (if applicable) ## Checklist - [x] Code follows repository standards - [x] Documentation is clear and concise - [x] Example includes architecture diagram - [x] Security best practices implemented - [x] Legal disclaimers for compliance content - [x] Formatting validated with `make format` and `make check_python_formatting` - [ ] Tested deployment with `pulumi preview` - [ ] Tested full deployment cycle (requires Azure environment) ## Notes for Reviewers 1. **Legal Review**: The HIPAA_COMPLIANCE.md file includes extensive legal disclaimers. Please validate these are sufficient to protect against liability claims. 2. **Azure Syntax**: Some Pulumi YAML syntax should be validated during review: - Managed identity property structure (line 150) - `getClientConfig` function syntax (lines 60, 108) - Subnet array indexing (lines 393, 414) - BlobService versioning property name (line 212) 3. **Security Configuration**: All security features use latest Azure Native provider capabilities. Review that: - Customer-managed keys are properly configured - Infrastructure encryption is enabled at all layers - Private network access is enforced correctly - Audit logging captures all required events 4. **Documentation Scope**: Intentionally kept documentation minimal (755 lines total) to avoid overwhelming users. More detailed operational guides were removed as they were too extensive for an example. ## Additional Context This example was created in response to requests for: - Modern SFTP support in Azure with latest security features - Customer-managed encryption implementation patterns - Healthcare/compliance-ready infrastructure examples (with appropriate disclaimers) - Pulumi YAML examples demonstrating complex Azure architectures
1 parent c3262f9 commit f055d6a

File tree

6 files changed

+1671
-0
lines changed

6 files changed

+1671
-0
lines changed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,4 @@ Gopkg.lock
3030
.classpath
3131
target/
3232
*-test/
33+
PR_MESSAGE.md

CLAUDE.md

Lines changed: 295 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,295 @@
1+
# Pulumi Examples Repository - Development Guide
2+
3+
This repository contains hundreds of Infrastructure as Code (IaC) examples using Pulumi across multiple cloud providers and programming languages.
4+
5+
## Repository Architecture
6+
7+
### High-Level Structure
8+
This is a monorepo containing **300+ examples** organized by cloud provider and programming language:
9+
10+
```
11+
{cloud}-{language}-{example-type}/
12+
├── aws-{go|py|ts|js|cs|fs}-{feature}/
13+
├── azure-{go|py|ts|js|cs|fs}-{feature}/
14+
├── gcp-{go|py|ts|js|cs|fs}-{feature}/
15+
├── kubernetes-{go|py|ts|js|cs|fs}-{feature}/
16+
└── testing-{integration|unit}-{language}/
17+
```
18+
19+
**Supported Languages:**
20+
- **TypeScript/JavaScript** (`ts`/`js`) - Most comprehensive coverage
21+
- **Python** (`py`) - Second-most coverage
22+
- **Go** (`go`) - Growing coverage, especially for AWS
23+
- **C#** (`cs`) - .NET examples
24+
- **F#** (`fs`) - Functional .NET examples
25+
26+
**Cloud Providers:**
27+
- AWS (largest coverage)
28+
- Azure (comprehensive)
29+
- Google Cloud Platform (GCP)
30+
- Kubernetes
31+
- DigitalOcean, Equinix Metal, Linode, OVHCloud
32+
33+
### Project Structure Patterns
34+
35+
Each example follows a consistent structure:
36+
```
37+
example-name/
38+
├── Pulumi.yaml # Project definition
39+
├── main.{go|py|ts|js} # Infrastructure code
40+
├── go.mod # Go modules (Go projects)
41+
├── package.json # Node dependencies (TS/JS)
42+
├── requirements.txt # Python dependencies
43+
├── README.md # Usage documentation
44+
├── app/ # Application code (if containerized)
45+
└── .gitignore
46+
```
47+
48+
### Core Architectural Patterns
49+
50+
1. **Infrastructure Components**: Reusable modules for common patterns
51+
- VPC/Network setup with public/private subnets
52+
- Load balancers with target groups
53+
- ECS/Fargate containers with ECR repositories
54+
- IAM roles and policies
55+
- Security groups with ingress/egress rules
56+
57+
2. **Multi-Tier Applications**: Complete application stacks
58+
- Web servers with databases
59+
- Container orchestration (ECS, EKS, AKS, GKE)
60+
- Serverless functions with API gateways
61+
- Static websites with CDNs
62+
63+
3. **Testing Patterns**: Comprehensive testing strategies
64+
- Unit tests with mocking
65+
- Integration tests with real deployments
66+
- Policy-as-Code validation
67+
68+
## Development Commands
69+
70+
### Essential Commands for Daily Development
71+
72+
#### Working with Individual Examples
73+
```bash
74+
# Navigate to an example and preview changes
75+
cd aws-typescript-eks && pulumi preview
76+
77+
# Deploy an example
78+
cd aws-typescript-eks && pulumi up
79+
80+
# Clean up resources
81+
cd aws-typescript-eks && pulumi destroy
82+
83+
# Check stack outputs
84+
cd aws-typescript-eks && pulumi stack output
85+
```
86+
87+
#### Building and Testing
88+
```bash
89+
# Quick validation of code formatting and linting
90+
make format && make lint
91+
92+
# Run all integration tests (4-hour timeout)
93+
make only_test
94+
95+
# Run tests for specific cloud/language combination
96+
make specific_test_set TestSet=AwsGo
97+
98+
# Run tests with specific tags
99+
make specific_tag_set TagSet=aws TestSet=Go
100+
101+
# Test a single example
102+
make test_example.TestAccAwsPyS3Folder
103+
104+
# Preview changes in PR mode (tests only changed examples)
105+
make pr_preview
106+
```
107+
108+
#### Language-Specific Linting and Formatting
109+
```bash
110+
# Repository-wide linting and formatting
111+
make lint # Run all linting across languages
112+
make format # Format all code (Python with Black)
113+
114+
# Python-specific
115+
make check_python_formatting # Validate Python formatting
116+
make setup_python # Setup Python virtual environment
117+
118+
# TypeScript linting (repository-wide)
119+
tslint -c tslint.json **/*.ts
120+
121+
# Go formatting (per-project)
122+
cd aws-go-fargate && go fmt ./...
123+
cd aws-go-fargate && go mod tidy
124+
125+
# C#/.NET (per-project)
126+
cd aws-cs-webserver && dotnet format
127+
```
128+
129+
#### Testing Infrastructure
130+
```bash
131+
# Setup test dependencies
132+
make ensure
133+
134+
# Go unit tests (specific project)
135+
cd testing-unit-go && go test
136+
137+
# Python unit tests
138+
cd testing-unit-py && python -m pytest
139+
140+
# TypeScript unit tests
141+
cd testing-unit-ts/mocha && npm test
142+
```
143+
144+
### CI/CD Workflows
145+
146+
The repository uses GitHub Actions with several specialized workflows:
147+
148+
#### Main Test Workflow (`test-examples.yml`)
149+
- **TypeScript Linting**: `tslint` validation across all TS files
150+
- **Unit Tests**: Language-specific unit test execution
151+
- **Python Formatting**: Black code formatting validation
152+
- **Integration Tests**: Matrix testing across cloud providers and languages
153+
- Platforms: AWS, Azure, GCP, DigitalOcean, Equinix Metal
154+
- Languages: Go, Python, TypeScript, JavaScript, C#, F#
155+
- **Kubernetes Tests**: Specialized K8s testing with Minikube
156+
157+
#### PR Preview Workflow
158+
- Detects changed examples vs base branch
159+
- Automatically seeds missing configuration with safe placeholders
160+
- Runs `pulumi preview` on changed examples only
161+
- Supports multiple runtime detection (Node.js, Python, Go, .NET, YAML)
162+
163+
### Testing Architecture
164+
165+
#### Integration Testing Framework
166+
Located in `misc/test/`, the testing system:
167+
168+
1. **Test Definitions**: Structured test configuration in `definitions/`
169+
- Tagged by language and cloud provider
170+
- Uses Pulumi's `integration.ProgramTestOptions`
171+
- Supports parallel execution (40 parallel tests)
172+
173+
2. **Test Execution**:
174+
- Go-based test runner with `gotestfmt` formatting
175+
- 4-hour timeout for full test suite
176+
- Comprehensive cloud provider authentication
177+
178+
3. **Test Categories**:
179+
- **Unit Tests**: Mock-based testing with `pulumi.WithMocks()`
180+
- **Integration Tests**: Deploy-check-destroy lifecycle
181+
- **Policy Tests**: Policy-as-Code validation
182+
183+
#### Example Unit Test Pattern (Go)
184+
```go
185+
// Mock resource creation
186+
func (mocks) NewResource(args pulumi.MockResourceArgs) (string, resource.PropertyMap, error) {
187+
outputs := args.Inputs.Mappable()
188+
if args.TypeToken == "aws:ec2/instance:Instance" {
189+
outputs["publicIp"] = "203.0.113.12"
190+
}
191+
return args.Name + "_id", resource.NewPropertyMapFromMap(outputs), nil
192+
}
193+
194+
// Test infrastructure properties
195+
func TestInfrastructure(t *testing.T) {
196+
err := pulumi.RunErr(func(ctx *pulumi.Context) error {
197+
infra, err := createInfrastructure(ctx)
198+
// Assert expected properties
199+
pulumi.All(infra.server.Tags).ApplyT(func(tags interface{}) error {
200+
assert.Contains(t, tags, "Name")
201+
return nil
202+
})
203+
return err
204+
}, pulumi.WithMocks("project", "stack", mocks(0)))
205+
assert.NoError(t, err)
206+
}
207+
```
208+
209+
### Development Workflow
210+
211+
#### Before Working on Examples
212+
1. **Prerequisites Setup**:
213+
- Configure cloud provider credentials (AWS CLI, Azure CLI, gcloud)
214+
- Install language-specific tools (Node.js, Python, Go, .NET)
215+
- Run `pulumi login` to authenticate with Pulumi
216+
217+
2. **Working with Existing Examples**:
218+
- Navigate to example directory: `cd aws-typescript-eks`
219+
- Install dependencies: `npm install` (or language equivalent)
220+
- Configure stack: `pulumi config set aws:region us-west-2`
221+
- Preview: `pulumi preview`
222+
- Deploy: `pulumi up`
223+
- Clean up: `pulumi destroy`
224+
225+
#### Adding New Examples
226+
1. **Naming and Structure**:
227+
- Follow `{cloud}-{language}-{feature}` naming convention
228+
- Create directory with standard structure (see Project Structure Patterns above)
229+
- Include comprehensive README with deployment steps
230+
231+
2. **Testing Integration**:
232+
- Add test definitions to `misc/test/definitions/`
233+
- Tag appropriately for CI matrix execution
234+
- Test locally before submitting PR
235+
236+
3. **Code Quality**:
237+
- Run language-specific linting and formatting
238+
- Ensure example works with `pulumi preview` and `pulumi up`
239+
- Include proper resource tagging and cleanup
240+
241+
#### PR Workflow
242+
- Run `make pr_preview` to test only changed examples
243+
- Automatic linting and formatting validation
244+
- Integration tests run on maintainer approval
245+
- Use `make format && make lint` before committing
246+
247+
## Key Dependencies and Tools
248+
249+
### Core Infrastructure
250+
- **Pulumi SDK**: v3.x across all languages
251+
- **Cloud Provider SDKs**: AWS SDK v7, Azure SDK, GCP SDK
252+
- **Docker**: For containerized examples
253+
254+
### Testing and CI
255+
- **gotestfmt**: Go test output formatting
256+
- **pytest**: Python testing framework
257+
- **Mocha**: TypeScript/JavaScript testing
258+
- **tslint**: TypeScript linting
259+
- **Black**: Python code formatting
260+
261+
### Development Tools
262+
- **Node.js 20**: TypeScript/JavaScript runtime
263+
- **Python 3.9+**: Python runtime
264+
- **Go 1.21+**: Go runtime
265+
- **Helm**: Kubernetes package management
266+
- **kubectl**: Kubernetes CLI
267+
268+
## Common Development Issues
269+
270+
### Authentication Problems
271+
- **Cloud Provider Credentials**: Ensure AWS CLI, Azure CLI, or gcloud is configured
272+
- **Pulumi Backend**: Run `pulumi login` to authenticate with Pulumi service
273+
- **Permission Errors**: Check IAM roles have sufficient permissions for resource creation
274+
275+
### Resource Conflicts
276+
- **Stack State**: Use unique stack names to avoid conflicts: `pulumi stack select dev-yourname`
277+
- **Resource Names**: Many examples use randomized names to avoid conflicts
278+
- **Region Conflicts**: Configure appropriate regions: `pulumi config set aws:region us-west-2`
279+
280+
### Testing Issues
281+
- **Long Test Times**: Full integration tests take up to 4 hours - use `make pr_preview` for quick validation
282+
- **Parallel Test Failures**: Tests run 40 in parallel - occasional timeouts are expected
283+
- **Resource Cleanup**: Always run `pulumi destroy` after testing to avoid costs
284+
285+
## Architecture Insights
286+
287+
This repository demonstrates several key IaC architectural patterns:
288+
289+
1. **Multi-Cloud Abstraction**: Examples show similar patterns across cloud providers
290+
2. **Language Polyglot**: Same infrastructure patterns implemented across multiple programming languages
291+
3. **Component Reusability**: Higher-level components built from cloud primitives
292+
4. **Testing Strategy**: Comprehensive unit, integration, and policy testing
293+
5. **CI/CD Integration**: Automated testing and validation at scale
294+
295+
The testing infrastructure alone processes hundreds of examples across multiple cloud providers, languages, and test types, making it one of the most comprehensive IaC testing frameworks available.

0 commit comments

Comments
 (0)