Skip to content

Commit 6f112db

Browse files
committed
feat(azure-yaml): add enterprise SFTP blob storage example with advanced security
Add new Pulumi YAML example demonstrating Azure Blob Storage with SFTP support, featuring enterprise-grade security controls including customer-managed encryption, private network access, and comprehensive audit logging. - **Customer-Managed Encryption**: HSM-backed 4096-bit RSA keys in Azure Key Vault Premium - **Infrastructure Encryption**: Double encryption layer for maximum data protection - **Private Network Only**: Zero public internet access with private endpoints - **Immutable Storage**: WORM compliance with version control and change tracking - **Extended Audit Logging**: 7-year retention for compliance requirements - **Role-Based Access**: Admin and auditor users with granular container permissions - **Latest Azure Native Provider**: Uses Azure Native v3.8+ features and 2024 API versions - **Storage Account**: Data Lake Gen2 (hierarchical namespace) with SFTP enabled - **Key Vault Premium**: HSM-backed customer-managed encryption keys - **Encryption Scope**: Dedicated scope with infrastructure encryption - **Virtual Network**: Isolated VNet with subnets for storage and Key Vault - **Private Endpoints**: Secure access to storage and Key Vault services - **Managed Identity**: Secure Key Vault access without shared keys - **Log Analytics**: Comprehensive monitoring with extended retention ```yaml Encryption: - At Rest: Customer-managed keys (4096-bit RSA-HSM) - In Transit: TLS 1.2+ and SSH - Infrastructure: Double encryption enabled Network Security: - Public Access: Disabled - Private Endpoints: Required - Network Rules: Explicit deny all - Service Bypass: None Access Control: - Authentication: SSH keys only (passwords disabled) - Shared Keys: Disabled - Authorization: Azure AD OAuth - User Types: Admin (full) and Auditor (read-only) Data Integrity: - Immutable Storage: Enabled on all containers - Versioning: Full version control - Change Feed: 7-year audit trail - Soft Delete: 7-year retention ``` - Clean technical implementation guide - Quick start with configuration examples - Architecture diagram with mermaid - Troubleshooting and support information - Minimal length focused on essentials - Healthcare security reference (educational only) - Comprehensive legal disclaimers - Technical security patterns for regulated industries - No compliance guarantees - professional validation required - Reduced from initial 3,754 lines (80% reduction) - Removed excessive operational guides - Focused on practical implementation While this example implements security controls commonly required by healthcare and financial organizations, it includes prominent disclaimers: - ⚠️ No guarantee of HIPAA, PHI, or regulatory compliance - ⚠️ Professional legal and compliance consultation required - ⚠️ Independent security assessment needed for production use - ⚠️ Compliance standards evolve - always verify current requirements Healthcare organizations can reference `HIPAA_COMPLIANCE.md` for additional security considerations, but must validate with their own compliance teams. To test this example: ```bash cd azure-yaml-sftp-blob pulumi stack init dev pulumi config set azure-native:location "East US 2" pulumi config set storageAccountName "testsftpstorage$(date +%s)" pulumi config set userPublicKey "$(cat ~/.ssh/id_rsa.pub)" pulumi preview pulumi up ``` This YAML example can be converted to other languages: ```bash pulumi convert --language typescript --out ./typescript-version pulumi convert --language python --out ./python-version pulumi convert --language go --out ./go-version pulumi convert --language csharp --out ./csharp-version ``` None - this is a new example. Closes #XXXX (if applicable) - [x] Code follows repository standards - [x] Documentation is clear and concise - [x] Example includes architecture diagram - [x] Security best practices implemented - [x] Legal disclaimers for compliance content - [x] Formatting validated with `make format` and `make check_python_formatting` - [ ] Tested deployment with `pulumi preview` - [ ] Tested full deployment cycle (requires Azure environment) 1. **Legal Review**: The HIPAA_COMPLIANCE.md file includes extensive legal disclaimers. Please validate these are sufficient to protect against liability claims. 2. **Azure Syntax**: Some Pulumi YAML syntax should be validated during review: - Managed identity property structure (line 150) - `getClientConfig` function syntax (lines 60, 108) - Subnet array indexing (lines 393, 414) - BlobService versioning property name (line 212) 3. **Security Configuration**: All security features use latest Azure Native provider capabilities. Review that: - Customer-managed keys are properly configured - Infrastructure encryption is enabled at all layers - Private network access is enforced correctly - Audit logging captures all required events 4. **Documentation Scope**: Intentionally kept documentation minimal (755 lines total) to avoid overwhelming users. More detailed operational guides were removed as they were too extensive for an example. This example was created in response to requests for: - Modern SFTP support in Azure with latest security features - Customer-managed encryption implementation patterns - Healthcare/compliance-ready infrastructure examples (with appropriate disclaimers) - Pulumi YAML examples demonstrating complex Azure architectures
1 parent 616b667 commit 6f112db

File tree

6 files changed

+1671
-0
lines changed

6 files changed

+1671
-0
lines changed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,4 @@ Gopkg.lock
3131
target/
3232
*-test/
3333
.claude/settings.local.json
34+
PR_MESSAGE.md

CLAUDE.md

Lines changed: 295 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,295 @@
1+
# Pulumi Examples Repository - Development Guide
2+
3+
This repository contains hundreds of Infrastructure as Code (IaC) examples using Pulumi across multiple cloud providers and programming languages.
4+
5+
## Repository Architecture
6+
7+
### High-Level Structure
8+
This is a monorepo containing **300+ examples** organized by cloud provider and programming language:
9+
10+
```
11+
{cloud}-{language}-{example-type}/
12+
├── aws-{go|py|ts|js|cs|fs}-{feature}/
13+
├── azure-{go|py|ts|js|cs|fs}-{feature}/
14+
├── gcp-{go|py|ts|js|cs|fs}-{feature}/
15+
├── kubernetes-{go|py|ts|js|cs|fs}-{feature}/
16+
└── testing-{integration|unit}-{language}/
17+
```
18+
19+
**Supported Languages:**
20+
- **TypeScript/JavaScript** (`ts`/`js`) - Most comprehensive coverage
21+
- **Python** (`py`) - Second-most coverage
22+
- **Go** (`go`) - Growing coverage, especially for AWS
23+
- **C#** (`cs`) - .NET examples
24+
- **F#** (`fs`) - Functional .NET examples
25+
26+
**Cloud Providers:**
27+
- AWS (largest coverage)
28+
- Azure (comprehensive)
29+
- Google Cloud Platform (GCP)
30+
- Kubernetes
31+
- DigitalOcean, Equinix Metal, Linode, OVHCloud
32+
33+
### Project Structure Patterns
34+
35+
Each example follows a consistent structure:
36+
```
37+
example-name/
38+
├── Pulumi.yaml # Project definition
39+
├── main.{go|py|ts|js} # Infrastructure code
40+
├── go.mod # Go modules (Go projects)
41+
├── package.json # Node dependencies (TS/JS)
42+
├── requirements.txt # Python dependencies
43+
├── README.md # Usage documentation
44+
├── app/ # Application code (if containerized)
45+
└── .gitignore
46+
```
47+
48+
### Core Architectural Patterns
49+
50+
1. **Infrastructure Components**: Reusable modules for common patterns
51+
- VPC/Network setup with public/private subnets
52+
- Load balancers with target groups
53+
- ECS/Fargate containers with ECR repositories
54+
- IAM roles and policies
55+
- Security groups with ingress/egress rules
56+
57+
2. **Multi-Tier Applications**: Complete application stacks
58+
- Web servers with databases
59+
- Container orchestration (ECS, EKS, AKS, GKE)
60+
- Serverless functions with API gateways
61+
- Static websites with CDNs
62+
63+
3. **Testing Patterns**: Comprehensive testing strategies
64+
- Unit tests with mocking
65+
- Integration tests with real deployments
66+
- Policy-as-Code validation
67+
68+
## Development Commands
69+
70+
### Essential Commands for Daily Development
71+
72+
#### Working with Individual Examples
73+
```bash
74+
# Navigate to an example and preview changes
75+
cd aws-typescript-eks && pulumi preview
76+
77+
# Deploy an example
78+
cd aws-typescript-eks && pulumi up
79+
80+
# Clean up resources
81+
cd aws-typescript-eks && pulumi destroy
82+
83+
# Check stack outputs
84+
cd aws-typescript-eks && pulumi stack output
85+
```
86+
87+
#### Building and Testing
88+
```bash
89+
# Quick validation of code formatting and linting
90+
make format && make lint
91+
92+
# Run all integration tests (4-hour timeout)
93+
make only_test
94+
95+
# Run tests for specific cloud/language combination
96+
make specific_test_set TestSet=AwsGo
97+
98+
# Run tests with specific tags
99+
make specific_tag_set TagSet=aws TestSet=Go
100+
101+
# Test a single example
102+
make test_example.TestAccAwsPyS3Folder
103+
104+
# Preview changes in PR mode (tests only changed examples)
105+
make pr_preview
106+
```
107+
108+
#### Language-Specific Linting and Formatting
109+
```bash
110+
# Repository-wide linting and formatting
111+
make lint # Run all linting across languages
112+
make format # Format all code (Python with Black)
113+
114+
# Python-specific
115+
make check_python_formatting # Validate Python formatting
116+
make setup_python # Setup Python virtual environment
117+
118+
# TypeScript linting (repository-wide)
119+
tslint -c tslint.json **/*.ts
120+
121+
# Go formatting (per-project)
122+
cd aws-go-fargate && go fmt ./...
123+
cd aws-go-fargate && go mod tidy
124+
125+
# C#/.NET (per-project)
126+
cd aws-cs-webserver && dotnet format
127+
```
128+
129+
#### Testing Infrastructure
130+
```bash
131+
# Setup test dependencies
132+
make ensure
133+
134+
# Go unit tests (specific project)
135+
cd testing-unit-go && go test
136+
137+
# Python unit tests
138+
cd testing-unit-py && python -m pytest
139+
140+
# TypeScript unit tests
141+
cd testing-unit-ts/mocha && npm test
142+
```
143+
144+
### CI/CD Workflows
145+
146+
The repository uses GitHub Actions with several specialized workflows:
147+
148+
#### Main Test Workflow (`test-examples.yml`)
149+
- **TypeScript Linting**: `tslint` validation across all TS files
150+
- **Unit Tests**: Language-specific unit test execution
151+
- **Python Formatting**: Black code formatting validation
152+
- **Integration Tests**: Matrix testing across cloud providers and languages
153+
- Platforms: AWS, Azure, GCP, DigitalOcean, Equinix Metal
154+
- Languages: Go, Python, TypeScript, JavaScript, C#, F#
155+
- **Kubernetes Tests**: Specialized K8s testing with Minikube
156+
157+
#### PR Preview Workflow
158+
- Detects changed examples vs base branch
159+
- Automatically seeds missing configuration with safe placeholders
160+
- Runs `pulumi preview` on changed examples only
161+
- Supports multiple runtime detection (Node.js, Python, Go, .NET, YAML)
162+
163+
### Testing Architecture
164+
165+
#### Integration Testing Framework
166+
Located in `misc/test/`, the testing system:
167+
168+
1. **Test Definitions**: Structured test configuration in `definitions/`
169+
- Tagged by language and cloud provider
170+
- Uses Pulumi's `integration.ProgramTestOptions`
171+
- Supports parallel execution (40 parallel tests)
172+
173+
2. **Test Execution**:
174+
- Go-based test runner with `gotestfmt` formatting
175+
- 4-hour timeout for full test suite
176+
- Comprehensive cloud provider authentication
177+
178+
3. **Test Categories**:
179+
- **Unit Tests**: Mock-based testing with `pulumi.WithMocks()`
180+
- **Integration Tests**: Deploy-check-destroy lifecycle
181+
- **Policy Tests**: Policy-as-Code validation
182+
183+
#### Example Unit Test Pattern (Go)
184+
```go
185+
// Mock resource creation
186+
func (mocks) NewResource(args pulumi.MockResourceArgs) (string, resource.PropertyMap, error) {
187+
outputs := args.Inputs.Mappable()
188+
if args.TypeToken == "aws:ec2/instance:Instance" {
189+
outputs["publicIp"] = "203.0.113.12"
190+
}
191+
return args.Name + "_id", resource.NewPropertyMapFromMap(outputs), nil
192+
}
193+
194+
// Test infrastructure properties
195+
func TestInfrastructure(t *testing.T) {
196+
err := pulumi.RunErr(func(ctx *pulumi.Context) error {
197+
infra, err := createInfrastructure(ctx)
198+
// Assert expected properties
199+
pulumi.All(infra.server.Tags).ApplyT(func(tags interface{}) error {
200+
assert.Contains(t, tags, "Name")
201+
return nil
202+
})
203+
return err
204+
}, pulumi.WithMocks("project", "stack", mocks(0)))
205+
assert.NoError(t, err)
206+
}
207+
```
208+
209+
### Development Workflow
210+
211+
#### Before Working on Examples
212+
1. **Prerequisites Setup**:
213+
- Configure cloud provider credentials (AWS CLI, Azure CLI, gcloud)
214+
- Install language-specific tools (Node.js, Python, Go, .NET)
215+
- Run `pulumi login` to authenticate with Pulumi
216+
217+
2. **Working with Existing Examples**:
218+
- Navigate to example directory: `cd aws-typescript-eks`
219+
- Install dependencies: `npm install` (or language equivalent)
220+
- Configure stack: `pulumi config set aws:region us-west-2`
221+
- Preview: `pulumi preview`
222+
- Deploy: `pulumi up`
223+
- Clean up: `pulumi destroy`
224+
225+
#### Adding New Examples
226+
1. **Naming and Structure**:
227+
- Follow `{cloud}-{language}-{feature}` naming convention
228+
- Create directory with standard structure (see Project Structure Patterns above)
229+
- Include comprehensive README with deployment steps
230+
231+
2. **Testing Integration**:
232+
- Add test definitions to `misc/test/definitions/`
233+
- Tag appropriately for CI matrix execution
234+
- Test locally before submitting PR
235+
236+
3. **Code Quality**:
237+
- Run language-specific linting and formatting
238+
- Ensure example works with `pulumi preview` and `pulumi up`
239+
- Include proper resource tagging and cleanup
240+
241+
#### PR Workflow
242+
- Run `make pr_preview` to test only changed examples
243+
- Automatic linting and formatting validation
244+
- Integration tests run on maintainer approval
245+
- Use `make format && make lint` before committing
246+
247+
## Key Dependencies and Tools
248+
249+
### Core Infrastructure
250+
- **Pulumi SDK**: v3.x across all languages
251+
- **Cloud Provider SDKs**: AWS SDK v7, Azure SDK, GCP SDK
252+
- **Docker**: For containerized examples
253+
254+
### Testing and CI
255+
- **gotestfmt**: Go test output formatting
256+
- **pytest**: Python testing framework
257+
- **Mocha**: TypeScript/JavaScript testing
258+
- **tslint**: TypeScript linting
259+
- **Black**: Python code formatting
260+
261+
### Development Tools
262+
- **Node.js 20**: TypeScript/JavaScript runtime
263+
- **Python 3.9+**: Python runtime
264+
- **Go 1.21+**: Go runtime
265+
- **Helm**: Kubernetes package management
266+
- **kubectl**: Kubernetes CLI
267+
268+
## Common Development Issues
269+
270+
### Authentication Problems
271+
- **Cloud Provider Credentials**: Ensure AWS CLI, Azure CLI, or gcloud is configured
272+
- **Pulumi Backend**: Run `pulumi login` to authenticate with Pulumi service
273+
- **Permission Errors**: Check IAM roles have sufficient permissions for resource creation
274+
275+
### Resource Conflicts
276+
- **Stack State**: Use unique stack names to avoid conflicts: `pulumi stack select dev-yourname`
277+
- **Resource Names**: Many examples use randomized names to avoid conflicts
278+
- **Region Conflicts**: Configure appropriate regions: `pulumi config set aws:region us-west-2`
279+
280+
### Testing Issues
281+
- **Long Test Times**: Full integration tests take up to 4 hours - use `make pr_preview` for quick validation
282+
- **Parallel Test Failures**: Tests run 40 in parallel - occasional timeouts are expected
283+
- **Resource Cleanup**: Always run `pulumi destroy` after testing to avoid costs
284+
285+
## Architecture Insights
286+
287+
This repository demonstrates several key IaC architectural patterns:
288+
289+
1. **Multi-Cloud Abstraction**: Examples show similar patterns across cloud providers
290+
2. **Language Polyglot**: Same infrastructure patterns implemented across multiple programming languages
291+
3. **Component Reusability**: Higher-level components built from cloud primitives
292+
4. **Testing Strategy**: Comprehensive unit, integration, and policy testing
293+
5. **CI/CD Integration**: Automated testing and validation at scale
294+
295+
The testing infrastructure alone processes hundreds of examples across multiple cloud providers, languages, and test types, making it one of the most comprehensive IaC testing frameworks available.

0 commit comments

Comments
 (0)