docs: expand AWS dynamic cloud configuration documentation

Enhance documentation for AWS dynamic configuration requirements:

- Detailed prerequisites and AWS CLI requirements
- AWS credentials configuration methods
- Required IAM permissions
- Implementation architecture details
- Troubleshooting guide for common issues
- Best practices for administrators
- Advanced usage scenarios
- Key design decisions and rationale

This helps developer and users understand that the system wraps the
official AWS CLI tool rather than implementing its own API client, and
requires proper AWS credentials configuration.

Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <[email protected]>

Author: mcgrof

docs/cloud-configuration.md

@@ -11,6 +11,116 @@ The cloud configuration system follows a pattern similar to Linux kernel refs ma
 - **No dependency on cloud CLI tools** for regular users
 - **Reduced API calls** to cloud providers
 
+## Prerequisites for Cloud Providers
+
+### AWS Prerequisites
+
+The AWS dynamic configuration system uses the official AWS CLI tool and requires proper authentication to access AWS APIs.
+
+#### Requirements
+
+1. **AWS CLI Installation**
+   ```bash
+   # Using pip
+   pip install awscli
+
+   # On Debian/Ubuntu
+   sudo apt-get install awscli
+
+   # On Fedora/RHEL
+   sudo dnf install aws-cli
+
+   # On macOS
+   brew install awscli
+   ```
+
+2. **AWS Credentials Configuration**
+
+   You need valid AWS credentials configured in one of these ways:
+
+   a. **AWS credentials file** (`~/.aws/credentials`):
+   ```ini
+   [default]
+   aws_access_key_id = YOUR_ACCESS_KEY
+   aws_secret_access_key = YOUR_SECRET_KEY
+   ```
+
+   b. **Environment variables**:
+   ```bash
+   export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
+   export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
+   export AWS_DEFAULT_REGION=us-east-1  # Optional
+   ```
+
+   c. **IAM Instance Role** (when running on EC2):
+   - Automatically uses instance metadata service
+   - No explicit credentials needed
+
+3. **Required AWS Permissions**
+
+   The IAM user or role needs the following read-only permissions:
+   ```json
+   {
+     "Version": "2012-10-17",
+     "Statement": [
+       {
+         "Effect": "Allow",
+         "Action": [
+           "ec2:DescribeRegions",
+           "ec2:DescribeAvailabilityZones",
+           "ec2:DescribeInstanceTypes",
+           "ec2:DescribeImages",
+           "pricing:GetProducts"
+         ],
+         "Resource": "*"
+       },
+       {
+         "Effect": "Allow",
+         "Action": [
+           "sts:GetCallerIdentity"
+         ],
+         "Resource": "*"
+       }
+     ]
+   }
+   ```
+
+#### Verifying AWS Setup
+
+Test your AWS CLI configuration:
+```bash
+# Check AWS CLI is installed
+aws --version
+
+# Verify credentials are configured
+aws sts get-caller-identity
+
+# Test EC2 access
+aws ec2 describe-regions --output table
+```
+
+#### Fallback Behavior
+
+If AWS CLI is not available or credentials are not configured:
+- The system automatically falls back to pre-defined static defaults
+- Basic instance families (M5, T3, C5, etc.) are still available
+- Common regions (us-east-1, eu-west-1, etc.) are provided
+- Default GPU AMI options are included
+- Users can still use kdevops without AWS API access
+
+### Lambda Labs Prerequisites
+
+Lambda Labs configuration requires an API key:
+
+1. **Obtain API Key**: Sign up at [Lambda Labs](https://lambdalabs.com) and generate an API key
+
+2. **Configure API Key**:
+   ```bash
+   export LAMBDA_API_KEY=your_api_key_here
+   ```
+
+3. **Fallback Behavior**: Without an API key, default GPU instance types are provided
+
 ## Configuration Generation Flow
 
 ```
@@ -133,6 +243,48 @@ No cloud CLI tools or API access required - everything loads from committed stat
 
 ## How It Works
 
+### Implementation Architecture
+
+The cloud configuration system consists of several key components:
+
+1. **API Wrapper Scripts** (`scripts/aws-cli`, `scripts/lambda-cli`):
+   - Provide CLI interfaces to cloud provider APIs
+   - Handle authentication and error checking
+   - Format API responses for Kconfig generation
+
+2. **API Libraries** (`scripts/aws_api.py`, `scripts/lambdalabs_api.py`):
+   - Core functions for API interactions
+   - Generate Kconfig syntax from API data
+   - Provide fallback defaults when APIs unavailable
+
+3. **Generation Orchestrator** (`scripts/generate_cloud_configs.py`):
+   - Coordinates parallel generation across providers
+   - Provides summary information
+   - Handles errors gracefully
+
+4. **Makefile Integration** (`scripts/dynamic-cloud-kconfig.Makefile`):
+   - Defines make targets
+   - Manages file dependencies
+   - Handles cleanup and updates
+
+### AWS Implementation Details
+
+The AWS implementation wraps the official AWS CLI tool rather than implementing its own API client:
+
+```python
+# scripts/aws_api.py
+def run_aws_command(command: List[str], region: str = None) -> Optional[Any]:
+    cmd = ["aws"] + command + ["--output", "json"]
+    # ... executes via subprocess
+```
+
+Key features:
+- **Parallel Generation**: Uses ThreadPoolExecutor to generate instance family files concurrently
+- **GPU Detection**: Automatically identifies GPU instances and enables GPU AMI options
+- **Categorized Instance Types**: Groups instances by use case (general, compute, memory, etc.)
+- **Pricing Integration**: Queries pricing API when available
+- **Smart Defaults**: Falls back to well-tested defaults when API unavailable
+
 ### Dynamic Configuration Detection
 
 kdevops automatically detects whether to use dynamic or static configurations:
@@ -251,18 +403,178 @@ make cloud-config
 make cloud-update
 ```
 
+## Troubleshooting
+
+### AWS Issues
+
+#### "AWS CLI not found" Error
+```bash
+# Verify AWS CLI installation
+which aws
+aws --version
+
+# Install if missing (see Prerequisites section)
+```
+
+#### "Credentials not configured" Error
+```bash
+# Check current identity
+aws sts get-caller-identity
+
+# If fails, configure credentials:
+aws configure
+# OR
+export AWS_ACCESS_KEY_ID=your_key
+export AWS_SECRET_ACCESS_KEY=your_secret
+```
+
+#### "Access Denied" Errors
+- Verify your IAM user/role has the required permissions (see Prerequisites)
+- Check if you're in the correct AWS account
+- Ensure your credentials haven't expired
+
+#### Slow Generation Times
+- Normal for AWS (6+ minutes due to API pagination)
+- Consider using `make cloud-update` with pre-generated configs
+- Run generation during off-peak hours
+
+#### Missing Instance Types
+```bash
+# Force regeneration
+make clean-cloud-config
+make cloud-config
+make cloud-update
+```
+
+### General Issues
+
+#### Static Files Not Loading
+```bash
+# Verify static files exist
+ls terraform/aws/kconfigs/*.static
+
+# If missing, regenerate:
+make cloud-config
+make cloud-update
+```
+
+#### Changes Not Reflected in Menuconfig
+```bash
+# Clear Kconfig cache
+make mrproper
+make menuconfig
+```
+
+#### Debugging API Calls
+```bash
+# Enable debug output
+export DEBUG=1
+make cloud-config
+
+# Test API directly
+scripts/aws-cli --output json regions list
+scripts/aws-cli --output json instance-types list --family m5
+```
+
+## Best Practices
+
+1. **Regular Updates**: Administrators should regenerate configurations monthly or when new instance types are announced
+
+2. **Commit Messages**: Include generation date and tool versions when committing static files:
+   ```bash
+   git commit -m "cloud: update AWS static configurations
+
+   Generated with AWS CLI 2.15.0 on 2024-01-15
+   - Added new G6e instance family
+   - Updated GPU AMI options
+   - 127 instance families now available"
+   ```
+
+3. **Testing**: Always test generated configurations before committing:
+   ```bash
+   make cloud-config
+   make cloud-update
+   make menuconfig  # Verify options appear correctly
+   ```
+
+4. **Partial Generation**: For faster testing, generate only specific providers:
+   ```bash
+   make cloud-config-aws      # AWS only
+   make cloud-config-lambdalabs  # Lambda Labs only
+   ```
+
+5. **CI/CD Integration**: Consider automating configuration updates in CI pipelines
+
+## Advanced Usage
+
+### Custom AWS Profiles
+```bash
+# Use non-default AWS profile
+export AWS_PROFILE=myprofile
+make cloud-config
+```
+
+### Specific Region Generation
+```bash
+# Generate for specific region (affects default selections)
+export AWS_DEFAULT_REGION=eu-west-1
+make cloud-config
+```
+
+### Parallel Generation
+The system automatically uses parallel processing:
+- AWS: Up to 10 concurrent instance family generations
+- Reduces total generation time significantly
+
+## File Reference
+
+### AWS Files
+- `terraform/aws/kconfigs/Kconfig.compute.{generated,static}` - Instance families
+- `terraform/aws/kconfigs/Kconfig.location.{generated,static}` - Regions and zones
+- `terraform/aws/kconfigs/Kconfig.gpu-amis.{generated,static}` - GPU AMI options
+- `terraform/aws/kconfigs/instance-types/Kconfig.*.{generated,static}` - Per-family sizes
+
+### Marker Files
+- `.aws_cloud_config_generated` - Enables dynamic AWS config
+- `.cloud.initialized` - General cloud config marker
+
+### Scripts
+- `scripts/aws-cli` - AWS CLI wrapper with user-friendly commands
+- `scripts/aws_api.py` - AWS API library and Kconfig generation
+- `scripts/generate_cloud_configs.py` - Main orchestrator for all providers
+- `scripts/dynamic-cloud-kconfig.Makefile` - Make targets and integration
+
 ## Implementation Details
 
-The cloud configuration system is implemented in:
+The cloud configuration system is implemented using:
+
+- **AWS CLI Wrapper**: Uses official AWS CLI via subprocess calls
+- **Parallel Processing**: ThreadPoolExecutor for concurrent API calls
+- **Fallback Defaults**: Pre-defined configurations when API unavailable
+- **Two-tier System**: Generated (dynamic) → Static (committed) files
+- **Kconfig Integration**: Seamless integration with Linux kernel-style configuration
+
+### Key Design Decisions
+
+1. **Why wrap AWS CLI instead of using boto3?**
+   - Reduces dependencies (AWS CLI often already installed)
+   - Leverages AWS's official tool and authentication methods
+   - Simpler credential management (uses standard AWS config)
+
+2. **Why the two-tier system?**
+   - Fast loading for regular users (no API calls needed)
+   - Fresh data when administrators regenerate
+   - Works offline and in restricted environments
 
-- `scripts/dynamic-cloud-kconfig.Makefile` - Make targets and build rules
-- `scripts/aws_api.py` - AWS configuration generator
-- `scripts/generate_cloud_configs.py` - Main configuration generator
-- `terraform/*/kconfigs/` - Provider-specific Kconfig files
+3. **Why 6 minutes generation time?**
+   - AWS API pagination limits (100 items per request)
+   - Comprehensive data collection (all regions, all instance types)
+   - Parallel processing already optimized
 
 ## See Also
 
 - [AWS Instance Types](https://aws.amazon.com/ec2/instance-types/)
 - [Azure VM Sizes](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes)
 - [GCE Machine Types](https://cloud.google.com/compute/docs/machine-types)
 - [kdevops Terraform Documentation](terraform.md)
+- [AWS CLI Documentation](https://docs.aws.amazon.com/cli/)