Add AES, KMS, MOST RECENT PROVIDER

Author: imabhichow

PerfTest/runtimes/python/DynamoDbEncryption/README.md

@@ -1,239 +1,142 @@
-# AWS Database Encryption SDK - DynamoDB Performance Tests
+# Python Performance Tests: DDBEC v3 vs DB-ESDK v4
 
-This directory contains comprehensive performance tests comparing the Legacy DynamoDB Encryption Client Python v3 (DDBEC v3) against the AWS Database Encryption SDK Python v4 (DB-ESDK v4).
+This directory contains comprehensive performance benchmarks comparing the **Legacy DynamoDB Encryption Client Python v3** (DDBEC v3) against the **AWS Database Encryption SDK Python v4** (DB-ESDK v4).
 
 ## Overview
 
-The performance tests benchmark encryption and decryption operations across:
+The performance tests benchmark encryption and decryption operations using pytest-benchmark across multiple data types and encryption providers/keyrings.
 
-### DDBEC v3 Providers
+### Libraries Under Test
+
+**DDBEC v3 Providers:**
 - AES Key Provider
 - AWS KMS Key Provider  
 - Most Recent Key Provider
 
-### DB-ESDK v4 Keyrings
+**DB-ESDK v4 Keyrings:**
 - Raw AES Keyring
 - AWS KMS Keyring
-- Hierarchy Keyring (simplified for testing)
-
-### Test Data Categories
-- **Single Attribute**: Simple string attribute
-- **Nested Attributes**: Complex nested JSON structure
-- **Flat Attributes**: Many top-level attributes
-
-## Prerequisites
+- Hierarchy Keyring
 
-1. **Python 3.8+** installed
-2. **AWS credentials** configured (for KMS tests)
-3. **AWS KMS key** with appropriate permissions
+### Test Data Types
+- **single_attribute**: Simple string data
+- **nested_attributes**: Complex nested JSON structures
+- **flat_attributes**: Multiple top-level attributes
 
-## Installation
+## Quick Start
 
-1. Install Poetry (if not already installed):
-```bash
-curl -sSL https://install.python-poetry.org | python3 -
-```
+### Prerequisites
+- Python 3.11+ 
+- Poetry installed
+- AWS credentials configured
+- Access to AWS KMS key: `arn:aws:kms:us-west-2:370957321024:key/9d989aa2-2f9c-438c-a745-cc57d3ad0126`
 
-2. Install dependencies:
+### Installation
 ```bash
-poetry install --with test
+cd PerfTest/runtimes/python/DynamoDbEncryption
+poetry install
 ```
 
-## Running the Tests
-
-### Using Tox (Recommended)
-
-Run all performance tests:
+### Run All Benchmarks
 ```bash
-# Run all tests
-tox -e py311-all
-
-# Run only v3 tests
-tox -e py311-v3
-
-# Run only v4 tests
-tox -e py311-v4
-
-# Run the benchmark script
 tox -e run-benchmarks
-```
 
-### Using Poetry Directly
+# Or
+python run_benchmarks.py
 
-```bash
-# Run all tests
-poetry run pytest src/ --benchmark-only
-
-# Run the benchmark script
+# Or using poetry
 poetry run python run_benchmarks.py
 ```
 
-### Command Line Options
+## Usage Examples
 
+### Basic Commands
 ```bash
-poetry run python run_benchmarks.py [OPTIONS]
-
-Options:
-  --version {v3,v4,all}  Which version to benchmark (default: all)
-  --test PATTERN         Run specific test pattern (e.g., 'aes' or 'encrypt')
-  --output-dir DIR       Directory to save results (default: benchmark_results)
-  --no-save             Don't save results to files
-  --pytest-args ARGS    Additional pytest arguments
-```
+# Run all tests with comparison report
+python run_benchmarks.py
 
-### Examples with Tox
+# Test only DDBEC v3
+python run_benchmarks.py --version v3
 
-```bash
-# Run with different Python versions
-tox -e py311-all
-tox -e py312-all
-tox -e py313-all
+# Test only DB-ESDK v4  
+python run_benchmarks.py --version v4
 
-# Run linting
-tox -e lint
+# Test specific provider (AES, KMS, etc.)
+python run_benchmarks.py --test kms
 
-# Format code
-tox -e format
+# Test only encryption operations
+python run_benchmarks.py --test encrypt
 ```
 
-### Examples with Poetry
-
-1. **Run only v3 tests:**
-   ```bash
-   poetry run python run_benchmarks.py --version v3
-   ```
-
-2. **Run only AES-based tests:**
-   ```bash
-   poetry run python run_benchmarks.py --test aes
-   ```
+### Advanced Options
+```bash
+# Don't save files, only show console output
+python run_benchmarks.py --no-save
 
-3. **Run only encryption tests:**
-   ```bash
-   poetry run python run_benchmarks.py --test encrypt
-   ```
+# Save to custom directory
+python run_benchmarks.py --output-dir my_results
 
-4. **Run with custom pytest options:**
-   ```bash
-   poetry run python run_benchmarks.py --pytest-args "--benchmark-rounds=20"
-   ```
+# Pass additional pytest arguments
+python run_benchmarks.py --pytest-args "--benchmark-rounds=10"
+```
 
 ### Direct pytest Usage
-
-You can also run pytest directly with more control:
-
 ```bash
-# Run all tests
-poetry run pytest -v --benchmark-only
-
-# Run only v3 tests
-poetry run pytest -v --benchmark-only -m v3
+# Run with pytest directly
+poetry run pytest src/ --benchmark-only -v
 
-# Run only v4 tests
-poetry run pytest -v --benchmark-only -m v4
+# Run specific test file
+poetry run pytest src/v3/aes_key_provider_test.py --benchmark-only -v
 
-# Run specific provider tests
-poetry run pytest -v --benchmark-only -k "aes"
-poetry run pytest -v --benchmark-only -k "kms"
-
-# Run with custom benchmark options
-poetry run pytest -v --benchmark-only --benchmark-columns=min,max,mean,ops
+# Run single test function
+poetry run pytest src/v4/raw_aes_keyring_test.py::test_encrypt_performance[single_attribute] --benchmark-only -v
 ```
 
 ## Output
 
-The benchmark script generates multiple output formats:
-
-1. **Console Output**: Immediate results with comparison table
-2. **CSV Files**: Raw results and comparison data
-3. **JSON Files**: Detailed benchmark data
-4. **HTML Report**: Visual comparison report
-
-Results are saved in the `benchmark_results/` directory with timestamps.
-
-## Understanding the Results
-
-### Performance Metrics
-- **Mean**: Average time per operation
-- **Min/Max**: Best and worst case times
-- **Ops/sec**: Operations per second
-- **StdDev**: Standard deviation of measurements
-
-### Comparison Report
-- **Negative percentages**: v4 is faster than v3
-- **Positive percentages**: v4 is slower than v3
-
-### Size Comparison
-For encryption operations, the report includes:
-- Original item size
-- Encrypted item size
-- Size increase ratio
-
-## Test Structure
+The benchmark script generates:
 
+### Console Report
 ```
-src/
-├── __init__.py
-├── test_constants.py         # Shared constants
-├── v3/                       # DDBEC v3 tests
-│   ├── __init__.py
-│   ├── test_base.py         # Base class for v3 tests
-│   ├── aes_key_provider_test.py
-│   ├── aws_kms_key_provider_test.py
-│   └── most_recent_key_provider_test.py
-└── v4/                       # DB-ESDK v4 tests
-    ├── __init__.py
-    ├── test_base.py         # Base class for v4 tests
-    ├── raw_aes_keyring_test.py
-    ├── aws_kms_keyring_test.py
-    └── hierarchy_keyring_test.py
-
-resources/                    # Test data files
-├── single_attribute.json
-├── nested_attributes.json
-└── flat_attributes.json
+PERFORMANCE COMPARISON: DDBEC v3 vs DB-ESDK v4
+===============================================
+
+Operation | Data Type | Provider | DDBEC(v3) Mean (ms) | DB ESDK(v4) Mean (ms) | Difference (ms) | DB ESDK(v4) vs DDBEC(v3) (%)
 ```
 
-## Environment Variables
+### Generated Files
+- `raw_results_TIMESTAMP.csv/json` - Detailed benchmark data
+- `comparison_TIMESTAMP.csv/json` - Side-by-side comparison  
+- `report_TIMESTAMP.html` - Visual HTML report
 
-- `AWS_REGION`: AWS region for KMS operations (default: us-west-2)
-- `AWS_PROFILE`: AWS profile to use for credentials
+All files saved to `benchmark_results/` directory.
 
-## Troubleshooting
+## Understanding Results
 
-### AWS Credentials
-Ensure AWS credentials are properly configured:
-```bash
-aws configure list
-```
+### Performance Metrics
+- **Mean (ms)**: Average execution time per operation in milliseconds
+- **Difference (ms)**: Time difference (DB-ESDK v4 - DDBEC v3) in milliseconds
+- **Percentage**: Performance change (negative = v4 faster, positive = v4 slower)
 
-### KMS Key Access
-Verify access to the KMS key:
-```bash
-aws kms describe-key --key-id arn:aws:kms:us-west-2:370957321024:key/9d989aa2-2f9c-438c-a745-cc57d3ad0126
-```
+### AWS Credentials
 
-### Import Errors
-If you encounter import errors, ensure you're in the correct directory:
+Configure credentials using ada and assume the ddb role:
 ```bash
-cd PerfTest/runtimes/python/DynamoDbEncryption
+# Configure ada credentials
+ada credentials update --provider isengard --role=ToolsDevelopment --once \
+                       --account 370957321024 --profile=aws-crypto-tools-team+optools-ci-ToolsDevelopment
+
+# Set region and assume role for tests
+export AWS_REGION="us-west-2"
+TMP_ROLE=$(aws sts assume-role --role-arn "arn:aws:iam::370957321024:role/GitHub-CI-DDBEC-Dafny-Role-us-west-2" --role-session-name "${USER}-Tests-DBEC" --profile aws-crypto-tools-team+optools-ci-ToolsDevelopment)
+export TMP_ROLE
+export AWS_ACCESS_KEY_ID=$(echo "${TMP_ROLE}" | jq -r '.Credentials.AccessKeyId')
+export AWS_SECRET_ACCESS_KEY=$(echo "${TMP_ROLE}" | jq -r '.Credentials.SecretAccessKey')
+export AWS_SESSION_TOKEN=$(echo "${TMP_ROLE}" | jq -r '.Credentials.SessionToken')
+
+# Verify credentials are working
+aws sts get-caller-identity
 ```
-
-## Notes
-
-1. **KMS Rate Limits**: Be aware of AWS KMS rate limits when running extensive benchmarks
-2. **Network Latency**: KMS operations include network calls and may vary based on location
-3. **Hierarchy Keyring**: The current implementation uses a simplified version for testing
-4. **Most Recent Provider**: Uses an in-memory store for testing (production would use DynamoDB)
-
-## Contributing
-
-When adding new tests:
-1. Follow the existing pattern in `test_base.py`
-2. Add appropriate pytest markers (`@pytest.mark.v3` or `@pytest.mark.v4`)
-3. Include all three data types in parametrized tests
-4. Update this README with any new requirements
-
 ## License
 
-This project is licensed under the Apache License 2.0. See the LICENSE file in the root directory.
+Licensed under the Apache License 2.0.

PerfTest/runtimes/python/DynamoDbEncryption/pytest.ini

@@ -13,6 +13,8 @@ addopts =
     --benchmark-sort=name
     --benchmark-autosave
     --benchmark-save-data
+    --benchmark-min-rounds=3
+    --benchmark-rounds=5
 markers =
     benchmark: mark test as a benchmark test
     v3: mark test as DDBEC v3 specific