docs: enhance deployment guide with CLI options and restructure testing methods

Bob Strahan · Bob Strahan · commit 8b9116394bb9 · 2025-10-15T23:13:40.000Z
diff --git a/README.md b/README.md
@@ -31,6 +31,7 @@ White-glove customization, deployment, and integration support for production us
 
 - **Serverless Architecture**: Built entirely on AWS serverless technologies including Lambda, Step Functions, SQS, and DynamoDB
 - **Modular, pluggable patterns**: Pre-built processing patterns using state-of-the-art models and AWS services
+- **Command Line Interface**: Programmatic batch processing with evaluation framework and analytics integration
 - **Advanced Classification**: Support for page-level and holistic document packet classification
 - **Few Shot Example Support**: Improve accuracy through example-based prompting
 - **Custom Business Logic Integration**: Inject custom prompt generation logic via Lambda functions for specialized document processing
@@ -73,23 +74,50 @@ To quickly deploy the GenAI-IDP solution in your AWS account:
 
 ### Processing Your First Document
 
-After deployment, you can quickly process a document and view results:
+After deployment, choose the processing method that fits your use case:
 
-1. **Upload a Document**:
-   - **Via Web UI**: Open the Web UI URL from the CloudFormation stack's Outputs tab, log in, and click "Upload Document"
-   - **Via S3**: Upload directly to the S3 input bucket (find the bucket URL in CloudFormation stack Outputs)
+#### Method 1: Web UI (Interactive)
 
-2. **Use Sample Documents**:
-   - For Patterns 1 (BDA) and Pattern 2: Use [samples/lending_package.pdf](./samples/lending_package.pdf)
-   - For Pattern 3 (UDOP): Use [samples/rvl_cdip_package.pdf](./samples/rvl_cdip_package.pdf)
+1. Open the Web UI URL from CloudFormation stack Outputs
+2. Log in and click "Upload Document"
+3. Upload a sample document:
+   - For Patterns 1 & 2: [samples/lending_package.pdf](./samples/lending_package.pdf)
+   - For Pattern 3: [samples/rvl_cdip_package.pdf](./samples/rvl_cdip_package.pdf)
+4. Monitor processing and view results in the dashboard
+
+#### Method 2: Direct S3 Upload (Simple)
+
+1. Upload to the InputBucket (URL in CloudFormation Outputs)
+2. Monitor via Step Functions console
+3. Results appear in OutputBucket automatically
+
+#### Method 3: IDP CLI (Batch/Programmatic)
+
+For batch processing, automation, or evaluation workflows:
+
+```bash
+# Install CLI
+cd idp_cli && pip install -e .
+
+# Process documents
+idp-cli run-inference \
+    --stack-name <your-stack-name> \
+    --dir ./samples/ \
+    --monitor
 
-3. **Monitor Processing**:
-   - **Via Web UI**: Track document status on the dashboard
-   - **Via Step Functions**: Open the StateMachine URL from CloudFormation stack Outputs to observe workflow execution
+# Download results
+idp-cli download-results \
+    --stack-name <your-stack-name> \
+    --batch-id <batch-id> \
+    --output-dir ./results/
+```
 
-4. **View Results**:
-   - **Via Web UI**: Access processing results through the document details page
-   - **Via S3**: Check the output bucket for structured JSON files with extracted data
+**See [IDP CLI Documentation](./idp_cli/README.md)** for:
+- CLI-based stack deployment and updates
+- Batch document processing
+- Complete evaluation workflows with baselines
+- Athena and Agent Analytics integration
+- CI/CD integration examples
 
 See the [Deployment Guide](./docs/deployment.md#testing-the-solution) for more detailed testing instructions.
 
@@ -124,6 +152,7 @@ For detailed deployment and testing instructions, see the [Deployment Guide](./d
 
 - [Architecture](./docs/architecture.md) - Detailed component architecture and data flow
 - [Deployment](./docs/deployment.md) - Build, publish, deploy, and test instructions
+- [IDP CLI](./idp_cli/README.md) - Command line interface for batch processing and evaluation workflows
 - [Web UI](./docs/web-ui.md) - Web interface features and usage
 - [Agent Analysis](./docs/agent-analysis.md) - Natural language analytics and data visualization feature
 - [Custom MCP Agent](./docs/custom-MCP-agent.md) - Integrating external MCP servers for custom tools and capabilities
diff --git a/docs/deployment.md b/docs/deployment.md
@@ -7,24 +7,101 @@ This guide covers how to deploy, build, publish, and test the GenAI Intelligent
 
 ## Deployment Options
 
+The GenAI IDP Accelerator can be deployed using either the AWS CloudFormation console (recommended for first-time users) or the IDP CLI (recommended for automation and programmatic deployments).
+
 ### Administrator Access Requirements
 
 **Important**: Deploying the GenAI IDP Accelerator requires administrator access to your AWS account. However, for organizations that want to enable non-administrator users to deploy and manage IDP stacks, we provide an optional CloudFormation service role approach:
 
 - **For Administrators**: Use the deployment options below with your existing administrator privileges
 - **For Delegated Access**: See [iam-roles/cloudformation-management/README.md](../iam-roles/cloudformation-management/README.md) for instructions on provisioning a CloudFormation service role that allows non-administrator users to deploy and maintain IDP stacks without requiring administrator permissions
 
-### One-Click Deployment
+### Option 1: One-Click CloudFormation Console Deployment (Recommended for First-Time Users)
+
+1. Choose your region and click the Launch Stack button:
 
+| Region name | Region code | Launch |
+| ----------- | ----------- | ------ |
+| US West (Oregon) | us-west-2 | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main.yaml&stackName=IDP) |
 | US East (N.Virginia) | us-east-1 | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/aws-ml-blog-us-east-1/artifacts/genai-idp/idp-main.yaml&stackName=IDP) |
 
-3. Review the template parameters and provide values as needed
-4. Check the acknowledgment box and click **Create stack**
-5. Wait for the stack to reach the `CREATE_COMPLETE` state
+2. Review the template parameters and provide values as needed
+3. Check the acknowledgment box and click **Create stack**
+4. Wait for the stack to reach the `CREATE_COMPLETE` state (10-15 minutes)
 
 > **Note**: When the stack is deploying for the first time, it will send an email with a temporary password to the address specified in the AdminEmail parameter. You will need to use this temporary password to log into the UI and set a permanent password.
 
-## Option 2: Build Deployment Assets from Source Code
+---
+
+### Option 2: CLI-Based Deployment (Recommended for Automation)
+
+For programmatic deployment, updates, and batch processing, use the IDP CLI.
+
+#### Install the CLI
+
+```bash
+cd idp_cli
+pip install -e .
+```
+
+#### Deploy a New Stack
+
+```bash
+idp-cli deploy \
+    --stack-name my-idp-stack \
+    --pattern pattern-2 \
+    --admin-email your.email@example.com \
+    --max-concurrent 100 \
+    --wait
+```
+
+**What this does:**
+- Creates all CloudFormation resources (~120 resources)
+- Waits for deployment to complete (10-15 minutes)
+- Sends email with temporary admin password
+- Returns stack outputs including Web UI URL and bucket names
+
+#### Deploy with Custom Configuration
+
+```bash
+# Deploy with local config file (automatically uploaded to S3)
+idp-cli deploy \
+    --stack-name my-idp-stack \
+    --pattern pattern-2 \
+    --admin-email your.email@example.com \
+    --custom-config ./config_library/pattern-2/bank-statement-sample/config.yaml \
+    --wait
+```
+
+#### Update an Existing Stack
+
+```bash
+# Update configuration
+idp-cli deploy \
+    --stack-name my-idp-stack \
+    --custom-config ./updated-config.yaml \
+    --wait
+
+# Update parameters
+idp-cli deploy \
+    --stack-name my-idp-stack \
+    --max-concurrent 200 \
+    --log-level DEBUG \
+    --wait
+```
+
+**Benefits of CLI deployment:**
+- Scriptable and automatable
+- Version-controlled deployments
+- Rapid iteration for configuration testing
+- Integration with CI/CD pipelines
+- No manual console clicking required
+
+**For complete CLI documentation**, see [IDP CLI Documentation](../idp_cli/README.md).
+
+---
+
+## Option 3: Build Deployment Assets from Source Code
 
 ### Dependencies
 
@@ -246,7 +323,103 @@ To update an existing GenAIIDP deployment to a new version:
 
 ## Testing the Solution
 
-### Basic Test
+### Method 1: CLI-Based Batch Testing (Recommended for Automation)
+
+For batch processing, evaluation workflows, or automated testing:
+
+#### Quick Batch Test
+
+```bash
+# Install CLI
+cd idp_cli && pip install -e .
+
+# Process sample documents
+idp-cli run-inference \
+    --stack-name IDP \
+    --dir ./samples/ \
+    --batch-id sample-test \
+    --monitor
+```
+
+**What you'll see:**
+```
+✓ Uploaded 3 documents to InputBucket
+✓ Sent 3 messages to processing queue
+
+Monitoring Batch: sample-test
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+ Status Summary
+ ─────────────────────────────────────
+ ✓ Completed      3     100%
+ ⏸ Queued         0       0%
+```
+
+#### Download Results
+
+```bash
+# Download all results
+idp-cli download-results \
+    --stack-name IDP \
+    --batch-id sample-test \
+    --output-dir ./test-results/
+
+# View extraction data
+cat ./test-results/sample-test/lending_package.pdf/sections/1/result.json | jq .
+```
+
+#### Evaluation Workflow Testing
+
+Test accuracy with validated baselines:
+
+```bash
+# 1. Process documents initially
+idp-cli run-inference \
+    --stack-name IDP \
+    --dir ./test-docs/ \
+    --batch-id baseline-run \
+    --monitor
+
+# 2. Download and validate results
+idp-cli download-results \
+    --stack-name IDP \
+    --batch-id baseline-run \
+    --output-dir ./baselines/ \
+    --file-types sections
+
+# 3. Manually review and correct baselines
+# (Edit ./baselines/baseline-run/*/sections/*/result.json as needed)
+
+# 4. Create manifest with baselines
+cat > eval-manifest.csv << EOF
+document_path,baseline_source
+./test-docs/invoice.pdf,./baselines/baseline-run/invoice.pdf/
+./test-docs/w2.pdf,./baselines/baseline-run/w2.pdf/
+EOF
+
+# 5. Reprocess with evaluation
+idp-cli run-inference \
+    --stack-name IDP \
+    --manifest eval-manifest.csv \
+    --batch-id eval-test \
+    --monitor
+
+# 6. Download evaluation results
+idp-cli download-results \
+    --stack-name IDP \
+    --batch-id eval-test \
+    --output-dir ./eval-results/ \
+    --file-types evaluation
+
+# 7. Review accuracy metrics
+cat ./eval-results/eval-test/invoice.pdf/evaluation/report.md
+```
+
+**For complete evaluation workflow documentation**, see [IDP CLI - Complete Evaluation Workflow](../idp_cli/README.md#complete-evaluation-workflow).
+
+---
+
+### Method 2: Web UI Testing (Interactive)
+### Method 3: Direct S3 Upload Testing (Simple)
 
 1. Open the `S3InputBucketConsoleURL` and `S3OutputBucketConsoleURL` from the stack Outputs tab
 2. Open the `StateMachineConsoleURL` from the stack Outputs tab
@@ -256,7 +429,21 @@ To update an existing GenAIIDP deployment to a new version:
 4. Monitor the Step Functions execution to observe the workflow
 5. When complete, check the Output bucket for the structured JSON file with extracted fields
 
-### Testing via the UI
+#### Upload Multiple Files for Volume Testing
+
+To copy a sample file multiple times:
+
+```bash
+n=10
+for i in `seq 1 $n`; do 
+  aws s3 cp ./samples/lending_package.pdf \
+    s3://idp-inputbucket-kmsxxxxxxxxx/lending_package-$i.pdf
+done
+```
+
+---
+
+### Method 4: Web UI Testing (Interactive)
 
 1. Open the Web UI URL from the CloudFormation stack's Outputs tab
 2. Log in using your credentials (the temporary password from the email if this is your first login)
@@ -266,27 +453,6 @@ To update an existing GenAIIDP deployment to a new version:
 6. Follow the upload process and observe the document processing in the UI
 7. View the extraction results once processing is complete
 
-### Testing without the UI
-
-You can test the solution without using the UI through the following methods:
-
-1. Direct S3 uploads as described in the Basic Test section
-2. Using the AWS CLI to upload documents to the input bucket:
-
-   ```bash
-   aws s3 cp ./samples/lending_package.pdf s3://idp-inputbucket-kmsxxxxxxxxx/
-   ```
-
-3. Using the AWS SDK in your application code to programmatically send documents for processing
-
-### Upload Multiple Sample Files
-
-To copy a sample file multiple times for testing:
-
-```bash
-n=10
-for i in `seq 1 $n`; do aws s3 cp ./samples/lending_package.pdf s3://idp-inputbucket-kmsxxxxxxxxx/lending_package-$i.pdf; done
-```
 
 ### Testing Individual Lambda Functions Locally
 
