Fix output + docs (#69)

* fix: fix output

* fix: fix output

* docs: update doc for rc

* docs: improve readme.md

Author: vburckhardt

.secrets.baseline

@@ -3,7 +3,7 @@
     "files": "go.sum|^.secrets.baseline$",
     "lines": null
   },
-  "generated_at": "2024-05-07T12:00:08Z",
+  "generated_at": "2024-05-08T11:42:40Z",
   "plugins_used": [
     {
       "name": "AWSKeyDetector"
@@ -76,34 +76,7 @@
       "name": "TwilioKeyDetector"
     }
   ],
-  "results": {
-    "README.md": [
-      {
-        "hashed_secret": "bbc4e9d52252171a3a306be55086c65b126189e8",
-        "is_secret": false,
-        "is_verified": false,
-        "line_number": 38,
-        "type": "Secret Keyword",
-        "verified_result": null
-      },
-      {
-        "hashed_secret": "d9e9019d9eb455a3d72a3bc252c26927bb148a10",
-        "is_secret": false,
-        "is_verified": false,
-        "line_number": 55,
-        "type": "Secret Keyword",
-        "verified_result": null
-      },
-      {
-        "hashed_secret": "b13d7622394e85c3b2694f426bc096b093764462",
-        "is_secret": false,
-        "is_verified": false,
-        "line_number": 59,
-        "type": "Secret Keyword",
-        "verified_result": null
-      }
-    ]
-  },
+  "results": {},
   "version": "0.13.1+ibm.62.dss",
   "word_list": {
     "file": null,

README.md

@@ -1,88 +1,121 @@
 # Retrieval Augmented Generation Pattern for Watsonx on IBM Cloud
 
-The following [deployable architecture](https://cloud.ibm.com/docs/secure-enterprise?topic=secure-enterprise-understand-module-da#what-is-da) automates the deployment of a sample GenAI Pattern on IBM Cloud, including all underlying infrastructure. This architecture implements the best practices for Watsonx GenAI Pattern deployment on IBM Cloud, as described in the [reference architecture](https://cloud.ibm.com/docs/pattern-genai-rag?topic=pattern-genai-rag-genai-pattern).
+The following [deployable architecture](https://cloud.ibm.com/docs/secure-enterprise?topic=secure-enterprise-understand-module-da#what-is-da) automates the deployment of a sample GenAI Pattern on IBM Cloud, including all underlying IBM Cloud infrastructure. This architecture implements the best practices for Watsonx GenAI Pattern deployment on IBM Cloud, as described in the [reference architecture](https://cloud.ibm.com/docs/pattern-genai-rag?topic=pattern-genai-rag-genai-pattern).
+
+This deployable architecture provides a comprehensive foundation for trust, observability, security, and regulatory compliance by configuring the IBM Cloud account to align with compliance settings, deploying key and secret management services, and deploying the infrastructure to support CI/CD/CC pipelines for secure application lifecycle management. These pipelines facilitate the deployment of the application, vulnerability checks, and auditability, ensuring a secure and trustworthy deployment of Generative AI applications on IBM Cloud.
+
+# Objective and Benefits
+
+This deployable architecture is designed to showcase a fully automated deployment of a retrieval augmented generation application through IBM Cloud Project, providing a flexible and customizable foundation for your own Watson-based application deployments on IBM Cloud. This architecture deploys the following [sample application](https://github.com/IBM/gen-ai-rag-watsonx-sample-application) by default.
+
+By leveraging this architecture, you can accelerate your deployment and tailor it to meet your unique business needs and enterprise goals.
+
+By using this architecture, you can:
+
+* Establish Trust: The architecture ensures trust by configuring the IBM Cloud account to align with compliance settings as defined in the [Financial Services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-about) framework.
+* Ensure Observability: The architecture provides observability by deploying services such as IBM Log Analysis, IBM Monitoring, IBM Activity Tracker, and log retention through Cloud Object Storage buckets.
+* Implement Security: The architecture ensures security by deploying IBM Key Protect and IBM Secrets Manager.
+* Achieve Regulatory Compliance: The architecture ensures regulatory compliance by implementing CI/CD/CC pipelines, along with IBM Security Compliance Center (SCC) for secure application lifecycle management.
+
 
 # Deployment Details
 
-To run the full stack, follow these steps. These steps will be updated as development progresses on the stack and underlying deployable architectures.
+To deploy this architecture, follow these steps.
+
+## 1. Prerequisites in Target Account
 
-## 1. Deploy the Stack in a New Project from Catalog
+Before deploying the deployable architecture, ensure you have:
 
-Catalog URL: https://cloud.ibm.com/catalog/7df1e4ca-d54c-4fd0-82ce-3d13247308cd/architecture/Retrieval_Augmented_Generation_Pattern-5fdd0045-30fc-4013-a8bc-6db9d5447a52?bss_account=9f9af00a96104f49b6509aa715f9d6a5
+* Created an API key in the target account with sufficient permissions. See [instructions](https://cloud.ibm.com/docs/account?topic=account-userapikey&interface=ui) Note the API key, as it will be used later. On evaluation environments, you may simply grant `Administrator` role on `IAM Identity Service`, `All Identity and Access enabled services` and `All Account Management` services. If you need to narrow down further access, for a production environment for instance, the minimum level of permissions is indicated in the [Permission tab](https://cloud.ibm.com/catalog/7df1e4ca-d54c-4fd0-82ce-3d13247308cd/architecture/Retrieval_Augmented_Generation_Pattern-5fdd0045-30fc-4013-a8bc-6db9d5447a52?kind=terraform&format=stack&version=32db3936-66fe-4b04-bbc8-61d5676a89f1#permissions) of the deployable architecture.
+* A signing key, which is the base64 key obtained from `gpg --gen-key` (if not generated before or expired) and then exported via `gpg --export-secret-key <Email Address> | base64` command. See the [devsecops image signing page](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-image-signing#cd-devsecops-gpg-export) for details. Key note of the key for later.
+* (Optional) Installed the IBM Cloud CLI's Project add-on using the `ibmcloud plugin install project` command. More information is available [here](https://cloud.ibm.com/docs/cli?topic=cli-projects-cli).
 
-Click the "Add to Project" button and select "Create in new project."
+## 2. Deploy the Stack in a New Project from Catalog
 
-## 2. Prerequisites in Target Account
+* Locate the [tile](https://cloud.ibm.com/catalog/7df1e4ca-d54c-4fd0-82ce-3d13247308cd/architecture/Retrieval_Augmented_Generation_Pattern-5fdd0045-30fc-4013-a8bc-6db9d5447a52) for the Deployable Architecture in the IBM Cloud Catalog.
+* Click the "Add to project" button.
 
-Before deploying the stack, ensure you have:
+    ![image](./images/min/1-catalog.png)
 
-* Created an API key in the target account with sufficient permissions. Note the API key, as it will be used later. For now, grant it admin privileges. The exact permissions required will be refined in future versions.
-* Installed the IBM Cloud CLI's Project add-on using the `ibmcloud plugin install project` command. More information is available here: https://cloud.ibm.com/docs/cli?topic=cli-projects-cli
+* Select "Create new" and input IBM Cloud Project details.
+   - Name and description. eg: "Retrieval Augmented Generation Pattern"
+   - Region and resource group for the project. Note that this is the region and resource group where the runtime (IBM Cloud Schematics) executiong the automation code is located. You can still deploy the resources to any region, any resource group, and any account.
+   - Configuration name - for example: "dev" or "prod".
+
+        ![project](./images/min/2-project.png)
+
+* Click "Add" to complete.
 
 ## 3. Set the Input Configuration for the Stack
 
-* Clone this repository locally.
-* Create a file named ".def.json" with the following content:
-
-**Important**:
-* Ensure the region is either us-south or eu-de, as Watsonx can only be deployed in those two locations for now.
-* Ensure that the prefix is globally unique. It is used for the container registry namespace (which needs to be globally unique) in this alpha version.
-* If specifying `existing_secrets_manager_crn`, the `ibmcloud_api_key` that is passed as an input must have the documented read and write access to the instance.
-* If specifying `existing_secrets_manager_crn`, ensure that the default security group does not contain secrets named `signing-key` and `ibmcloud-api-key`. The RAG DA currently always attempts to create a secret with those names (temporary issue - to be fixed).
-* The signing key is the base64 key obtained from `gpg --gen-key` (if not generated before or expired) and then exported via `gpg --export-secret-key <Email Address> | base64` command. See https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-image-signing#cd-devsecops-gpg-export for details.
-
-```json
-{
-    "inputs": {
-        "prefix": "<prefix for resources name - ensure unique>",
-        "ibmcloud_api_key": "<API Key of the target account with sufficient permissions>",
-        "resource_group_name": "<target resource group - name of a new resource group that the stack will create>",
-        "region": "<region where all resources are deployed>",
-        "sample_app_git_url": "https://github.com/IBM/gen-ai-rag-watsonx-sample-application",
-        "watsonx_admin_api_key": "<optional - admin key to use for Watsonx if different from ibmcloud_api_key>",
-        "signing_key": "signing key used to sign build artifacts",
-        "existing_secrets_manager_crn": "<optional> - reuse an existing secret manager instance",
-        "enable_platform_logs_metrics": "<optional> - set to true to enable observability instance to capture regional logs"
-    }
-}
-```
+After completing `Step 2 - Deploy the Stack in a New Project from Catalog`, you are directed to a page allowing you to enter the configuration for you deployment:
+* Under Security -> Authentication, enter the API Key from the prereqs in the `api_key` field.
+* Under Requires, input the signing key in the `signing_key` field.
 
-Example:
-```json
-{
-    "inputs": {
-        "prefix": "0418",
-        "ibmcloud_api_key": "<your api key>",
-        "resource_group_name": "stack-service-rg",
-        "region": "eu-de",
-        "sample_app_git_url": "https://github.com/IBM/gen-ai-rag-watsonx-sample-application",
-        "watsonx_admin_api_key": "<optional - admin key to use for Watsonx if different from ibmcloud_api_key>",
-        "signing_key": "signing key used to sign build artifacts",
-        "enable_platform_logs_metrics": "false",
-        "existing_secrets_manager_crn": "crn:v1:bluemix:public:secrets-manager:us-south:a/190c293e9fda4c6684b5acf4b17871b8:14580411-4fa2-42d3-af3f-ab7fc6371b6d::"
-    }
-}
-```
+You may explore the other available inputs, such as the region and resource group name (under optional tab), leave them as is, or modify them as needed.
+
+![inputs](./images/min/3-inputs.png)
+
+Once ready, click the "Save" button at the top of the screen.
+
+## 4. Deploy the Architecture
+
+Navigate to the project deployment view by clicking the project name in the breadcrumb menu.
 
+![menu](./images/min/4-bread.png)
 
-## 4. Run ./deploy-many.sh
 
-* Ensure you are logged in to the account containing the Cloud project with the stack using `ibmcloud login --sso`.
-* Execute `./deploy-many.sh` with the project name, stack name (name of the configuration given when deploying the tile), and optional configuration name pattern. The selected non-stack configurations will be processed by their name in alphabetical order. Using the configuration name pattern (regex can be used - make sure to enclose it in quotes), you can choose which configurations are deployed.
+You should be directed to a screen looking like:
 
-Example 1 - Update stack inputs for stack configuration `RAG` and process all non-stack configurations in the project:
+![validate](./images/min/5-validate.png)
+
+Two approaches to deploy the architecture:
+1. Through the UI - note that using the UI requires a large number of clicks each elements on the architecture stack.
+2. Automated - se the  script `./deploy-many.sh` is provided.
+
+### Approach 1: Deployment through the UI
+
+1. Click on validate
+
+    ![validate button](./images/min/5b-validate.png)
+
+2. Wait for validation
+
+    ![validation](./images/min/6-validation.png)
+
+3. Approve and click the deploy button
+
+    ![deploy](./images/min/7-deploy.png)
+
+4. Wait for deployment
+
+5. Repeat step 1 for the next configuration in the architecture.
+
+### Approach 2: Run ./deploy-many.sh
+
+* Clone the repository at https://github.com/terraform-ibm-modules/stack-retrieval-augmented-generation/tree/main
+* Ensure you are logged in to the account containing the Cloud project with the stack using `ibmcloud login`.
+* Execute `./deploy-many.sh` with the project name, stack name, and optional configuration name pattern. The selected non-stack configurations will be processed by their name in alphabetical order. Using the configuration name pattern (regex can be used - make sure to enclose it in quotes), you can choose which configurations are deployed.
+
+Example 1 - Update stack inputs for stack configuration `dev` and process all non-stack configurations in the project:
 ```bash
-./deploy-many.sh my-test-project RAG
+./deploy-many.sh my-test-project dev
 ```
 
 Example 2 - Update stack inputs and process some configurations in the project:
 ```bash
-./deploy-many.sh my-test-project RAG 'RAG-1|RAG-4|RAG-5'
+./deploy-many.sh my-test-project dev 'RAG-1|RAG-4|RAG-5'
 ```
 
 Example 3 - Simulate updating stack inputs and validating some configurations in the project in dry-run mode (no changes or actual validation or deployments are done):
 ```bash
-DRY_RUN=true ./deploy-many.sh my-test-project RAG 'RAG-1|RAG-4|RAG-5'
+DRY_RUN=true ./deploy-many.sh my-test-project dev 'RAG-1|RAG-4|RAG-5'
 ```
 
-Tips: If deployment fail in one of the DA, you may re-run the script as is. It will skip existing installed configuration.
+Tips: If deployment fail for one of the configuration, you may re-run the script as is. It will skip existing installed configurations and continue where it last failed.
+
+## 5. Post deployment steps
+
+At this point, all of the infrastructure is deployed in the target account. The initial build for the sample app has also started in the DevOps service, and the app is deployed to IBM Cloud Code Engine.
+
+There are some remaining steps, specific to the sample app, that can be completed to fully enable Watson assistant in the app. To complete the installation - follow the steps at https://github.com/IBM/gen-ai-rag-watsonx-sample-application/blob/main/artifacts/artifacts-README.md

ibm_catalog.json

@@ -104,8 +104,24 @@
 					"architecture": {
 						"features": [
 							{
-								"title": "Deploy a banking retrieval augmented generation (RAG) app to IBM Cloud Code Engine using Continous Delivery.",
-								"description": ""
+								"title": "Retrieval Augmented Generation sample pattern.",
+								"description": "Deploy a banking retrieval augmented generation (RAG) sample application to IBM Cloud Code Engine using Continous Delivery."
+							},
+							{
+								"title": "Ensure Observability",
+								"description": "The architecture provides observability by deploying services such as IBM Log Analysis, IBM Monitoring, IBM Activity Tracker, and log retention through Cloud Object Storage buckets."
+							},
+							{
+								"title": "Implement Security",
+								"description": "The architecture ensures security by deploying IBM Key Protect and IBM Secrets Manager."
+							},
+							{
+								"title": "Achieve Regulatory Compliance",
+								"description": "The architecture ensures regulatory compliance by implementing CI/CD/CC pipelines, along with IBM Security Compliance Center (SCC) for secure application lifecycle management."
+							},
+							{
+								"title": "Establish Trust",
+								"description": "The architecture ensures trust by configuring the IBM Cloud account to align with compliance settings as defined in the Financial Services framework."
 							}
 						],
 						"diagrams": [