More cleaning

Author: lmfarley10

.gitignore

@@ -49,13 +49,21 @@ __pycache__/
 *.pyd
 *.egg-info
 
-# Images 
+# Images (trusted & untrusted.png are required for client server)
+# vbcs app requires the excepted images as well 
 *.png
+!clientApp/trusted.png
+!clientApp/untrusted.png 
+!/vbcs_oda_archives/vbcs_apps/vbcs-askdata/webApps/nl2sqlbot/resources/images/*
+!nl2sql-main/vbcs_oda_archives/vbcs_apps/vbcs-askdata/webApps/nl2sqlbot/resources/images
+
 *.jpeg
 *.jpg
+!/vbcs_oda_archives/vbcs_apps/vbcs-askdata/webApps/nl2sqlbot/resources/images/*
 *.gif
 *.webp
 *.ico
+!/vbcs_oda_archives/vbcs_apps/vbcs-askdata/webApps/nl2sqlbot/resources/images/*
 
 *.docx 
 
@@ -66,6 +74,7 @@ __pycache__/
 *.xlsx
 *.zip
 /rest/nl2sql-trust/.coverage
+*.tgz
 
 # WebSDK should be included in VBCS Applications
 /oda/websdk-apache-deploy/
@@ -179,8 +188,10 @@ mkdocs/docs/deployment/VBCS.md
 # devops sensitive info 
 /rest/script/devops
 
+/rest/script/docker/
 # docker artifactory info 
 /rest/script/docker/artifacts/
+/rest/script/docker/artifacts/promote_apex.sh
 
 # oracle instant client reference?
 /rest/script/docker/oracle-instantclient-23ai-ol8.repo
@@ -190,6 +201,7 @@ mkdocs/docs/deployment/VBCS.md
 /rest/script/docker/run_dev.sh
 /rest/script/docker/run.sh
 
+/rest/script/docker_released/
 /rest/script/docker_released/artifacts
 
 # another oracle instant client reference? Add link to public download instead?
@@ -202,6 +214,7 @@ mkdocs/docs/deployment/VBCS.md
 
 # ubuntu image? Is this needed? Replace with olinux instead? 
 /rest/script/dockerssh/Dockerfile
+/rest/script/dockerssh/
 
 # Reference to google apis, twitter open source, do we need this? 
 /rest/script/html_client/

README.md

@@ -39,3 +39,105 @@ Licensed under the Universal Permissive License (UPL), Version 1.0.
 See [LICENSE](LICENSE.txt) for more details.
 
 ORACLE AND ITS AFFILIATES DO NOT PROVIDE ANY WARRANTY WHATSOEVER, EXPRESS OR IMPLIED, FOR ANY SOFTWARE, MATERIAL OR CONTENT OF ANY KIND CONTAINED OR PRODUCED WITHIN THIS REPOSITORY, AND IN PARTICULAR SPECIFICALLY DISCLAIM ANY AND ALL IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE.  FURTHERMORE, ORACLE AND ITS AFFILIATES DO NOT REPRESENT THAT ANY CUSTOMARY SECURITY REVIEW HAS BEEN PERFORMED WITH RESPECT TO ANY SOFTWARE, MATERIAL OR CONTENT CONTAINED OR PRODUCED WITHIN THIS REPOSITORY. IN ADDITION, AND WITHOUT LIMITING THE FOREGOING, THIRD PARTIES MAY HAVE POSTED SOFTWARE, MATERIAL OR CONTENT TO THIS REPOSITORY WITHOUT ANY REVIEW. USE AT YOUR OWN RISK. 
+## Deployment Steps
+
+1. Deploy Terraform Script
+2. Configure Business App
+    - Configure Business DB
+    - Configure Trust DB
+3. Configure IDCS App
+4. Configure API Gateway
+5. Configure Trust App
+6. Configure ODA Skills
+7. Configure VB Apps 
+
+### TBD 
+8. Deploy Trust Framework
+9. Deploy APEX
+
+## Overview
+
+I ran a dry deployment using the cleaned repo on our tenancy. Listed are some of my findings & outstanding questions. 
+
+- I was able to get the client business app deployed on our tenancy. A few things I noticed - 
+    - The code expects a llama model, at least for the on demand configuration
+    - Would have to be refactored if using e.g. cohere on demand 
+- The existing documentation suggests deploying api gateway on private subnet due to lack of authentication. 
+    - I was able to deploy the api gateway on public subnet with oauth2.0 authentication to idcs server and invoke from oda with token, so not sure why private subnet deployment for api gateway is suggested
+- The existing code expects api keys and wallets configured directly on the server, at least for the client business engine 
+- Redis is required but no instructions given 
+- There was no example data given for the vendors table, so I exported the table from the main deployment as a csv and imported to ours
+    - Vendors table is for client business db 
+    - How do we want to provide this csv?
+- The business app is dependent on the trust db (trust library), i.e. the trusted prompts, which isn't intuitive 
+    - The code expects the TRUST_LIBRARY table to have at least one entry, otherwise it bombs 
+    - I provided a sample entry in the sql file 
+The sql for the client business db is outside the clientapp directory, & there are no instructions given on which db to upload to (upload to business db)
+
+The business app is exposed on an API Gateway with the main entry /prompt, which maps to <business-app-ip>:8000... this will be used by ODA app.
+
+There are a handful of files required and provided within the repo, but no instructions on how/when to use them. 
+Examples include : 
+1. Under sql, nl2sql_datamodel_schema.sql needds to be ran in the trust database 
+    - Was able to run the client engine without the sql/sample_setup_ras.sql file. 
+    - This is a prequisite to run the business client engine, which isn't intuitive 
+2. Within the rest (trust) directory there are docker files... I think we can remove these? 
+    - Haven't tested full trust deployment yet.
+
+Some files I don't think are necessary at all 
+1. autoprompt 
+2. mkdocs... this is documentation we can perhaps repurpose (at least the md files)
+
+Below are the various requirements and findings from deploying the infrastructure components on our tenancy - 
+
+### ODA 
+
+1. Configure IDCS Server first 
+    - ODA skill expects a token to execute
+    - Configured manually for now. Expects ODA client callback to work
+2. Configured API gateway with nl2sql engine backend - provide api gateway endpoint in skill
+    - Configured bearer token with IDCS app 
+3. There are two skill zips that don't seem to be used and not referenced in the documentation - 
+    - oda-skill-EmbeddedCont.changeit.zip
+    - oda-skill-ExtOracleFn.changeit.zip 
+
+### VBCS 
+
+1. The zipped askdata vbcs app in here looks to be outdated. 
+    - It's a more simple implementation, which might be best for now
+2. Reference to websocket - to be deprecated? Need to test if working without 
+    - Tested, seems to be unnecessary
+3. App import relies on various images in the directory... do we want to provide these images in the repo?
+    - I'm going to include them for now, since they are dependencies
+
+### API Gateway 
+1. API Gateway is configured with Single Authentication OAuth2.0 which requires a vault. 
+2. The existing mkdocs documentation said to deploy the api gateway to private subnet due to lack of authentication... authentication can be added to api gateway on public subnet 
+
+### Redis 
+1. Redis is required but there were no instructions given on configuration
+
+### Generative AI 
+1. Existing deployment was using dedicated ai cluster/data science. I was able to get it to work with on demand model. 
+    - Code is hard coded to handle only the llama models. Cohere models would require refactoring. 
+
+### NL2SQL Business Engine 
+1. The code is currently expecting the user to upload their own api keys 
+2. Database wallets need to be uploaded manually 
+3. png files trusted & untrusted.png are required to run the server.
+4. autoprompt - need to test if working without (deprecated?)
+
+### Client Database
+1. The vendors table didn't have any example data. I had to export the table from the existing implementation and import as csv 
+    - Do we want to include this csv in the repo, or add to object storage bucket and provide link? 
+
+### Trust Database
+1. Engine code expects at least one entry in Trust library table before execution
+    - Provided an example entry with sample embedding
+
+### Trust REST Framework
+Haven't gotten this far, but the basic prompt endpoint does use the trust library table. 
+
+## APEX
+tbd ... do we want to include this in the first version? 
+

clientApp/ConfigFile.properties

@@ -34,15 +34,18 @@ max.resultset=50
 filter.upn=notused
 filter.ignoreupntablelist=notused
 [vbcs]
+# endpoint url should be the root e.g. endpoint.url=https://<your-vb>.oraclecloud.com/ic/builder/rt/
 endpoint.url=<vbcs-endpoint-url>
+# graph url should be relative to endpoint url e.g. <your-graph-app>/1.0/webApps/nl2sql_interactivegraph/
 graph_app.url=<graph-app-url>
+# table graph should also be relative
 idata_app.url=<table-graph-url>
 [OCI]
 #Sao Paulo
 #serviceendpoint.url=https://inference.generativeai.sa-saopaulo-1.oci.oraclecloud.com
 #Scenario2
 # DAC or DS or GAI
-serviceendpoint.active=DS
+serviceendpoint.active=GAI
 serviceendpoint.ds_endpt=<service-endpoint-url>
 serviceendpoint.ds_model=<service-endpoint-ds-model>
 serviceendpoint.url=https://inference.generativeai.us-chicago-1.oci.oraclecloud.com

clientApp/helper_methods.py

@@ -18,8 +18,8 @@
 config = configparser.RawConfigParser()
 config.read('ConfigFile.properties')
 
-rediscache = redis.StrictRedis(host=config.get('RedisSection', 'url'), charset='utf-8', ssl=True, decode_responses=True, port=config.get('RedisSection', 'port'))
-rediscache_obj = redis.StrictRedis(host=config.get('RedisSection', 'url'), charset='utf-8', ssl=True, decode_responses=False, port=config.get('RedisSection', 'port'))
+rediscache = redis.StrictRedis(host=config.get('RedisSection', 'url'), ssl=True, decode_responses=True, port=config.get('RedisSection', 'port'))
+rediscache_obj = redis.StrictRedis(host=config.get('RedisSection', 'url'), ssl=True, decode_responses=False, port=config.get('RedisSection', 'port'))
 queryttl=2*24*60*60
 
 def setup_logger(log_file_name='application.log'):

clientApp/trusted.png

@@ -2997,6 +2997,7 @@ Insert into  ACCOUNT_PAYABLES_TBL (VENDOR_NAME,VENDOR_NUMBER,VENDOR_SITE_DETAILS
 Insert into  ACCOUNT_PAYABLES_TBL (VENDOR_NAME,VENDOR_NUMBER,VENDOR_SITE_DETAILS,INVOICE_NUMBER,INVOICE_DATE,GL_DATE,INVOICE_TYPE,DUE_DATE,PAST_DUE_DAYS,AMOUNT_DUE) values ('GE Capital',1001,'MILANO ERS Milano','ERS-7048-212135',to_date('01-SEP-23','DD-MON-RR'),to_date('06-SEP-23','DD-MON-RR'),'STANDARD',to_date('04-SEP-23','DD-MON-RR'),255,7928.64);
 Insert into  ACCOUNT_PAYABLES_TBL (VENDOR_NAME,VENDOR_NUMBER,VENDOR_SITE_DETAILS,INVOICE_NUMBER,INVOICE_DATE,GL_DATE,INVOICE_TYPE,DUE_DATE,PAST_DUE_DAYS,AMOUNT_DUE) values ('GE Capital',1001,'MILANO ERS Milano','ERS-7057-212300',to_date('01-SEP-23','DD-MON-RR'),to_date('07-SEP-23','DD-MON-RR'),'STANDARD',to_date('04-SEP-23','DD-MON-RR'),255,7928.64);
 Insert into  ACCOUNT_PAYABLES_TBL (VENDOR_NAME,VENDOR_NUMBER,VENDOR_SITE_DETAILS,INVOICE_NUMBER,INVOICE_DATE,GL_DATE,INVOICE_TYPE,DUE_DATE,PAST_DUE_DAYS,AMOUNT_DUE) values ('Advanced Network Devices',1013,'MILANO ERS Milano','ERS-7041-212016',to_date('01-SEP-23','DD-MON-RR'),to_date('05-SEP-23','DD-MON-RR'),'STANDARD',to_date('04-SEP-23','DD-MON-RR'),255,21960.29);
+commit;
 Insert into  ACCOUNT_PAYABLES_TBL (VENDOR_NAME,VENDOR_NUMBER,VENDOR_SITE_DETAILS,INVOICE_NUMBER,INVOICE_DATE,GL_DATE,INVOICE_TYPE,DUE_DATE,PAST_DUE_DAYS,AMOUNT_DUE) values ('Advanced Network Devices',1013,'MILANO ERS Milano','ERS-7058-212383',to_date('01-SEP-23','DD-MON-RR'),to_date('08-SEP-23','DD-MON-RR'),'STANDARD',to_date('04-SEP-23','DD-MON-RR'),255,21960.29);
 Insert into  ACCOUNT_PAYABLES_TBL (VENDOR_NAME,VENDOR_NUMBER,VENDOR_SITE_DETAILS,INVOICE_NUMBER,INVOICE_DATE,GL_DATE,INVOICE_TYPE,DUE_DATE,PAST_DUE_DAYS,AMOUNT_DUE) values ('GE Capital',1001,'MILANO ERS Milano','ERS-7033-211643',to_date('26-AUG-23','DD-MON-RR'),to_date('29-AUG-23','DD-MON-RR'),'STANDARD',to_date('29-AUG-23','DD-MON-RR'),261,3567.89);
 Insert into  ACCOUNT_PAYABLES_TBL (VENDOR_NAME,VENDOR_NUMBER,VENDOR_SITE_DETAILS,INVOICE_NUMBER,INVOICE_DATE,GL_DATE,INVOICE_TYPE,DUE_DATE,PAST_DUE_DAYS,AMOUNT_DUE) values ('GE Capital',1001,'MILANO ERS Milano','ERS-7033-211642',to_date('25-AUG-23','DD-MON-RR'),to_date('29-AUG-23','DD-MON-RR'),'STANDARD',to_date('28-AUG-23','DD-MON-RR'),262,8675.79);

clientApp/untrusted.png

@@ -1,13 +1,13 @@
 As admin user
 
-create user app_data identified by H0tsummer2025
+create user app_data identified by <password>
 default tablespace data;
 
 grant connect, resource to app_data;
 alter user app_data quota unlimited on data;
 
 drop user ras_admin cascade;
-create user ras_admin identified by H0tsummer#2025
+create user ras_admin identified by <password>
 default tablespace data;
 grant connect, resource to ras_admin;
 alter user ras_admin quota unlimited on data;
@@ -57,20 +57,20 @@ grant nl2sql_role to inv_type_redact_role;
 
 SELECT STANDARD_HASH('test', 'SHA256') FROM dual;
 
-exec  sys.xs_principal.create_user(name => 'rajarora', schema => 'app_data');
-exec  sys.xs_principal.set_password('rajarora', 'G0ingtothest#rs');
-exec  sys.xs_principal.grant_roles('rajarora', 'XSCONNECT');
-exec  sys.xs_principal.grant_roles('rajarora', 'superuser_app_role');
+exec  sys.xs_principal.create_user(name => '<user>', schema => 'app_data');
+exec  sys.xs_principal.set_password('<user>', '<password>');
+exec  sys.xs_principal.grant_roles('<user>', 'XSCONNECT');
+exec  sys.xs_principal.grant_roles('<user>', 'superuser_app_role');
 
-exec  sys.xs_principal.create_user(name => 'rajarora1', schema => 'app_data');
-exec  sys.xs_principal.set_password('rajarora1', 'G0ingtothest#rs');
-exec  sys.xs_principal.grant_roles('rajarora1', 'XSCONNECT');
-exec  sys.xs_principal.grant_roles('rajarora1', 'inv_type_redact_role');
+exec  sys.xs_principal.create_user(name => '<user2>', schema => 'app_data');
+exec  sys.xs_principal.set_password('<user2>', '<password-user-2>');
+exec  sys.xs_principal.grant_roles('<user2>', 'XSCONNECT');
+exec  sys.xs_principal.grant_roles('<user2>', 'inv_type_redact_role');
 
-exec  sys.xs_principal.create_user(name => 'rajarora2', schema => 'app_data');
-exec  sys.xs_principal.set_password('rajarora2', 'G0ingtothest#rs');
-exec  sys.xs_principal.grant_roles('rajarora2', 'XSCONNECT');
-exec  sys.xs_principal.grant_roles('rajarora2', 'inv_type_limited_role');
+exec  sys.xs_principal.create_user(name => '<user3>', schema => 'app_data');
+exec  sys.xs_principal.set_password('<user3>', '<password-user-3>');
+exec  sys.xs_principal.grant_roles('<user3>', 'XSCONNECT');
+exec  sys.xs_principal.grant_roles('<user3>', 'inv_type_limited_role');
 
 
 

data/demodata.sql

@@ -0,0 +1,16 @@
+{
+  "type" : "USER",
+  "autoCompleteTrainingStatus" : "NO_DATA",
+  "category" : "null",
+  "description" : "calls Rest EP-MVP1(from v5g - APIGW(.252), auth, group, domain, odaclarify, EH+, FBFmtd, CC, dm_spc)",
+  "dialogVersion" : "2.0",
+  "displayName" : "callGenAiSQLEndPointv5h",
+  "intentTrainer" : "Sakura",
+  "multilingualMode" : "NATIVE",
+  "name" : "callGenAiSQLEndPointv5h",
+  "platformVersion" : "25.04",
+  "primaryLanguageTag" : "en",
+  "requiresAutoCompleteTraining" : false,
+  "trackingId" : "F59CDEF3-58DE-4895-B72F-824DE1EF2EDC",
+  "version" : "1.0"
+}