Merge branch 'main' into open_source_bp_2

Author: bobpeulen

data-platform/analytical-data-platform-lakehouse/README.md

@@ -55,6 +55,8 @@ Reviewed: 18.01.2024
   - Blog post describing and comparing the different insert methods in Autonomous Database to support low latency data ingestion for IoT workloads.
 - [Managing Active Metadata with Oracle Data Platform](https://gianlucarossi06.github.io/data-organon/2024/05/31/Active-Metadata-4-OCI-Data-Platform.html)
   - Blog post describing how to define and store active metadata using OCI Data Platform, using a practical example. Active Metadata can be anything stored as custom properties in a data catalog allowing users to understand, for instance, data freshness.
+- [Streaming IoT Data into Object Storage with Streaming service](https://jakubillner.github.io/2024/06/28/streaming-ingest.html)
+  - Blog post describing how to ingest and store IoT data for an analytical workload using OCI Streaming, Connector Hub, and Object Storage.
 
 
 ## YouTube

data-platform/analytics/oracle-analytics-cloud/README.md

@@ -13,6 +13,7 @@ Reviewed: 31.10.2023
 ## Specialists Blogs for various features & functionality
 |Content Link |Functionality|Descripton|
 | ------------ |------------|----------|
+|[Unleash the Power of Template Viewer: Streamlined Testing for Flawless Oracle Analytics Publisher Reports](https://www.linkedin.com/pulse/unleash-power-template-viewer-streamlined-testing-flawless-kasetty-bxiqc/)|Oracle Analytics Publisher Template Viewer|How to leverage Template Viewer to test Oracle Analytics Publisher templates.
 |[Leverage the OCI Modern Data Platform to implement an Enterprise Analytics Solution](https://blogs.oracle.com/coretec/post/leverage-oci-modern-data-platform-to-implement-enterprise-analytics-solution)|OAC-Enterprise Analytics Solution |How to leverage the OCI modern data platform to implement an enterprise analytics solution.
 |[Top 5 reasons Oracle Analytics Cloud stands apart in the ML/AI Analytics landscape](https://blogs.oracle.com/analytics/post/top-5-reasons-oracle-analytics-cloud-stands-apart-in-the-mlai-analytics-landscape)|OAC Machine Learning|What are primary reasons to choose Oracle Analytics Cloud (OAC) from an ML/AI perspective.
 |[Oracle Analytics Cloud: Set up and configure Oracle Analytics Cloud environments using Terraform](https://blogs.oracle.com/analytics/post/oracle-analytics-cloud-set-up-and-configure-oracle-analytics-cloud-environments-using-terraform)|OAC Setup & Configure|How to provision and configure Oracle analytics cloud on OCI using Terraform.
@@ -61,6 +62,8 @@ Reviewed: 31.10.2023
 ## OAC Latest Release and Announcements
  |Content Link |Descripton|
 | ------------ |------------|
+|[Oracle Analytics Cloud new features - Jul 2024](https://www.youtube.com/watch?v=0BVxTCvDmaQ&list=PL6gBNP-Fr8KXAOF9RgJIU5ykJvD8fHoxj)|Oracle Analytics Cloud Jul-2024 new features videos|
+|[Oracle Analytics Cloud new features - May 2024](https://www.youtube.com/watch?v=eoNmcRZ5wYI&list=PL6gBNP-Fr8KU55dSbzkEKySjSDWlL3BWm)|Oracle Analytics Cloud May-2024 new features videos|
 |[Oracle Analytics Cloud new features - March 2024](https://www.youtube.com/playlist?list=PL6gBNP-Fr8KWlnpaELiCxQJii-F4c7Ehz)|Oracle Analytics Cloud March-2024 new features videos|
 |[Oracle Analytics Cloud new features - January 2024](https://www.youtube.com/playlist?list=PL6gBNP-Fr8KUGvVDRGC8IyXo8yQzUtMiD)|Oracle Analytics Cloud January-2024 new features videos|
 |[Oracle Analytics New Capabilities - November 2023](https://www.youtube.com/playlist?list=PL6gBNP-Fr8KXVh3PVwWfl1nC_TyHi_yl8)|Oracle Analytics Cloud November-2023 release|
@@ -113,7 +116,7 @@ Reviewed: 31.10.2023
 |Content Link |Descripton|
 | ------------ |------------|
 |[OAC vs PowerBI vs Tableau](https://www.oracle.com/business-analytics/comparison-chart.html)|Comparison of Oracle Analytics Cloud with other leading business analytics products|
-|[Gartner Analytics Review 2023](https://www.youtube.com/watch?v=nYNbpGeu_nw)|Oracle Analytics as visionary in the 2023 Gartner’s Magic Quadrant|
+|[Gartner Analytics Review 2024](https://www.oracle.com/news/announcement/oracle-named-leader-in-2024-gartner-magic-quadrant-for-analytics-and-business-intelligence-platforms-2024-06-24/)|Oracle Named a Leader in the 2024 Gartner® Magic Quadrant™ for Analytics and Business Intelligence Platforms|
 
 
 ## Blogs for AI/ML with Oracle Analytics Platform

data-platform/analytics/oracle-analytics-cloud/files/Machine Learning in Oracle Analytics - Understanding parameters of Naive Bayes.pdf

@@ -0,0 +1,42 @@
+# AI Vector Search
+
+Oracle AI Vector Search is designed for Artificial Intelligence (AI) workloads and allows you to query data based on semantics, rather than keywords. The VECTOR data type is introduced with the release of Oracle Database 23ai, providing the foundation to store vector embeddings alongside business data in the database. Using embedding models, you can transform unstructured data into vector embeddings that can then be used for semantic queries on business data.
+
+Reviewed Date: 17.07.2024
+
+# Useful Links
+
+## Documentation  
+ 
+- [Oracle.com](https://www.oracle.com/database/ai-vector-search/)
+- [Oracle AI Vector Search User's Guide](https://docs.oracle.com/en/database/oracle/oracle-database/23/vecse/overview-ai-vector-search.html)
+- [PL/SQL Packages and Types Reference: DBMS_VECTOR](https://docs.oracle.com/en/database/oracle/oracle-database/23/arpls/dbms_vector1.html#GUID-F9FCB225-821A-4CCA-92B5-58B9927234FA)
+- [PL/SQL Packages and Types Reference: DBMS_VECTOR_CHAIN](https://docs.oracle.com/en/database/oracle/oracle-database/23/arpls/dbms_vector_chain1.html#GUID-D80DDBEF-F1A9-4267-9D3C-A54D237D95C1)
+- [Oracle AI Vector Search FAQ](https://www.oracle.com/database/ai-vector-search/faq/)
+
+## Blogs & Videos
+
+- [Oracle Announces General Availability of AI Vector Search in Oracle Database 23ai](https://blogs.oracle.com/database/post/oracle-announces-general-availability-of-ai-vector-search-in-oracle-database-23ai)
+- [OML4Py: Leveraging ONNX and Hugging Face for AI Vector Search](https://blogs.oracle.com/machinelearning/post/oml4py-leveraging-onnx-and-hugging-face-for-advanced-ai-vector-search)
+- [Use AI Vector Search to Build GenAI Apps with Enterprise Data| Oracle DatabaseWorld AI Edition](https://www.youtube.com/watch?v=5o5Ds8KLqVw&list=PLcFwxJMrxygALJRhZCbnjtDBYWCpWXPGz&index=3)
+
+
+## LiveLabs Workshops
+
+- [Oracle AI Vector Search - 15 Minute Basics](https://apexapps.oracle.com/pls/apex/r/dbpm/livelabs/view-workshop?wid=3975&clear=RR,180&session=3449305441143)
+- [Oracle AI Vector Search - Basics](https://apexapps.oracle.com/pls/apex/r/dbpm/livelabs/view-workshop?wid=1070&clear=RR,180)
+- [AI Vector Search - Complete RAG Application using PL/SQL in Oracle Database 23ai](https://apexapps.oracle.com/pls/apex/r/dbpm/livelabs/view-workshop?wid=3934&clear=RR,180&session=11020955624236)
+- [AI Vector Search - 7 Easy Steps to Building a RAG Application using LangChain](https://apexapps.oracle.com/pls/apex/r/dbpm/livelabs/view-workshop?wid=3927&clear=RR,180&session=11020955624236)
+  
+
+# Team Publications
+
+- [Getting started with vectors in 23ai](https://blogs.oracle.com/coretec/post/getting-started-with-vectors-in-23ai)
+
+# License
+
+Copyright (c) 2024 Oracle and/or its affiliates.
+
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.

data-platform/analytics/oracle-analytics-cloud/files/OAC Guidelines.pdf

@@ -0,0 +1,22 @@
+REM Creation of tablespace, user and directory
+
+-- connect as user sys to FREEPDB1
+
+-- create tablespace
+
+create bigfile tablespace TBS_VECTOR datafile size 256M autoextend on maxsize 2G;
+
+-- create user with the new role DB_DEVELOPER_ROLE
+DROP USER vector_user cascade;
+
+create user vector_user identified by "Oracle_4U"
+default tablespace TBS_VECTOR temporary tablespace TEMP
+quota unlimited on TBS_VECTOR;
+
+grant create mining model to vector_user;
+grant DB_DEVELOPER_ROLE to vector_user;
+
+-- create directory
+
+CREATE OR REPLACE DIRECTORY dm_dump as '&directorypath';
+GRANT all ON DIRECTORY dm_dump TO vector_user;

data-platform/core-converged-db/ai-vector-search/README.md

@@ -0,0 +1,27 @@
+REM Create an external table and check the data set
+ 
+-- copy the data set in the directory DM_DUMP
+-- dataset_200K.txt is located in: https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/fro8fl9kuqli/b/AIVECTORS/o/dataset_200K.txt
+
+-- connect as user VECTOR_USER and create an external table
+
+connect vector_user/Oracle_4U@FREEPDB1
+
+CREATE TABLE if not exists CCNEWS_TMP (sentence VARCHAR2(4000))
+  ORGANIZATION EXTERNAL (TYPE ORACLE_LOADER DEFAULT DIRECTORY dm_dump
+                         ACCESS PARAMETERS
+                           (RECORDS DELIMITED BY 0x'0A'
+                            READSIZE 100000000
+                            FIELDS (sentence CHAR(4000)))
+                         LOCATION (sys.dm_dump:'dataset_200K.txt'))
+  PARALLEL
+  REJECT LIMIT UNLIMITED;
+
+-- Check that the external table is correct
+
+select count(*) from CCNEWS_TMP; 
+
+-- Check the 4 first rows
+
+select * from CCNEWS_TMP where rownum < 4;
+

data-platform/core-converged-db/ai-vector-search/files/0_setup_ts_user_directory.sql

@@ -0,0 +1,16 @@
+REM  load a vector embedding model into the database: we can load a ONNX model inside the database using DBMS_VECTOR package 
+
+/* 
+The ONNX file in this example was generated by Oracle OML4Py utility from the all-MiniLM-L6-v2 sentence transformer model. 
+It can be downloaded from https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/fro8fl9kuqli/b/AIVECTORS/o/all-MiniLM-L6-v2.onnx
+Copy the all-MiniLM-L6-v2.onnx file to the path corresponding to the DM_DUMP directory.
+  */ 
+
+-- connect to the VECTOR_USER 
+
+connect vector_user/Oracle_4U@FREEPDB1
+
+
+-- load the ONNX model in the database
+
+EXECUTE DBMS_VECTOR.LOAD_ONNX_MODEL('DM_DUMP','all-MiniLM-L6-v2.onnx','doc_model');

data-platform/core-converged-db/ai-vector-search/files/1_externaltable.sql

@@ -0,0 +1,23 @@
+REM  Load the ONNX model into the database with DBMS_VECTOR
+
+connect vector_user/Oracle_4U@FREEPDB1
+
+col model_name format a12
+col mining_function format a12
+col algorithm format a12
+col attribute_name format a20
+col data_type format a30
+col vector_info format a30
+set lines 120
+
+SELECT model_name, mining_function, algorithm,
+algorithm_type, model_size
+FROM user_mining_models
+WHERE model_name = 'DOC_MODEL'
+ORDER BY model_name;
+
+
+SELECT model_name, attribute_name, attribute_type, data_type, vector_info
+FROM user_mining_model_attributes
+WHERE model_name = 'DOC_MODEL'
+ORDER BY attribute_name;

data-platform/core-converged-db/ai-vector-search/files/2_load_vector_embedding_in_db.sql

@@ -0,0 +1,25 @@
+REM  Calculate embeddings directly in the database
+
+-- connect to the VECTOR_USER 
+
+connect vector_user/Oracle_4U@FREEPDB1
+
+-- create table CCNEWS
+create table if not exists CCNEWS (
+    id number(10) not null,
+    info VARCHAR2(4000),
+    vec VECTOR
+);
+
+
+-- Use the doc_model previously loaded to calculate the vector embeddings
+-- Please plan additional time to complete this statement 
+
+insert into CCNEWS (id, info, vec)
+select rownum,
+       sentence,
+       TO_VECTOR(VECTOR_EMBEDDING(doc_model USING sentence as data))
+from CCNEWS_TMP;
+
+
+commit;