|
| 1 | +# Document Evaluation Tool using Oracle Generative AI, OCI Document Understanding and Oracle Integration Cloud |
| 2 | + |
| 3 | +Reviewed: 14.03.2024 |
| 4 | + |
| 5 | +# Introduction |
| 6 | + |
| 7 | +In this article, we'll explore how to make a handy tool that helps to evaluate documents using Oracle Generative AI, OCI Document Understanding, and Oracle Integration Cloud (OIC). |
| 8 | +This tool can be super useful in different situations: |
| 9 | +For example, it could help recruiters in identifying the top nN% of resumes to get the best candidate for a job, as well it could help to check the RFP compliance in a vendor bid, or easily figure out what's in a bunch of email attachments without opening each one, but these are only a few examples. It can be applied to any industry and is adaptable to any department like HR, Procurement, Marketing, Customer Service, and others. |
| 10 | + |
| 11 | +Here are 2 use cases for how you may use the document evaluation sample application: |
| 12 | + |
| 13 | +A CV Evaluation App as part of a recruitment process: |
| 14 | + |
| 15 | +<img src="./images/1_CVEvaluation.png"></img> |
| 16 | +<img src="./images/2_CVEvaluationResults.png"></img> |
| 17 | + |
| 18 | + |
| 19 | +A Supplier Proposal Evaluation App as part of a vendor bid. |
| 20 | + |
| 21 | +<img src="./images/3_RFPEvaluation.png"></img> |
| 22 | +<img src="./images/4_RFPEvaluationResults.png"></img> |
| 23 | + |
| 24 | +In this article we will use as a case study: **"CV Evaluation APP"** |
| 25 | + |
| 26 | +# Prerequisites |
| 27 | + |
| 28 | +Before getting started, make sure you have access to the following Oracle Cloud Infrastructure (OCI) services: |
| 29 | + |
| 30 | +- Oracle Generative AI Service (GenAI) |
| 31 | +- Oracle Document Understanding Service |
| 32 | +- Oracle Integration Cloud (OIC) with Visual Builder(VBCS) enabled |
| 33 | +- Oracle Object Storage |
| 34 | + |
| 35 | + |
| 36 | +# Solution Architecture |
| 37 | + |
| 38 | +In this section, we'll dive into the building blocks of the solution architecture. |
| 39 | +<img src="./images/5_SolutionArchitecture-oci-lowcode-ai-rag-logical-blocks.png"></img> |
| 40 | + |
| 41 | +We've built the application using Oracle Visual Builder (as part of OIC), and it smoothly runs through Oracle Integration Cloud as the main, low-code orchestration tool. OCI Document Understanding is there to handle the document processing, and Generative AI Multi-Chain Prompt technique for extracting key information from these CVs, summarizing them, and helping to facilitate the decision-making process: |
| 42 | + |
| 43 | +1. Document Evaluation Tool App interface built using VBCS: |
| 44 | + |
| 45 | +- The “CV Evaluation App” discussed in this context is crafted within the VBCS platform. Oracle Visual Builder Cloud Service serves as a hosted environment for building your application development infrastructure. It offers an open-source, standards-based development service for creating, collaborating on, and deploying applications within the Oracle Cloud. |
| 46 | + |
| 47 | +2. Document Processing with OCI Document Understanding.: |
| 48 | +- The role of OCI Document Understanding is to extract content from document images and other document file types, as CVs can be in different formats. |
| 49 | +- The service employs machine learning and optical character recognition capabilities(OCR) to analyze and extract information from various types of documents. |
| 50 | + |
| 51 | +3. Key Criteria Extraction using OCI Generative AI Service: |
| 52 | +- The extracted CV data from Document Understanding is sent to the OCI Generative AI Service to perform the following two tasks: CV Summarization and evaluation criteria extraction. This action is performed via one single prompt, which we will later explore in the section “Prompting with Oracle Generative AI”. |
| 53 | + |
| 54 | +4. Integration and orchestration using OIC: |
| 55 | +- Oracle Integration Cloud adopts a low-code approach, allowing the creation of integrations and workflows with minimal coding effort and this accelerates the development process. OIC includes monitoring and analytics tools that provide real-time visibility into integration flows. This helps to track performance, identify bottlenecks, and change and optimize LLM integration flows like the one we are exploring here. |
| 56 | + |
| 57 | +5. Document Evaluation using Generative AI: |
| 58 | +- The decision-making process could be facilitated in this case with the help of Oracle Generative AI. We are using prompting techniques to help recruiters with useful information that they can use in the decision-making process. |
| 59 | + |
| 60 | + |
| 61 | + |
| 62 | +# Application Flow in Detail |
| 63 | + |
| 64 | +In this application: |
| 65 | + |
| 66 | +**Step1.** The File Picker action in VBCS allows the Recruiter to upload all the CVs. |
| 67 | + |
| 68 | +<img src="./images/6_Step1VBCS_FilePicker.png"></img> |
| 69 | + |
| 70 | +<img src="./images/7_Step1VBCS_FilePicker2.png"></img> |
| 71 | + |
| 72 | +**Step2.** Evaluator select hard and soft criteria(input fields) from a predifined list. |
| 73 | + |
| 74 | +<img src="./images/8_Step2VBCS_CriteriaSelection.png"></img> |
| 75 | + |
| 76 | +**Step3.** Evaluator press the button “Evaluate” |
| 77 | + |
| 78 | +<img src="./images/9_Step3VBCS_Evaluation1.png"></img> |
| 79 | +<img src="./images/10_Step3VBCS_Evaluation2.png"></img> |
| 80 | + |
| 81 | +**Step4.** After pressing the button “Evaluate”, each CV is processed via Document Understanding, triggering the **OIC Data Loader Low-Code Child Integration Flow** (CVs Data Extraction with Document Understanding REST API) |
| 82 | + |
| 83 | +<img src="./images/11_Step4OIC_DataLoader.png"></img> |
| 84 | +<img src="./images/12_Step4OIC_DataLoaderConfig.png"></img> |
| 85 | +<img src="./images/13_Step4OIC_DataLoaderRestInvoke.png"></img> |
| 86 | +<img src="./images/14_Step4OIC_DataLoaderSampleJson.png"></img> |
| 87 | +<img src="./images/15_Step4OIC_DataLoaderSapleJson2.png"></img> |
| 88 | +<img src="./images/16_Step4OIC_DocUnderstanding.png"></img> |
| 89 | + |
| 90 | + |
| 91 | +**Step5.** At this step, using Generative AI the summary of the CV is created, and also evaluation criteria are extracted, triggering the **OIC Query Engine Low-Code Parent Integration Flow** (CV Summary Generation and Evaluation Criteria extraction with Generative AI Inference REST API). |
| 92 | + |
| 93 | +<img src="./images/17_Step5OIC_Summarization.png"></img> |
| 94 | +<img src="./images/18_Step5OIC_SummarizationProperties.png"></img> |
| 95 | +<img src="./images/19_Step5OIC_SummarizationJsonDetails1.png"></img> |
| 96 | +<img src="./images/20_Step5OIC_SummarizationJsonDetails2.png"></img> |
| 97 | +<img src="./images/21_Step5OIC_SummarizationJsonDetails3.png"></img> |
| 98 | +<img src="./images/22_Step5OIC_GenAIPrompt.png"></img> |
| 99 | + |
| 100 | +**Step6.** Decision logic together with Evaluator Criteria and Extracted Criteria from the Candidate CV are sent to Generative AI Inference REST API, to obtain one of the responses “Good Fit”/ “No Fit” |
| 101 | + |
| 102 | +<img src="./images/23_Step6OIC_Evaluation.png"></img> |
| 103 | +<img src="./images/24_Step6OIC_GenAIPrompt.png"></img> |
| 104 | + |
| 105 | +**Step7.** The list of all candidates is shown on the screen together with the response if the candidate is a good fit or not. Also, a summary of each CV is available. |
| 106 | + |
| 107 | +<img src="./images/25_Step6OIC_VBCSEvaluationResult.png"></img> |
| 108 | + |
| 109 | +# Prompting with Oracle Generative AI |
| 110 | + |
| 111 | +In the following CV Evaluation APP use case we are using a Generative AI Multi-Chain Prompting technique to get the results. It consists of 2 prompts: |
| 112 | + |
| 113 | +**Important Note:** |
| 114 | +**All examples and data presented in this article are purely fictive and for showcasing purposes only. Any resemblance to real persons, or actual companies is purely coincidental. Dummy users and entities have been created for this demo, and no real data or information about individuals or organizations is being disclosed. We prioritize privacy and ethical considerations in the presentation of information, and any similarities to real-world entities are unintentional.** |
| 115 | + |
| 116 | +## Prompt 1: Summarization and Key Criteria Extraction |
| 117 | + |
| 118 | +To extract the right skills from the CV that match the Evaluator input, we have created a prompt in a Zero-shot technique, where we explicitly mention the output we need to gather out of the document: |
| 119 | + |
| 120 | + |
| 121 | +<img src="./images/22_Step5OIC_GenAIPrompt.png"></img> |
| 122 | + |
| 123 | +Please access the full prompt <a href="./images/CVEvaluationAppPrompts.pdf">here</a> |
| 124 | + |
| 125 | +## Prompt 2: Final candidate skills evaluation |
| 126 | + |
| 127 | +To evaluate all the CVs, a single prompt was built using two combined techniques: **Few-shot Prompting** and **Chain-of-Thought Prompting**. |
| 128 | + |
| 129 | +<img src="./images/24_Step6OIC_GenAIPrompt.png"></img> |
| 130 | + |
| 131 | +Please access the full prompt <a href="./images/CVEvaluationAppPrompts.pdf">here</a> |
| 132 | + |
| 133 | +# Code |
| 134 | + VBCS app - RecruitmentApp-1.0.zip |
| 135 | + OIC CV Summary Generation and Evaluation Criteria extraction Parent Integration Flow - CV_JOB_MATCH_FINDER_01.00.0000.iar |
| 136 | + OIC CVs Data Extraction Child Integration Flow - AIDOCUMENTPROCESSING_01.00.0000.iar |
| 137 | + Generative AI Prompts: CVEvaluationAppPrompts.pdf |
| 138 | + CV Samples - cv_samples folder |
| 139 | + |
| 140 | +Please find the **document_evaluation_app_resources.zip** archive in <a href="./document_evaluation_app_resources.zip">/files/document_evaluation_app_resources.zip</a> |
| 141 | + |
| 142 | +# Conclusion |
| 143 | + |
| 144 | +In conclusion, using Oracle Generative AI, OCI Document Understanding, and Oracle Integration Cloud (OIC) you may build a nice solution that may sort and help in evaluating documents across various industries and departments. Its adaptability and ease of use can become a handy tool for streamlining document management processes, saving time, and helping enhance productivity across a wide range of business scenarios. |
| 145 | + |
| 146 | +### Authors |
| 147 | + |
| 148 | +<a href="https://github.com/jcgocol">@jcgocol</a>, <a href="https://github.com/clemnaru">@clemnaru</a> |
| 149 | + |
| 150 | + |
| 151 | +# License |
| 152 | + |
| 153 | +Copyright (c) 2024 Oracle and/or its affiliates. |
| 154 | + |
| 155 | +Licensed under the Universal Permissive License (UPL), Version 1.0. |
| 156 | + |
| 157 | +See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details. |
| 158 | + |
0 commit comments