Added the review suggesions

SurajBaloni · SurajBaloni · commit 48a9a0e00b7d · 2025-08-27T11:02:25.000+05:30
diff --git a/samples/04_gis_analysts_data_scientists/use_multimodal_inputs_for_information_extraction_task_using_a_third-party_llm .ipynb b/samples/04_gis_analysts_data_scientists/use_multimodal_inputs_for_information_extraction_task_using_a_third-party_llm .ipynb
@@ -4,35 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Leveraging Multimodal Inputs for Information Extraction Task Using a Third-Party Large Language Model"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "<!-- <h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
-    "<div class=\"toc\">\n",
-    "<ul class=\"toc-item\">\n",
-    "<li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\">Introduction</a></span></li>\n",
-    "<li><span><a href=\"#Prerequisites\" data-toc-modified-id=\"Prerequisites-2\">Prerequisites</a></span></li>\n",
-    "<li><span><a href=\"#Imports\" data-toc-modified-id=\"Imports-3\">Imports</a></span></li>\n",
-    "<li><span><a href=\"#Data-preparation\" data-toc-modified-id=\"Data-preparation-4\">Data preparation</a></span></li>\n",
-    "<li><span><a href=\"#EntityRecognizer-model\" data-toc-modified-id=\"EntityRecognizer-model-5\">EntityRecognizer model</a></span></li>\n",
-    "<ul class=\"toc-item\">\n",
-    "<li><span><a href=\"#Finding-optimum-learning-rate\" data-toc-modified-id=\"Finding-optimum-learning-rate-5.1\">Finding optimum learning rate</a></span>    \n",
-    "<li><span><a href=\"#Model-training\" data-toc-modified-id=\"Model-training-5.2\">Model training</a></span>\n",
-    "<li><span><a href=\"#Evaluate-model-performance\" data-toc-modified-id=\"Evaluate-model-performance-5.3\">Evaluate model performance</a></span>\n",
-    "<li><span><a href=\"#Validate-results\" data-toc-modified-id=\"Validate-results-5.4\">Validate results</a></span></li>\n",
-    "<li><span><a href=\"#Save-and-load-trained-models\" data-toc-modified-id=\"Save-and-load-trained-models-5.5\">Save and load trained models</a></span></li>\n",
-    "</ul>\n",
-    "<li><span><a href=\"#Model-inference\" data-toc-modified-id=\"Model-inference-6\">Model inference</a></span></li>\n",
-    "<li><span><a href=\"#Publishing-the-results-as-feature-layer\" data-toc-modified-id=\"Publishing-the-results-as-feature-layer-7\">Publishing the results as feature layer</a></span></li>\n",
-    "<li><span><a href=\"#Visualize-crime-incident-on-map\" data-toc-modified-id=\"Visualize-crime-incident-on-map- 8\">Visualize crime incident on map</a></span></li>\n",
-    "<li><span><a href=\"#Create-a-hot-spot-map-of-crime-densities\" data-toc-modified-id=\"Create-a-hot-spot-map-of-crime-densities-9\">Create a hot spot map of crime densities</a></span></li>\n",
-    "<li><span><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-10\">Conclusion</a></span></li>\n",
-    "<li><span><a href=\"#References\" data-toc-modified-id=\"References-11\">References</a></span></li>\n",
-    "</ul></div> -->"
+    "# Use multimodal inputs for information extraction with a third-party large language model"
    ]
   },
   {
@@ -79,11 +51,9 @@
     "\n",
     "As businesses increasingly digitize operations, vast amounts of transactional data—such as sales receipts—are often stored as scanned images or photos. Extracting meaningful, structured information from these image-based documents can support a variety of analytical and operational workflows, such as sales tracking, sales management, and customer insights.\n",
     "\n",
-    "Recent advancements in large language models (LLMs) have opened new possibilities for interpreting and extracting information from such inputs with greater accuracy and flexibility. The **GeoAI toolbox** in ArcGIS Pro supports integration of third-party language models, allowing users to process and analyze text using external AI services. Custom third-party models can be wrapped in ESRI Deep Learning Package (.dlpk) files and used within GeoAI tools and the arcgis.learn API. In this sample, we demonstrate how one such model—**GPT-4o** from OpenAI— used with the **Process Text Using AI Model** tool to extract relevant entities from receipt images. Third-party model support in **ArcGIS Pro** and the `arcgis.learn` API allows users to bring in AI models—whether hosted by providers like OpenAI, Azure, etc., built from open-source code, or fine-tuned for a specific task—to enhance natural language processing directly within ArcGIS workflows.\n",
+    "Recent advancements in large language models (LLMs) have opened new possibilities for interpreting and extracting information from such inputs with greater accuracy and flexibility. The **GeoAI toolbox** in ArcGIS Pro supports integration of third-party language models, allowing users to process and analyze text using external AI services. Custom third-party models can be wrapped in ESRI Deep Learning Package (.dlpk) files and used within GeoAI tools and the arcgis.learn API. In this sample, we demonstrate how one such model, **GPT-4o** from OpenAI, can be used with the **Process Text Using AI Model** tool to extract relevant entities from receipt images. Third-party model support in **ArcGIS Pro** and the `arcgis.learn` API allows users to bring in AI models (whether hosted by providers like OpenAI, Azure, etc., built from open-source code, or fine-tuned for a specific task) to enhance natural language processing directly within ArcGIS workflows.\n",
     "\n",
-    "\n",
-    "\n",
-    "For this use case, we use a set of sales receipt images and perform entity extraction to identify key pieces of information. The extracted entities can be used for downstream tasks such as sales analytics, customer profiling, inventory management, or tax auditing. The model will extracts the following entities:\n",
+    "For this use case, we use a set of sales receipt images and perform entity extraction to identify key pieces of information. The extracted entities can be used for downstream tasks like sales analytics, customer profiling, inventory management, or tax auditing. The model extracts the following entities:\n",
     "\n",
     "- Sale Date  \n",
     "- Customer Name  \n",
@@ -103,7 +73,7 @@
    "source": [
     "# Dataset \n",
     "\n",
-    "For this use case, we use a sample set of images of printed sales receipts from various retail outlets such as Walmart, Costco Wholesale, etc.\n"
+    "For this use case, we use a sample set of images of printed sales receipts from various retail outlets like Walmart, Costco Wholesale, etc.\n"
    ]
   },
   {
@@ -140,8 +110,8 @@
    "metadata": {},
    "source": [
     "# Prerequisites\n",
-    "- Refer to the section **\"Install deep learning dependencies of arcgis.learn module\"** [on this page](https://developers.arcgis.com/python/guide/install-and-set-up/#Install-deep-learning-dependencies) for detailed documentation on installation of the dependencies.\n",
-    "- To learn more about how to create and integrate third-party model, refer [Use third party language models with ArcGIS](https://developers.arcgis.com/python/latest/guide/use-third-party-language-models-with-arcgis/)"
+    "- Refer to the section **\"Install deep learning dependencies\"** [on this page](https://developers.arcgis.com/python/guide/install-and-set-up/#Install-deep-learning-dependencies) for detailed documentation on installation of the dependencies.\n",
+    "- To learn more about how to create and integrate third-party models, refer to [Use third party language models with ArcGIS](https://developers.arcgis.com/python/latest/guide/use-third-party-language-models-with-arcgis/)"
    ]
   },
   {
@@ -150,7 +120,7 @@
    "source": [
     "# Create the Third-Party Deep Learning Package (.dlpk)\n",
     "\n",
-    "The first step in using a third-party language model is to prepare a Deep Learning Package file (`.dlpk`). This package includes your custom NLP Python function to interact with external models, along with an Esri Model Definition (`.emd`) file. In this use case, we use a third-party hosted model—**GPT-4o**—to extract key entities from scanned sales receipt images. Please note that if you use a web-hosted LLM, the data processed will be sent to the LLM provider. Use these models only if you trust their source.\n",
+    "The first step in using a third-party language model is to prepare a Deep Learning Package file (`.dlpk`). This package includes your custom NLP Python function to interact with external models, along with an Esri Model Definition (`.emd`) file. In this use case, we use a third-party hosted model**(GPT-4o)** to extract key entities from scanned sales receipt images. Please note that if you use a web-hosted LLM, the data processed will be sent to the LLM provider. Use these models only if you trust their source.\n",
     "\n",
     "### Components of the Third-Party Deep Learning Package (.dlpk)\n",
     "\n",
@@ -265,21 +235,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [],
    "source": [
-    "import ast\n",
-    "from math import e\n",
     "import arcpy\n",
     "import json\n",
     "import base64\n",
-    "import pandas as pd\n",
-    "import requests\n",
-    "from pydantic import BaseModel\n",
-    "from typing import Optional, List, Dict, Tuple\n",
     "from arcgis.features import FeatureSet\n",
-    "from concurrent.futures import ThreadPoolExecutor\n",
     "import keyring\n",
     "from arcgis.learn import AIServiceConnection\n"
    ]
@@ -290,7 +253,7 @@
    "source": [
     "### Define the ```__init__``` function\n",
     "\n",
-    "The `__init__` method initializes instance variables such as `name`, `description`, and other attributes essential for the NLP function."
+    "The `__init__` method initializes instance variables like `name`, `description`, and other attributes essential for the NLP function."
    ]
   },
   {
@@ -309,13 +272,13 @@
    "metadata": {},
    "source": [
     "### Define ```initialize``` function\n",
-    "The initialize method is called at the start of the custom Python NLP function, within this function we will set up the necessary variables. It accepts two parameters via `kwargs`:\n",
+    "The initialize method is called at the start of the custom Python NLP function. Within this function, we will set up the necessary variables. It accepts two parameters via `kwargs`:\n",
     "\n",
     "#### Parameters in `kwargs`\n",
     "- **`model`**: The path to the ESRI Model Definition (.emd) file.\n",
-    "- **`device`**: The name of the device (either GPU or CPU), which is particularly important for on-premises models.\n",
+    "- **`device`**: The name of the device (either GPU or CPU). This is particularly important for on-premises models.\n",
     "\n",
-    "`initialize` reads the ESRI Model Definition (.emd) file and configures the essential variables needed for inference.\n"
+    "The `initialize` method reads the ESRI Model Definition (.emd) file and configures the essential variables needed for inference. This method is called only once, making it the ideal place to load any resources or dependencies required throughout the inference process.\n"
    ]
   },
   {
@@ -338,7 +301,7 @@
    "source": [
     "### Define the ```getParameterInfo``` function\n",
     "\n",
-    "This function is designed to collect parameters from the user through the GeoAI tools. For our use case, it gathers the text prompt used to instruct the language model on which entities to extract from the input receipt text, as well as the connection file required to authenticate with the third-party model provider. Refer to the section [getParameterInfo](https://developers.arcgis.com/python/latest/guide/use-third-party-language-models-with-arcgis/) to get a detail explanation of the ```getParameterInfo``` function."
+    "The ```getParameterInfo``` function defines the parameters that will be exposed in the GeoAI tool, including their data types and allowable values. For our use case, it gathers the text prompt used to instruct the language model on which entities to extract from the input receipt text, as well as the connection file required to authenticate with the third-party model provider. Refer to the [getParameterInfo](https://developers.arcgis.com/python/latest/guide/use-third-party-language-models-with-arcgis/) section to get a detailed explanation of the ```getParameterInfo``` function."
    ]
   },
   {
@@ -398,10 +361,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The `getConfiguration` method sets up and manages the parameters required by the NLP function. It receives keyword arguments (`kwargs`) that contain the values provided by the user—either through the GeoAI tool interface or programmatically.\n",
+    "The `getConfiguration` method sets up and manages the parameters required by the NLP function. It receives keyword arguments (`kwargs`) that contain the values provided by the user—either through the GeoAI tool interface or programmatically in the ```getParameterInfo``` function.\n",
     "\n",
     "This method is responsible for:\n",
-    "- Extracting user-specified values (e.g., prompt, AI connection file path).\n",
+    "- Extracting input from the tool (e.g., prompt, AI connection file path).\n",
     "- Storing these values in class-level variables or a configuration dictionary.\n",
     "- Controlling how the model processes the input and generates the output based on the updated parameters.\n",
     "\n",
@@ -412,9 +375,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "For our use case, we are using a large language model (LLM) provided by an external AI service—**GPT-4o from OpenAI**. To enable communication between our custom Python NLP function and the OpenAI endpoint, we need a connection file. You can refer to the *[Create AI Service Connection File](#create-ai-service-connection-file)* section for instructions on how to create one.\n",
+    "For our use case, we are using a large language model (LLM) provided by an external AI service, **GPT-4o from OpenAI**. To enable communication between our custom Python NLP function and the OpenAI endpoint, we need a connection file. You can refer to the *[Create AI Service Connection File](#create-ai-service-connection-file)* section for instructions on how to create one.\n",
     "\n",
-    "This connection file securely stores the required credentials, which will be retrieved and used to authenticate and initialize the third-party model. These saved credentials will be accessed from the connection file and set within the current class context in the following function.\n"
+    "This connection file securely stores the required credentials that will be retrieved and used to authenticate and initialize the third-party model. These saved credentials will be accessed from the connection file and set within the current class context in the following function.\n"
    ]
   },
   {
@@ -427,11 +390,11 @@
     "    # Get ai_connection_file parameter value     \n",
     "    connection_file_path = kwargs.get(\"ai_connection_file\", None)\n",
     "    # Read the connection file using AIServiceConnection class inside arcgis.learn\n",
-    "    conn = AIServiceConnection(connection_file_path)\n",
-    "    # Set the  values\n",
-    "    conn_dict = conn.get_dict()\n",
-    "    # self.API_KEY = conn_dict['serviceProviderProperties']['token']\n",
-    "    # self.model = conn_dict['serviceProviderProperties']['model']\n",
+    "    from arcgis.learn import AIServiceConnection\n",
+    "    con = AIServiceConnection(ais_filename)\n",
+    "    conn_param_v = con.get_dict()\n",
+    "    self.API_KEY = conn_param_v[\"authenticationSecrets\"][\"token\"]\n",
+    "    self.model = conn_param_v[\"serviceProviderProperties\"][\"model\"]\n",
     "\n",
     "    # Get prompt parameter value \n",
     "    self.prompt_txt = kwargs.get(\"prompt\", \"\")\n",
@@ -446,6 +409,13 @@
     "### Define the ```predict``` function"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The ```predict``` method performs inference, that is, it generates predictions with the NLP model. This method is passed a FeatureSet containing the input features (or rows in the case of a table) and kwargs containing the field name which contains the input strings. This method returns the results in the form of a FeatureSet object."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -578,14 +548,14 @@
     "\n",
     "\n",
     "- **Zip the Folder**:  \n",
-    "Compress the folder into a ZIP archive.  \n",
+    "Compress the files into a ZIP archive.  \n",
     "Rename the `.zip` file to match the `.emd` file name, but with the `.dlpk` extension.\n",
     "\n",
     "**Example final file name:**\n",
     "\n",
     "```LLMEntityExtractor.dlpk```\n",
     "\n",
-    "This `.dlpk` file is now ready for use with the ```Process Text Using AI Model``` inside ArcGIS Pro."
+    "This `.dlpk` file is now ready for use with the ```Process Text Using AI Model``` tool inside ArcGIS Pro."
    ]
   },
   {
@@ -666,11 +636,17 @@
    "source": [
     "# Conclusion\n",
     "\n",
-    "This guide demonstrates how powerful AI models like GPT-4o can be seamlessly integrated into ArcGIS Pro through the Process text Using AI Model tool to extract structured information from unstructured receipt images. By using the \"Process Text Using AI Model\" tool and a custom LLM wrapped in a Deep Learning Package (.dlpk), users can automate the extraction of key entities such as sale date, customer name, product details, and payment method.\n",
+    "This guide demonstrates how powerful AI models like GPT-4o can be seamlessly integrated into ArcGIS Pro through the ```Process text Using AI Model``` tool to extract structured information from unstructured receipt images. By using the ```Process Text Using AI Model``` tool and a custom LLM wrapped in a Deep Learning Package (.dlpk), users can automate the extraction of key entities like sale date, customer name, product details, and payment method.\n",
     "\n",
-    "This approach reduces the need for manual data entry or annotation while delivering high-quality, analysis-ready outputs that can support a range of GIS and business workflows—including sales analytics, inventory tracking, and customer insights. With built-in support for third-party AI services, ArcGIS Pro enables users to bring the latest in NLP and machine learning directly into their geospatial data pipelines.\n",
-    "\n"
+    "This approach reduces the need for manual data entry or annotation, while delivering high-quality, analysis-ready outputs that can support a range of GIS and business workflows like sales analytics, inventory tracking, and customer insights. With built-in support for third-party AI services, ArcGIS Pro enables users to bring the latest in NLP and machine learning directly into their geospatial data pipelines.\n"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
