Skip to content

Commit 25df7b7

Browse files
chienyuanchangchienyuanchang
committed
draft of pro mode notebook
1 parent 2e3e9ce commit 25df7b7

File tree

8 files changed

+361
-22
lines changed

8 files changed

+361
-22
lines changed

‎analyzer_templates/invoice_contract_verification_pro_mode.json

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
{
2+
"baseAnalyzerId": "prebuilt-documentAnalyzer",
3+
"mode": "pro",
4+
"config": {
5+
"disableContentFiltering": true
6+
},
7+
"processingLocation": "global",
8+
"fieldSchema": {
9+
"name": "InvoiceContractVerification",
10+
"description": "Analyze invoice to confirm total consistency with signed contract.",
11+
"fields": {
12+
"PaymentTermsInconsistencies": {
13+
"type": "array",
14+
"method": "generate",
15+
"description": "List all areas of inconsistency identified in the invoice with corresponding evidence.",
16+
"items": {
17+
"$ref": "#/$defs/InvoiceInconsistency"
18+
}
19+
},
20+
"ItemInconsistencies": {
21+
"type": "array",
22+
"method": "generate",
23+
"description": "List all areas of inconsistency identified in the invoice in the goods or services sold (including detailed specifications for every line item).",
24+
"items": {
25+
"$ref": "#/$defs/InvoiceInconsistency"
26+
}
27+
},
28+
"BillingLogisticsInconsistencies": {
29+
"type": "array",
30+
"method": "generate",
31+
"description": "List all areas of inconsistency identified in the invoice regarding billing logistics and administrative or legal issues.",
32+
"items": {
33+
"$ref": "#/$defs/InvoiceInconsistency"
34+
}
35+
},
36+
"PaymentScheduleInconsistencies": {
37+
"type": "array",
38+
"method": "generate",
39+
"description": "List all areas of inconsistency identified in the invoice with corresponding evidence.",
40+
"items": {
41+
"$ref": "#/$defs/InvoiceInconsistency"
42+
}
43+
},
44+
"TaxOrDiscountInconsistencies": {
45+
"type": "array",
46+
"method": "generate",
47+
"description": "List all areas of inconsistency identified in the invoice with corresponding evidence regarding taxes or discounts.",
48+
"items": {
49+
"$ref": "#/$defs/InvoiceInconsistency"
50+
}
51+
}
52+
},
53+
"definitions": {
54+
"InvoiceInconsistency": {
55+
"type": "object",
56+
"method": "generate",
57+
"description": "Area of inconsistency in the invoice with the company's contracts.",
58+
"properties": {
59+
"Evidence": {
60+
"type": "string",
61+
"method": "generate",
62+
"description": "Evidence or reasoning for the inconsistency in the invoice."
63+
},
64+
"InvoiceField": {
65+
"type": "string",
66+
"method": "generate",
67+
"description": "Invoice field or the aspect that is inconsistent with the contract."
68+
}
69+
}
70+
}
71+
}
72+
}
73+
}

‎data/invoice_contract_verification/input_docs/contoso_lifts_invoice.pdf

130 Bytes
Binary file not shown.

‎data/invoice_contract_verification/reference_docs/BUILDERS LIMITED WARRANTY.pdf

130 Bytes
Binary file not shown.

‎data/invoice_contract_verification/reference_docs/HOLDING TANK SERVICING CONTRACT.pdf

130 Bytes
Binary file not shown.

‎data/invoice_contract_verification/reference_docs/PROPERTY MANAGEMENT AGREEMENT.pdf

130 Bytes
Binary file not shown.

‎data/invoice_contract_verification/reference_docs/purchase_contract.pdf

129 Bytes
Binary file not shown.

‎notebooks/content_understanding_pro_mode.ipynb

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Conduct complex analysis by Pro Mode\n",
8+
"\n",
9+
"> #################################################################################\n",
10+
">\n",
11+
"> Note: Currently this feature is exclusively available for `document` data. \n",
12+
"> Current supported file types: pdf, tiff, jpg, jpeg, png, bmp, heif\n",
13+
">\n",
14+
"> #################################################################################\n",
15+
"\n",
16+
"This notebook demonstrates how to use **Pro mode** in Azure AI Content Understanding to enhance your analyzer with multiple inputs and optional reference data. Pro mode is designed for advanced use cases, particularly those requiring multi-step reasoning, and complex decision-making (for instance, identifying inconsistencies, drawing inferences, and making sophisticated decisions). The pro mode allows input from multiple content files and includes the option to provide reference data at analyzer creation time.\n",
17+
"\n",
18+
"In this notebook, we will demonstrate how you can create an analyzer with reference data and analyze your files using Pro mode.\n",
19+
"\n",
20+
"For more details on Pro mode, see the [Azure AI Content Understanding: Standard and Pro Modes](https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/concepts/standard-pro-modes) documentation."
21+
]
22+
},
23+
{
24+
"cell_type": "markdown",
25+
"metadata": {},
26+
"source": [
27+
"## Prerequisites\n",
28+
"1. Ensure Azure AI service is configured following [steps](../README.md#configure-azure-ai-service-resource)\n",
29+
"1. If using reference documents, please set up `REFERENCE_DOC_SAS_URL` and `REFERENCE_DOC_PATH` in `.env` file. Can see similar steps in [Set labeled data](../docs/set_env_for_labeled_data.md) for the reference.\n",
30+
" - `REFERENCE_DOC_SAS_URL`: SAS URL of the Azure blob container. \n",
31+
" - `REFERENCE_DOC_PATH`: The designated folder path under the Azure blob container. \n",
32+
"1. Install the required packages to run the sample."
33+
]
34+
},
35+
{
36+
"cell_type": "code",
37+
"execution_count": null,
38+
"metadata": {},
39+
"outputs": [],
40+
"source": [
41+
"%pip install -r ../requirements.txt"
42+
]
43+
},
44+
{
45+
"cell_type": "markdown",
46+
"metadata": {},
47+
"source": [
48+
"## Analyzer template and local files\n",
49+
"- `analyzer_template`: In this sample we define an analyzer template for invoice contract verification.\n",
50+
"- `input_docs`: We can have multiple input document files in one folder or designate a single document file location \n",
51+
"- `reference_docs`: During analyzer creation, we can provide documents that can aid in providing context that references the service at inference time. We will get ocr results for these files if needed, generate a reference jsonl file, and upload these files to a designated Azure blob storage.\n",
52+
"\n",
53+
"> For example, if you're looking to analyze invoices to ensure they're consistent with a contractual agreement, you can supply the invoice and other relevant documents (for example, a purchase order) as inputs, and supply the contract files as reference data. The service applies reasoning to validate the input documents according to your schema, which might be to identify discrepancies to flag for further review."
54+
]
55+
},
56+
{
57+
"cell_type": "code",
58+
"execution_count": null,
59+
"metadata": {},
60+
"outputs": [],
61+
"source": [
62+
"analyzer_template = \"../analyzer_templates/invoice_contract_verification_pro_mode.json\"\n",
63+
"input_docs = \"../data/invoice_contract_verification/input_docs\"\n",
64+
"reference_docs = \"../data/invoice_contract_verification/reference_docs/\""
65+
]
66+
},
67+
{
68+
"cell_type": "markdown",
69+
"metadata": {},
70+
"source": [
71+
"## Create Azure content understanding client\n",
72+
"> The [AzureContentUnderstandingClient](../python/content_understanding_client.py) is utility Class which contain the functions to interact with the Content Understanding server. Before Content Understanding SDK release, we can regard it as a lightweight SDK. Fill the constant **AZURE_AI_ENDPOINT**, **AZURE_AI_API_VERSION**, **AZURE_AI_API_KEY** with the information from your Azure AI Service.\n",
73+
"\n",
74+
"> ⚠️ Important:\n",
75+
"You must update the code below to match your Azure authentication method.\n",
76+
"Look for the `# IMPORTANT` comments and modify those sections accordingly.\n",
77+
"If you skip this step, the sample may not run correctly.\n",
78+
"\n",
79+
"> ⚠️ Note: Using a subscription key works, but using a token provider with Azure Active Directory (AAD) is much safer and is highly recommended for production environments."
80+
]
81+
},
82+
{
83+
"cell_type": "code",
84+
"execution_count": null,
85+
"metadata": {},
86+
"outputs": [],
87+
"source": [
88+
"import logging\n",
89+
"import json\n",
90+
"import os\n",
91+
"import sys\n",
92+
"from pathlib import Path\n",
93+
"from dotenv import find_dotenv, load_dotenv\n",
94+
"from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
95+
"\n",
96+
"# import utility package from python samples root directory\n",
97+
"parent_dir = Path(Path.cwd()).parent\n",
98+
"sys.path.append(str(parent_dir))\n",
99+
"from python.content_understanding_client import AzureContentUnderstandingClient\n",
100+
"\n",
101+
"load_dotenv(find_dotenv())\n",
102+
"logging.basicConfig(level=logging.INFO)\n",
103+
"\n",
104+
"# For authentication, you can use either token-based auth or subscription key, and only one of them is required\n",
105+
"AZURE_AI_ENDPOINT = os.getenv(\"AZURE_AI_ENDPOINT\")\n",
106+
"# IMPORTANT: Replace with your actual subscription key or set up in \".env\" file if not using token auth\n",
107+
"AZURE_AI_API_KEY = os.getenv(\"AZURE_AI_API_KEY\")\n",
108+
"AZURE_AI_API_VERSION = os.getenv(\"AZURE_AI_API_VERSION\", \"2025-05-01-preview\")\n",
109+
"\n",
110+
"credential = DefaultAzureCredential()\n",
111+
"token_provider = get_bearer_token_provider(credential, \"https://cognitiveservices.azure.com/.default\")\n",
112+
"\n",
113+
"client = AzureContentUnderstandingClient(\n",
114+
" endpoint=AZURE_AI_ENDPOINT,\n",
115+
" api_version=AZURE_AI_API_VERSION,\n",
116+
" # IMPORTANT: Comment out token_provider if using subscription key\n",
117+
" token_provider=token_provider,\n",
118+
" # IMPORTANT: Uncomment this if using subscription key\n",
119+
" # subscription_key=AZURE_AI_API_KEY,\n",
120+
" # x_ms_useragent=\"azure-ai-content-understanding-python/pro_mode\", # This header is used for sample usage telemetry, please comment out this line if you want to opt out.\n",
121+
")"
122+
]
123+
},
124+
{
125+
"cell_type": "markdown",
126+
"metadata": {},
127+
"source": [
128+
"## Prepare reference data\n",
129+
"In this step, we will use Azure AI service to get ocr results for the reference document files if needed, generate a reference jsonl file, and upload these files to a designated Azure blob storage.\n",
130+
"\n",
131+
"We use **REFERENCE_DOC_SAS_URL** and **REFERENCE_DOC_PATH** that's set in the prerequisite step."
132+
]
133+
},
134+
{
135+
"cell_type": "code",
136+
"execution_count": null,
137+
"metadata": {},
138+
"outputs": [],
139+
"source": [
140+
"REFERENCE_DOC_SAS_URL = os.getenv(\"REFERENCE_DOC_SAS_URL\")\n",
141+
"REFERENCE_DOC_PATH = os.getenv(\"REFERENCE_DOC_PATH\")\n",
142+
"\n",
143+
"await client.generate_knowledge_base_on_blob(reference_docs, REFERENCE_DOC_SAS_URL, REFERENCE_DOC_PATH, skip_analyze=False)"
144+
]
145+
},
146+
{
147+
"cell_type": "markdown",
148+
"metadata": {},
149+
"source": [
150+
"## Create analyzer with defined schema for Pro Mode\n",
151+
"Before creating the custom fields analyzer, you should fill the constant ANALYZER_ID with a business-related name. Here we randomly generate a name for demo purpose.\n",
152+
"\n",
153+
"We use **REFERENCE_DOC_SAS_URL** and **REFERENCE_DOC_PATH** that's set up in `.env` file and used in the previous step."
154+
]
155+
},
156+
{
157+
"cell_type": "code",
158+
"execution_count": null,
159+
"metadata": {},
160+
"outputs": [],
161+
"source": [
162+
"import uuid\n",
163+
"CUSTOM_ANALYZER_ID = \"pro-mode-sample-\" + str(uuid.uuid4())\n",
164+
"\n",
165+
"response = client.begin_create_analyzer(\n",
166+
" CUSTOM_ANALYZER_ID,\n",
167+
" analyzer_template_path=analyzer_template,\n",
168+
" pro_mode_reference_docs_storage_container_sas_url=REFERENCE_DOC_SAS_URL,\n",
169+
" pro_mode_reference_docs_storage_container_path_prefix=REFERENCE_DOC_PATH,\n",
170+
")\n",
171+
"result = client.poll_result(response)\n",
172+
"if result is not None and \"status\" in result and result[\"status\"] == \"Succeeded\":\n",
173+
" logging.info(f\"Here is the analyzer detail for {result['result']['analyzerId']}\")\n",
174+
" logging.info(json.dumps(result, indent=2))\n",
175+
"else:\n",
176+
" logging.info(\n",
177+
" \"Check your service please, may be some issues in configuration and deployment\"\n",
178+
" )"
179+
]
180+
},
181+
{
182+
"cell_type": "markdown",
183+
"metadata": {},
184+
"source": [
185+
"## Use created analyzer to do data analysis of the input documents\n",
186+
"After the analyzer is successfully created, we can use it to analyze our input files.\n",
187+
"> NOTE: Pro mode does multi-step reasoning and may take longer time to analyze."
188+
]
189+
},
190+
{
191+
"cell_type": "code",
192+
"execution_count": null,
193+
"metadata": {},
194+
"outputs": [],
195+
"source": [
196+
"response = client.begin_analyze(CUSTOM_ANALYZER_ID, file_location=input_docs)\n",
197+
"result_json = client.poll_result(response, timeout_seconds=360)\n",
198+
"\n",
199+
"logging.info(json.dumps(result_json, indent=2))"
200+
]
201+
},
202+
{
203+
"cell_type": "markdown",
204+
"metadata": {},
205+
"source": [
206+
"## Delete exist analyzer in Content Understanding Service\n",
207+
"This snippet is not required, but it's only used to prevent the testing analyzer from residing in your service. The custom fields analyzer could be stored in your service for reusing by subsequent business in real usage scenarios."
208+
]
209+
},
210+
{
211+
"cell_type": "code",
212+
"execution_count": null,
213+
"metadata": {},
214+
"outputs": [],
215+
"source": [
216+
"client.delete_analyzer(CUSTOM_ANALYZER_ID)"
217+
]
218+
}
219+
],
220+
"metadata": {
221+
"kernelspec": {
222+
"display_name": "Python 3",
223+
"language": "python",
224+
"name": "python3"
225+
},
226+
"language_info": {
227+
"codemirror_mode": {
228+
"name": "ipython",
229+
"version": 3
230+
},
231+
"file_extension": ".py",
232+
"mimetype": "text/x-python",
233+
"name": "python",
234+
"nbconvert_exporter": "python",
235+
"pygments_lexer": "ipython3",
236+
"version": "3.11.12"
237+
}
238+
},
239+
"nbformat": 4,
240+
"nbformat_minor": 2
241+
}

0 commit comments

Comments
 (0)