Edits1 in readme and BFR_Sample_Rest notebook

Author: HeidiSteen

Image-Processing/BFR_Sample_Rest.ipynb

@@ -7,7 +7,11 @@
     "# Azure Cognitive Search sample \n",
     "## Passing Images as Binary File References\n",
     "\n",
-    "Cognitive Search skillsets that need to pass images to custom skills use a binary file reference to serialize the images to pass them to and from skills. This sample demonstrates an example of how skills can be configured to accept an image as an input from the skillset and return images as outputs to the skillset. This example does nothing more than segment an image based on the layout from OCR. The sole purpose of this sample is to demonstrate how you pass images to skills and how skills can return images.\n",
+    "Skillsets that pass images to custom skills use a binary file reference to serialize the images before passing them to other skills. This sample demonstrates how skills can be configured to accept image inputs and return image outputs. \n",
+    "\n",
+    "While the other steps in this skillset, such as OCR and redaction, have relevance, the key takeaway is configuring and passing binary file references. The custom skill does the heavy lifting. Each input record contains an image that is serialized as a `Base64` encoded string. The input also contains the layout text of image, as returned from the OCR skill. Upon receiving the input, the custom skill segments the image into smaller images based on the coordinates of the layout text. It then returns a list of images, each `Base64` encoded, back to the skillset. While this is not a particularly realistic exercise, it demonstrates techniques that could be leverage in more interesting ways, such as in a [Custom Vision](https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Vision/CustomVision) skill that performs useful inferences on your images.\n",
+    "\n",
+    "For more information about the skills used in this example, see [OCR skill](https://docs.microsoft.com/azure/search/cognitive-search-skill-ocr), [PII skill](https://docs.microsoft.com/azure/search/cognitive-search-skill-pii-detection), and [custom skills](https://docs.microsoft.com/azure/search/cognitive-search-custom-skill-web-api).\n",
     "\n"
    ]
   },
@@ -17,34 +21,36 @@
    "source": [
     "### Prerequisites \n",
     "\n",
-    "Provision the required services:\n",
-    "1. [Azure Cognitive Search](https://docs.microsoft.com/azure/search/search-create-service-portal)\n",
-    "2. [Azure Functions](https://docs.microsoft.com/azure/azure-functions/) used for hosting an API endpoint.\n",
-    "3. [Storage Account](https://docs.microsoft.com/azure/storage/blobs/)\n"
+    "+ [Azure subscription](https://Azure.Microsoft.com/subscription/free)\n",
+    "+ [Azure Cognitive Search service](https://docs.microsoft.com/azure/search/search-create-service-portal) (get the full service endpoint and an admin API key)\n",
+    "+ [Azure Blob storage service](https://docs.microsoft.com/azure/storage/common/storage-account-create) (get the connection string)\n",
+    "+ [Azure Cognitive Services](https://docs.microsoft.com/azure/cognitive-services/cognitive-services-apis-create-account) (get the account name)\n",
+    "+ [Python 3.6+](https://www.python.org/downloads/)\n",
+    "+ [Jupyter Notebook](https://jupyter.org/install)\n",
+    "+ [Visual Studio Code](https://code.visualstudio.com/download) with the [Azure Functions extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions) and the [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python)\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Deploy the Azure functions app \n",
-    "The ```SplitImage``` folder contains an Azure function that will accept an input in the [custom skill format](https://docs.microsoft.com/azure/search/cognitive-search-custom-skill-web-api#skill-inputs). \n",
-    "Each input record contains an image that is serialized as a ```Base64``` encoded string and the layout text returned from the OCR skill.\n",
-    "The skill then segments the image into smaller images based on the coordinates of the layout text. It then returns a list of images, each ```Base64``` encoded back to the skillset. While this is not very useful, you could build a [Custom Vision](https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Vision/CustomVision) skill to perform a useful inference on your images.\n",
+    "### Configure inputs\n",
     "\n",
-    "Follow the [Azure Functions tutorial](https://docs.microsoft.com/azure/developer/python/tutorial-vs-code-serverless-python-05) to deploy the function. Once the deployment completes, navigate to the function app in the portal, select the function (SplitImage) and click the Get Function Url button. Save the function url as we will use it in the next step."
+    "Follow the instructions in the [readme](https://github.com/Azure-Samples/azure-search-python-samples/blob/master/Image-Processing/README.md) to set up the inputs used by the indexer, data source, and skillset.\n",
+    "\n",
+    "Besides connection information, you will need a blob container for the sample JPEG file, and a function app that provides the code used in the custom skill. All the necessary files are provided. The `SplitImage` folder contains an Azure function that will accept an input in the [custom skill format](https://docs.microsoft.com/azure/search/cognitive-search-custom-skill-web-api#skill-inputs). "
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "### Create the enrichment pipeline\n",
-    "In the next few steps we will configure the Cognitive Search enrichment pipeline with the following steps:\n",
-    "1. Create a blob storage data source. Ensure you have a blob storage container with at least one file containing images.\n",
-    "2. Create a skillset to enrich the documents in the data source\n",
-    "3. Create an index\n",
-    "4. Create an indexer to move documents from the data source to the index while invoking the skillset\n"
+    "In the next few steps, configure the Cognitive Search enrichment pipeline, creating these objects on your search service:\n",
+    "1. Create an indexer data source. The data source references a blob storage container with at least one image file.\n",
+    "2. Create a skillset that performs image analysis. The skillset references a Cognitive Services account, a custom function app, and a knowledge store.\n",
+    "3. Create a search index.\n",
+    "4. Create an indexer to move documents from the data source to the index while invoking the skillset.\n"
    ]
   },
   {
@@ -59,36 +65,40 @@
     "import json\n",
     "import requests\n",
     "\n",
-    "# Configure all required variables for Cognitive Search. Replace each with the credentials from your accounts.\n",
+    "# Configure all required variables for this exerences. Replace each with the credentials from your accounts.\n",
     "\n",
-    "# Replace with Search Service name, API key, and endpoint from the Azure portal.\n",
-    "search_service = \"\" # In the format \"https://searchservicename.search.windows.net\"\n",
-    "api_key = 'your search service API key'\n",
+    "# Replace with a full search service endpoint the format \"https://searchservicename.search.windows.net\"\n",
+    "# Paste in an admin API key. Both values can be obtained from the Azure portal.\n",
+    "search_service = \"https://<YOUR-SEARCH-SERVICE>.search.windows.net\"\n",
+    "api_key = '<YOUR-ADMIN-API-KEY>'\n",
     "\n",
     "# Leave the API version and content_type as they are listed here.\n",
     "api_version = '2020-06-30'\n",
     "content_type = 'application/json'\n",
     "\n",
-    "# Replace with a Cognitive Services all in one key.\n",
+    "# Replace with a Cognitive Services account name and all-in-one key.\n",
     "cog_svcs_key = '' #Required only if processing more than 20 documents\n",
-    "cog_svcs_acct = 'your cog services account name'\n",
+    "cog_svcs_acct = '<YOUR-COGNITIVE-SERVICE-ACCOUNT-NAME>'\n",
     "\n",
-    "#Connection string to the storage account. This will be used for the datasource, knowledge store and cache\n",
-    "STORAGECONNSTRING = \"DefaultEndpointsProtocol=https;AccountName=<Storage Acct>;AccountKey=<KEY>;EndpointSuffix=core.windows.net\"\n",
-    "# The container with your files containing images\n",
-    "datasource_container = 'bfrsample' # Replace with the container containging your files\n",
-    "# This sample assumes you will use the same storage account for the datasource, knowledge store and indexer cache. The knowledge store will contain the projected images\n",
+    "# Your Azure Storage account will be used for the datasource, knowledge store and cache\n",
+    "# Replace with a connection string to your Azure Storage account. \n",
+    "STORAGECONNSTRING = \"DefaultEndpointsProtocol=https;AccountName=<YOUR-ACCOUNT>;AccountKey=<YOUR-ACCOUNT-KEY>;EndpointSuffix=core.windows.net\"\n",
+    "# Replace with the blob container containing your image files\n",
+    "datasource_container = 'bfr-sample' \n",
+    "# Use the same storage account for knowledge store and indexer cache. The knowledge store will contain the projected images\n",
     "know_store_cache = STORAGECONNSTRING\n",
-    "# Container where the sliced images will be projected to\n",
+    "# Container where the sliced images will be projected to. Use the value provided below.\n",
     "know_store_container = \"obfuscated\"\n",
-    "skill_uri = \"https://<skillname>.azurewebsites.net/api/SplitImage?code=CODE\""
+    "\n",
+    "# Replace with the Function HTTP URL of the app deployed to Azure Function\n",
+    "skill_uri = \"<YOUR-FUNCTION-APP-URL>\""
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Create a helper function to invoke the Cognitive Search REST API"
+    "Create a helper function to invoke the Cognitive Search REST APIs. "
    ]
   },
   {
@@ -153,7 +163,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Create the skillset"
+    "#### Create the skillset\n",
+    "\n",
+    "Besides skills, a skillset also specifies the knowledge store that will contain the final output."
    ]
   },
   {
@@ -344,7 +356,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Create the index"
+    "#### Create the index\n",
+    "\n",
+    "This exercise doesn't have steps for using the index, but having an index is an indexer requirement. You can use Search Explorer in the Azure portal to query the index on your own."
    ]
   },
   {
@@ -515,7 +529,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Create the indexer"
+    "#### Create the indexer\n",
+    "\n",
+    "The indexer connects to the data source, invokes the skillset, and outputs results. This indexer is scheduled to run every two hours. In the following step, you'll run the indexer to start the process immediately."
    ]
   },
   {
@@ -597,7 +613,7 @@
    "metadata": {},
    "source": [
     "### View Results\n",
-    "The following cell downloads the image so that you can verify skillset success."
+    "The following cell downloads the output image so that you can verify skillset success."
    ]
   },
   {
@@ -630,8 +646,17 @@
    "metadata": {},
    "source": [
     "### Next Steps\n",
-    "You now know how to pass images into skills and even return images to the skillset. As a next step, you can start from scratch and build a [custom AML Skill](https://docs.microsoft.com/en-us/azure/search/cognitive-search-aml-skill) to perform inferences on images or use the Custom Vision service to build a skill. The power skills github repository has a [sample custom vision skill](https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Vision/CustomVision) to help you get started."
+    "You now know how to pass images into skills and return the modified images to the skillset for further processing. \n",
+    "\n",
+    "As a next step, you can start from scratch and build a [custom AML Skill](https://docs.microsoft.com/azure/search/cognitive-search-aml-skill) to perform inferences on images, or use the Custom Vision service to build a skill. The power skills github repository has a [sample custom vision skill](https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Vision/CustomVision) to help you get started."
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -650,7 +675,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.4"
+   "version": "3.7.3"
   }
  },
  "nbformat": 4,

Image-Processing/README.md

@@ -1,23 +1,96 @@
+---
+page_type: sample
+languages:
+  - python
+name: Image processing in Python
+products:
+  - azure
+  - azure-cognitive-search
+description: |
+  Skillsets in Cognitive Search can process images, making that content usable in other scenarios. This sample demonstrates an image file workflow, using OCR and redaction of personal information.
+urlFragment: python-sample-image-processing
+---
+
 # Image Processing Sample
 
-Cognitive Search can enrich images with text or images with other images. This sample demonstrates how to pass images to a custom skill and return images from the  custom skill back to the skillset.
+Cognitive Search can analyze images with text, or images with other images, to create searchable or analyzable text. This sample focuses on a specific aspect of image analysis in a Cognitive Search pipeline: passing images to a custom skill, and return images back to the skillset for further processing.
+
+In this demonstration, you will use a sample JPEG file, services and tools, and fully formulated requests in notebook to perform the following tasks:
+
+1. Crack a sample source JPG file from Blob storage and scrape the image for text, using the [Optical Character Recognition (OCR) skill](https://docs.microsoft.com/azure/search/cognitive-search-skill-ocr) in Cognitive Search.
+1. Analyze the resulting text for personal information, such as phone numbers, using the [PII skill](https://docs.microsoft.com/azure/search/cognitive-search-skill-pii-detection).
+1. Split the text into smaller units and blur the units that contain phone numbers. Use a [custom skill](https://docs.microsoft.com/azure/search/cognitive-search-custom-skill-web-api) for this task.
+1. Reconstitute the image in Blob storage. Use the [TextMerge skill](https://docs.microsoft.com/azure/search/cognitive-search-skill-textmerger) for this step.
+
+Post-OCR, the skillset runs the extracted text through the PII detection skill to identify personal information (phone numbers). The custom skill then obfuscates the phone numbers by accepting as inputs the image, the layout text from  OCR step, and the identified personal information. Output of the custom skill is the image with obfuscated sections. The output is then returned to the skillset and projected to the knowledge store.
+
+The source input and resulting output are stored in Azure Blob storage, so you will need a storage account to complete this tutorial.
+
+Predefined skills, such as OCR skill, are backed by Cognitive Services. You will need the Cognitive Services account name for this tutorial, but because the number of transformations is limited, there is no charge to your account.
+
+Custom skills must be hosted as a URL-accessible module. This tutorial uses Azure Functions to satisfy this requirement, but you could use another mechanism for your own solutions.
+
+## Prerequisites
+
++ [Azure subscription](https://Azure.Microsoft.com/subscription/free)
++ [Azure Cognitive Search service](https://docs.microsoft.com/azure/search/search-create-service-portal) (get the full service endpoint and an admin API key)
++ [Azure Blob storage service](https://docs.microsoft.com/azure/storage/common/storage-account-create) (get the connection string)
++ [Azure Cognitive Services](https://docs.microsoft.com/azure/cognitive-services/cognitive-services-apis-create-account) (get the account name)
++ [Python 3.6+](https://www.python.org/downloads/)
++ [Jupyter Notebook](https://jupyter.org/install)
++ [Visual Studio Code](https://code.visualstudio.com/download) with the [Azure Functions extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions) and the [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
+
+## Configure the components
+
+Before you open the notebook, assemble the resources that are referenced by the skillset.
+
+1. Download the **azure-search-python-samples** repository and extract its contents. 
+
+1. Open the **image-processing** sample folder to find the files used in this sample.
 
-## Redacting PII information from images
+### In Azure portal
 
-This sample deploys a skill to obfuscate or redact phone numbers from images. The skillset contains three skills:
-1. OCR 
-2. PII detection
-3. Custom Skill to redact PII
+1. Set up the data source. In Azure portal, in Azure Blob storage, create a container named "brf-sample", and then upload the sample JPEG file (microsoft.jpg) from the sample folder.
 
-The skillset OCR's the images and runs the extracted text through the PII detection skill to identify PII information. The custom skill then takes the image, layout text from  OCR and the identified PII information to obfuscate the image. The image with the PII infomration obfuscted is then returned to the skillset and projected to the knwoledge store.
+1. From **Keys**, copy the Azure Storage connection string and paste it into NotePad.
 
-## Confingure the components
+1. Navigate to your search service, copy the search endpoint (http://<SERVICE-NAME>.search.windows.net) and an admin API key.
 
-This sample contains a Azure function and a Jupyter Python3 .ipynb file. Start by deploying the Azure function and saving the URL and code. 
+1. Navigate to Cognitive Services, copy the account name. 
 
-The folder also contains a sample image with a phone number. Save this image to a container in a storage account. This container will be your data source for the enrichment pipeline.
+### In Visual Studio Code
 
-Open the norebook in this folder and set the URL and other required variables in the first cell of the notebook, execute the cells of the notebook to configure and run the solution.
+1. Set up the function app that contains the custom code. In Visual Studio code, navigate to the **image-processing** sample folder.
+
+1. Right-click the **SplitImage** folder and select **Deploy to Function App**. You will be prompted for a subscription, region, and other properties that are required to set up the app.
+
+   ![Deploy as function](media/image-process-split-image-deploy-function-app.png)
+
+1. Monitor notifications in the Output window (**View** > **Output**) for a "Deployment successful" message.
+
+1. Still in Visual Studio Code, switch to **Azure: Function** explorer.
+
+1. Open the subscription folder and find the app you just deployed.
+
+1. Open the Functions folder, right-click **ImageSkill HTTP**, and then copy the function URL. Save it Notepad.
+
+   ![Copy function URL](media/image-process-function-url.png)
+
+## Run the Python code in the notebook
+
+1. Open the BFR_Sample_Rest.ipynb file in Jupyter Notebook.
+
+1. In the first cell, paste in the following information:
+
+   * search service endpoint
+   * search service admin API key
+   * Azure Storage connection string
+   * Cognitive Services account name
+   * Blob container that contains the JPEG
+   * Function app URL
+
+1. Run each cell.
 
 ## Validation
-Once the indexer completes, you will see a container `obfuscated` in the knowledge store with the phone number redacted. For comparision the original images are stored in a container `images`.
+
+Once the indexer completes, use the Azure portal and Storage explorer in your Azure Storage account to find the final output. You will see a container `obfuscated` in the knowledge store that contains the same image you started with, but with the phone number redacted. For comparison, the original image is stored in a container `images`.