Merge pull request #115263 from MicrosoftDocs/release-build-cogserv-personalizer

Release build cogserv personalizer

Author: SyntaxC4

articles/cognitive-services/personalizer/concept-apprentice-mode.md

@@ -0,0 +1,90 @@
+---
+title: Apprentice mode - Personalizer
+description:
+ms.topic: conceptual
+ms.date: 05/01/2020
+---
+
+# Use Apprentice mode to train Personalizer without affecting your existing application
+
+Due to the nature of **real-world** Reinforcement Learning, a Personalizer model can only be trained in a production environment. When deploying a new use case, the Personalizer model is not performing efficiently because it takes time for the model to be sufficiently trained.  **Apprentice mode** is a learning behavior that eases this situation and allows you to gain confidence in the model – without the developer changing any code.
+
+[!INCLUDE [Important Blue Box - Apprentice mode pricing tier](./includes/important-apprentice-mode.md)]
+
+## What is Apprentice mode?
+
+Similar to how an apprentice learns from a master, and with experience can get better; Apprentice mode is a _behavior_ that lets Personalizer learn by observing the results obtained from existing application logic.
+
+Personalizer trains by mimicking the same output as the application. As more events flow, Personalizer can _catch up_ to the existing application without impacting the existing logic and outcomes. Metrics, available from the Azure portal and the API, help you understand the performance as the model learns.
+
+Once Personalizer has learned and attained a certain level of understanding, the developer can change the behavior from Apprentice mode to Online mode. At that time, Personalizer starts influencing the actions in the Rank API.
+
+## Purpose of Apprentice Mode
+
+Apprentice mode gives you trust in the Personalizer service and its machine learning capabilities, and provides reassurance that the service is sent information that can be learned from – without risking online traffic.
+
+The two main reasons to use Apprentice mode are:
+
+* Mitigating **Cold Starts**: Apprentice mode helps manage and assess the cost of a "new" model's learning time -  when it is not returning the best action and not achieved a satisfactory level of effectiveness of around 75-85%.
+* **Validating Action and Context Features**: Features sent in actions and context may be inadequate or inaccurate - too little, too much, incorrect, or too specific to train Personalizer to attain the ideal effectiveness rate. Use [feature evaluations](concept-feature-evaluation.md) to find and fix issues with features.
+
+## When should you use Apprentice mode?
+
+Use Apprentice mode to train Personalizer to improve its effectiveness through the following scenarios while leaving the experience of your users unaffected by Personalizer:
+
+* You are implementing Personalizer in a new use case.
+* You have significantly changed the features you send in Context or Actions.
+* You have significantly changed when and how you calculate rewards.
+
+Apprentice mode is not an effective way of measuring the impact Personalizer is having on reward scores. To measure how effective Personalizer is at choosing the best possible action for each Rank call, use [Offline evaluations](concepts-offline-evaluation.md).
+
+## Who should use Apprentice mode?
+
+Apprentice mode is useful for developers, data scientists and business decision makers:
+
+* **Developers** can use Apprentice mode to make sure the Rank and Reward APIs are being used correctly in the application, and that features being sent to Personalizer from the application contains no bugs, or non-relevant features such as a timestamp or UserID element.
+
+* **Data scientists** can use Apprentice mode to validate that the features are effective to train the Personalizer models, that the reward wait times aren’t too long or short.
+
+* **Business Decision Makers** can use Apprentice mode to assess the potential of Personalizer to improve results (i.e. rewards) compared to existing business logic. This allows them to make a informed decision impacting user experience, where real revenue and user satisfaction are at stake.
+
+## Comparing Behaviors - Apprentice mode and Online mode
+
+Learning when in Apprentice mode differs from Online mode in the following ways.
+
+|Area|Apprentice mode|Online mode|
+|--|--|--|
+|Impact on User Experience|You can use existing user behavior to train Personalizer by letting it observe (not affect) what your **default action** would have been and the reward it obtained. This means your users’ experience and the business results from them won’t be impacted.|Display top action returned from Rank call to affect user behavior.|
+|Learning speed|Personalizer will learn more slowly when in Apprentice mode than when learning in Online mode. Apprentice mode can only learn by observing the rewards obtained by your **default action**, which limits the speed of learning, as no exploration can be performed.|Learns faster because it can both exploit the current model and explore for new trends.|
+|Learning effectiveness "Ceiling"|Personalizer can approximate, very rarely match, and never exceed the performance of your base business logic (the reward total achieved by the **default action** of each Rank call).|Personalizer should exceed applications baseline, and over time where it stalls you should conduct on offline evaluation and feature evaluation to continue to get improvements to the model. |
+|Rank API value for rewardActionId|The users' experience doesn’t get impacted, as _rewardActionId_ is always the first action you send in the Rank request. In other words, the Rank API does nothing visible for your application during Apprentice mode. Reward APIs in your application should not change how it uses the Reward API between one mode and another.|Users' experience will be changed by the _rewardActionId_ that Personalizer chooses for your application. |
+|Evaluations|Personalizer keeps a comparison of the reward totals that your default business logic is getting, and the reward totals Personalizer would be getting if in Online mode at that point. A comparison is available in the Azure portal for that resource|Evaluate Personalizer’s effectiveness by running [Offline evaluations](concepts-offline-evaluation.md), which let you compare the total rewards Personalizer has achieved against the potential rewards of the application’s baseline.|
+
+A note about apprentice mode's effectiveness:
+
+* Personalizer's effectiveness in Apprentice mode will rarely achieve near 100% of the application's baseline; and never exceed it.
+* Best practices would be not to try to get to 100% attainment; but a range of 75 – 85% should be targeted depending on the use case.
+
+## Using Apprentice mode to train with historical data
+
+If you have a significant amount of historical data, you’d like to use to train Personalizer, you can use Apprentice mode to replay the data through Personalizer.
+
+Set up the Personalizer in Apprentice Mode and create a script that calls Rank with the actions and context features from the historical data. Call the Reward API based on your calculations of the records in this data. You will need approximately 50,000 historical events to see some results but 500,000 is recommended for higher confidence in the results.
+
+When training from historical data, it is recommended that the data sent in (features for context and actions, their layout in the JSON used for Rank requests, and the calculation of reward in this training data set), matches the data (features and calculation of reward) available from the existing application.
+
+Offline and post-facto data tends to be more incomplete and noisier and differs in format. While training from historical data is possible, the results from doing so may be inconclusive and not a good predictor of how well Personalizer will learn, especially if the features vary between past data and the existing application.
+
+Typically for Personalizer, when compared to training with historical data, changing behavior to Apprentice mode and learning from an existing application is a more effective path to having an effective model, with less labor, data engineering, and cleanup work.
+
+## Using Apprentice Mode versus A/B Tests
+
+It is only useful to do A/B tests of Personalizer treatments once it has been validated and is learning in Online mode. In Apprentice mode, only the **default action** is used, which means all users would effectively see the control experience.
+
+Even if Personalizer is just the _treatment_, the same challenge is present when validating the data is good for training Personalizer. Apprentice mode could be used instead, with 100% of traffic, and with all users getting the control (unaffected) experience.
+
+Once you have a use case using Personalizer and learning online, A/B experiments allow you to do controlled cohorts and scientific comparison of results that may be more complex than the signals used for rewards. An example question an A/B test could answer is: `In a retail website, Personalizer optimizes a layout and gets more users to _check out_ earlier, but does this reduce total revenue per transaction?`
+
+## Next steps
+
+* Learn about [active and inactive events](concept-active-inactive-events.md)

articles/cognitive-services/personalizer/how-to-learning-behavior.md

@@ -0,0 +1,67 @@
+---
+title: Configure learning behavior
+description: Apprentice mode gives you confidence in the Personalizer service and its machine learning capabilities, and provides metrics that the service is sent information that can be learned from – without risking online traffic.
+ms.topic: how-to
+ms.date: 05/01/2020
+---
+
+# Configure the Personalizer learning behavior
+
+[Apprentice mode](concept-apprentice-mode.md) gives you trust and confidence in the Personalizer service and its machine learning capabilities, and provides assurance that the service is sent information that can be learned from – without risking online traffic.
+
+[!INCLUDE [Important Blue Box - Apprentice mode pricing tier](./includes/important-apprentice-mode.md)]
+
+## Configure Apprentice mode
+
+1. Sign in to the [Azure portal](https://portal.azure.com), for your Personalizer resource.
+
+1. On the **Configuration** page, on the **Learning behavior** tab, select **Return baseline action, learn as an apprentice** then select **Save**.
+
+> [!div class="mx-imgBorder"]
+> ![Screenshot of configuring apprentice mode learning behavior in Azure portal](media/settings/configure-learning-behavior-azure-portal.png)
+
+## Changes to the existing application
+
+Your existing application shouldn't change how it currently selects actions to display or how the application determines the value, **reward** of that action. The only change to the application might be the order of the actions sent to Personalizer's Rank API. The action your application currently displays is sent as the _first action_ in the action list. The [Rank API](https://westus2.dev.cognitive.microsoft.com/docs/services/personalizer-api/operations/Rank) uses this first action to train your Personalizer model.
+
+### Configure your application to call the Rank API
+
+In order to add Personalizer to your application, you need to call the Rank and Reward APIs.
+
+1. Add the [Rank API](https://westus2.dev.cognitive.microsoft.com/docs/services/personalizer-api/operations/Rank) call after the point in your existing application logic where you determine the list of actions and their features. The first action in the actions list needs to be the action selected by your existing logic.
+
+1. Configure your code to display the action associated with the Rank API response's **Reward Action ID**.
+
+### Configure your application to call Reward API
+
+1. Use your existing business logic to calculate the **reward** of the displayed action. The value needs to be in the range from 0 to 1. Send this reward to Personalizer using the [Reward API](https://westus2.dev.cognitive.microsoft.com/docs/services/personalizer-api/operations/Reward). The reward value is not expected immediately and can be delayed over a time period - depending on your business logic.
+
+1. If you don't return the reward within the configured **Reward wait time**, the default reward will be used instead.
+
+## Evaluate Apprentice mode
+
+In the Azure portal, on the **Evaluations** page for your Personalizer resource, review the **Current learning behavior performance**.
+
+> [!div class="mx-imgBorder"]
+> ![Screenshot of reviewing evaluation of apprentice mode learning behavior in Azure portal](media/settings/evaluate-apprentice-mode.png)
+
+Apprentice mode provides the following **evaluation metrics**:
+* **Baseline – average reward**:  Average rewards of the application’s default (baseline).
+* **Personalizer – average reward**: Average of total rewards Personalizer would potentially have reached.
+* **Reward achievement ratio over most recent 1000 events**: Ratio of Baseline and Personalizer reward – normalized over the most recent 1000 events.
+
+## Evaluate Apprentice mode features
+
+Evaluate the features using an [offline evaluation](how-to-offline-evaluation.md).
+
+## Switch behavior to Online mode
+
+When you determine Personalizer is trained with an average of 75-85% rolling average, the model is ready to switch to Online mode.
+
+In the Azure portal for your Personalizer resource, on the **Configuration** page, on the **Learning behavior** tab, select **Return the best action** then select **Save**.
+
+You do not need to make any changes to the Rank and Reward API calls.
+
+## Next steps
+
+* [Manage model and learning settings](how-to-manage-model.md)

articles/cognitive-services/personalizer/how-to-settings.md

@@ -2,7 +2,7 @@
 title: Configure Personalizer
 description: Service configuration includes how the service treats rewards, how often the service explores, how often the model is retrained, and how much data is stored.
 ms.topic: conceptual
-ms.date: 02/19/2020
+ms.date: 04/29/2020
 ---
 
 # Configure Personalizer learning loop
@@ -14,6 +14,23 @@ Configure the learning loop on the **Configuration** page, in the Azure portal f
 <a name="configure-service-settings-in-the-azure-portal"></a>
 <a name="configure-reward-settings-for-the-feedback-loop-based-on-use-case"></a>
 
+## Planning configuration changes
+
+Because some configuration changes [reset your model](#settings-that-include-resetting-the-model), you should plan your configuration changes.
+
+If you plan to use [Apprentice mode](concept-apprentice-mode.md), make sure to review your Personalizer configuration before switching to Apprentice mode.
+
+<a name="clear-data-for-your-learning-loop"></a>
+
+## Settings that include resetting the model
+
+The following actions trigger a retraining of the model using data available upto the last 2 days.
+
+* Reward
+* Exploration
+
+To [clear](how-to-manage-model.md) all your data, use the **Model and learning settings** page.
+
 ## Configure rewards for the feedback loop
 
 Configure the service for your learning loop's use of rewards. Changes to the following values will reset the current Personalizer model and retrain it with the last 2 days of data.
@@ -61,16 +78,7 @@ After changing this value, make sure to select **Save**.
 
 After changing this value, make sure to select **Save**.
 
-<a name="clear-data-for-your-learning-loop"></a>
-
-## Settings that include resetting the model
-
-The following actions include an immediate retraining of the model with the last 2 days of data.
-
-* Reward
-* Exploration
 
-To [clear](how-to-manage-model.md) all your data, use the **Model and learning settings ** page.
 
 ## Next steps
 

articles/cognitive-services/personalizer/includes/important-apprentice-mode.md

@@ -0,0 +1,16 @@
+---
+title: include file
+description: include file
+services: cognitive-services
+author: diberry
+manager: nitinme
+ms.service: cognitive-services
+ms.subservice: personalizer
+ms.topic: include
+ms.custom: include file
+ms.date: 04/29/2020
+ms.author: diberry
+---
+
+> [!Important]
+> **Apprentice mode** (in Public Preview) is only available on the E0 pricing tier. Please see pricing for details. You can select the E0 tier at resource creation or upgrade to E0 from the Subscriptions tab in the Azure portal. If you are on another tier, and upgrade to E0, your existing Personalizer resources will automatically be migrated to the E0 tier.

articles/cognitive-services/personalizer/index.yml

@@ -5,7 +5,7 @@ summary: Learn how to use Personalizer to allow your application to choose the b
 
 metadata:
   ms.topic: landing-page
-  ms.date: 10/03/2019
+  ms.date: 05/01/2020
   ms.author: nitinme
   author: nitinme
   ms.service: cognitive-services
@@ -99,6 +99,8 @@ landingContent:
     linkLists:
       - linkListType: concept
         links:
+          - text: Use Apprentice mode
+            url: how-to-learning-behavior.md
           - text: Improve loop with offline evaluations
             url: concepts-offline-evaluation.md
           - text: Active and inactive events