You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Transparency Note for Auto-Generate Prompt Variants in Prompt Flow
2
+
title: Transparency Note for auto-generate prompt variants in prompt flow
3
3
titleSuffix: Azure Machine Learning
4
-
description: Transparency Note for Auto-Generate Prompt Variants in Prompt Flow
4
+
description: Transparency Note for auto-generate prompt variants in prompt flow
5
5
author: prakharg-msft
6
6
ms.author: prakharg
7
7
manager: omkarm
@@ -11,25 +11,25 @@ ms.date: 10/20/2023
11
11
ms.topic: article
12
12
---
13
13
14
-
# Transparency Note for Auto-Generate Prompt Variants in Prompt Flow
14
+
# Transparency Note for auto-generate prompt variants in prompt flow
15
15
16
16
## What is a Transparency Note?
17
17
18
18
An AI system includes not only the technology, but also the people who use it, the people who are affected by it, and the environment in which it's deployed. Creating a system that is fit for its intended purpose requires an understanding of how the technology works, what its capabilities and limitations are, and how to achieve the best performance. Microsoft's Transparency Notes are intended to help you understand how our AI technology works, the choices system owners can make that influence system performance and behavior, and the importance of thinking about the whole system, including the technology, the people, and the environment. You can use Transparency Notes when developing or deploying your own system, or share them with the people who will use or be affected by your system.
19
19
20
20
Microsoft's Transparency Notes are part of a broader effort at Microsoft to put our AI Principles into practice. To find out more, see the [Microsoft's AI principles](https://www.microsoft.com/ai/responsible-ai).
21
21
22
-
## The basics of Auto-Generate Prompt Variants in Prompt Flow
22
+
## The basics of auto-generate prompt variants in prompt flow
23
23
24
24
### Introduction
25
25
26
-
Prompt engineering is at the center of building applications using Large Language Models. Microsoft's Prompt Flow offers rich capabilities to interactively edit, bulk test, and evaluate prompts with built-in flows to pick the best prompt. With the Auto-Generate Prompt Variants feature in Prompt Flow, we provide the ability to automatically generate variations of a user's base prompt with help of large language models and allow users to test them in Prompt Flow to reach the optimal solution for the user's model and use case needs.
26
+
Prompt engineering is at the center of building applications using Large Language Models. Microsoft's prompt flow offers rich capabilities to interactively edit, bulk test, and evaluate prompts with built-in flows to pick the best prompt. With the auto-generate prompt variants feature in prompt flow, we provide the ability to automatically generate variations of a user's base prompt with help of large language models and allow users to test them in prompt Flow to reach the optimal solution for the user's model and use case needs.
27
27
28
28
### Key terms
29
29
30
30
|**Term**|**Definition**|
31
31
| --- | --- |
32
-
| Prompt flow | Prompt Flow offers rich capabilities to interactively edit prompts and bulk test them with built-in evaluation flows to pick the best prompt. More information available at [What is prompt flow](./overview-what-is-prompt-flow.md)|
32
+
| Prompt flow | Prompt flow offers rich capabilities to interactively edit prompts and bulk test them with built-in evaluation flows to pick the best prompt. More information available at [What is prompt flow](./overview-what-is-prompt-flow.md)|
33
33
| Prompt engineering | The practice of crafting and refining input prompts to elicit more desirable responses from a large language model, particularly in large language models. |
34
34
| Prompt variants | Different versions or modifications of a given input prompt designed to test or achieve varied responses from a large language model. |
35
35
| Base prompt | The initial or primary prompt that serves as a starting point for eliciting response from large language models. In this case it is provided by the user and is modified to create prompt variants. |
@@ -39,43 +39,43 @@ Prompt engineering is at the center of building applications using Large Languag
39
39
40
40
### System behavior
41
41
42
-
The Auto-Generate Prompt Variants feature, as part of the Prompt Flow experience, provides the ability to automatically generate and easily assess prompt variations to quickly find the best prompt for your use case. This feature further empowers Prompt Flow's rich set of capabilities to interactively edit and evaluate prompts, with the goal of simplifying prompt engineering.
42
+
The auto-generate prompt variants feature, as part of the prompt flow experience, provides the ability to automatically generate and easily assess prompt variations to quickly find the best prompt for your use case. This feature further empowers prompt flow's rich set of capabilities to interactively edit and evaluate prompts, with the goal of simplifying prompt engineering.
43
43
44
-
When provided with the user's base prompt the Auto-Generate Prompt Variants feature generates several variations using the generative power of Azure OpenAI models and an internal system prompt. While Azure OpenAI provides content management filters, we recommend verifying any prompts generated before using them in production scenarios.
44
+
When provided with the user's base prompt the auto-generate prompt variants feature generates several variations using the generative power of Azure OpenAI models and an internal system prompt. While Azure OpenAI provides content management filters, we recommend verifying any prompts generated before using them in production scenarios.
45
45
46
46
### Use cases
47
47
48
48
#### Intended uses
49
49
50
-
Auto-Generate Prompt Variants can be used in the following scenarios. The system's intended use is:
50
+
Auto-generate prompt variants can be used in the following scenarios. The system's intended use is:
51
51
52
52
**Generate new prompts from a provided base prompt**: "Generate Variants" feature will allow the users of prompt flow to automatically generate variants of their provided base prompt with help of LLMs (Large Language Models).
53
53
54
54
#### Considerations when choosing a use case
55
55
56
-
**Do not use Auto-Generate Prompt Variants for decisions that might have serious adverse impacts.**
56
+
**Do not use auto-generate prompt variants for decisions that might have serious adverse impacts.**
57
57
58
-
Auto-Generate Prompt Variants was not designed or tested to recommend items that require additional considerations related to accuracy, governance, policy, legal, or expert knowledge as these often exist outside the scope of the usage patterns carried out by regular (non-expert) users. Examples of such use cases include medical diagnostics, banking, or financial recommendations, hiring or job placement recommendations, or recommendations related to housing.
58
+
Auto-generate prompt variants was not designed or tested to recommend items that require additional considerations related to accuracy, governance, policy, legal, or expert knowledge as these often exist outside the scope of the usage patterns carried out by regular (non-expert) users. Examples of such use cases include medical diagnostics, banking, or financial recommendations, hiring or job placement recommendations, or recommendations related to housing.
59
59
60
60
## Limitations
61
61
62
62
Explicitly in the generation of prompt variants, it is important to understand that while AI systems are incredibly valuable tools, they are **non-deterministic**. This means that perfect **accuracy** (the measure of how well the system-generated events correspond to real events that happened in a space) of predictions is not possible. A good model will have high accuracy, but it will occasionally output incorrect predictions. Failure to understand this limitation can lead to over-reliance on the system and unmerited decisions that can impact stakeholders.
63
63
64
-
Furthermore, the prompt variants that are generated using LLMs, are returned to the user as is. It is encouraged to evaluate and compare these variants to determine the best prompt for a given scenario. There are **additional concerns** here because many of the evaluations offered in the Prompt Flow ecosystems also depend on LLMs, potentially further decreasing the utility of any given prompt. Manual review is strongly recommended.
64
+
Furthermore, the prompt variants that are generated using LLMs, are returned to the user as is. It is encouraged to evaluate and compare these variants to determine the best prompt for a given scenario. There are **additional concerns** here because many of the evaluations offered in the prompt flow ecosystems also depend on LLMs, potentially further decreasing the utility of any given prompt. Manual review is strongly recommended.
65
65
66
66
### Technical limitations, operational factors, and ranges
67
67
68
-
As mentioned previously, the Auto-Generate Prompt Variants feature does not provide a measurement or evaluation of the provided prompt variants. It is strongly recommended that the user of this feature evaluates the suggested prompts in the way which best aligns with their specific use case and requirements.
68
+
As mentioned previously, the auto-generate prompt variants feature does not provide a measurement or evaluation of the provided prompt variants. It is strongly recommended that the user of this feature evaluates the suggested prompts in the way which best aligns with their specific use case and requirements.
69
69
70
-
The Auto-Generate Prompt Variants feature is limited to generating a maximum of five variations from a given base prompt. If more are required, additional prompt variants can be generated after modifying the original base prompt.
70
+
The auto-generate prompt variants feature is limited to generating a maximum of five variations from a given base prompt. If more are required, additional prompt variants can be generated after modifying the original base prompt.
71
71
72
-
Auto-Generate Prompt Variants only supports Azure OpenAI models at this time. In addition to limiting users to only the models which are supported by Azure OpenAI, it also limits content to what is acceptable in terms of the Azure OpenAI's content management policy. Uses outside of this policy are not supported by this feature.
72
+
Auto-generate prompt variants only supports Azure OpenAI models at this time. In addition to limiting users to only the models which are supported by Azure OpenAI, it also limits content to what is acceptable in terms of the Azure OpenAI's content management policy. Uses outside of this policy are not supported by this feature.
73
73
74
74
## System performance
75
75
76
-
Performance for the Auto-Generate Prompt Variants feature is determined by the user's use case in each individual scenario – in this way the feature does not evaluate each prompt or generate metrics.
76
+
Performance for the auto-generate prompt variants feature is determined by the user's use case in each individual scenario – in this way the feature does not evaluate each prompt or generate metrics.
77
77
78
-
Operating in the Prompt Flow ecosystem, which focuses on Prompt Engineering, provides a strong story for error handling. Often retrying the operation will resolve an error. One error which might arise specific to this feature is response filtering from the Azure OpenAI resource for content or harm detection, this would happen in the case that content in the base prompt is determined to be against Azure OpenAI's content management policy. To resolve these errors please update the base prompt in accordance with the guidance at [Azure OpenAI Service content filtering](/azure/ai-services/openai/concepts/content-filter).
78
+
Operating in the prompt flow ecosystem, which focuses on Prompt Engineering, provides a strong story for error handling. Often retrying the operation will resolve an error. One error which might arise specific to this feature is response filtering from the Azure OpenAI resource for content or harm detection, this would happen in the case that content in the base prompt is determined to be against Azure OpenAI's content management policy. To resolve these errors please update the base prompt in accordance with the guidance at [Azure OpenAI Service content filtering](/azure/ai-services/openai/concepts/content-filter).
79
79
80
80
### Best practices for improving system performance
81
81
@@ -85,23 +85,23 @@ To improve performance there are several parameters which can be modified, depen
85
85
-**Number of Variants**: This parameter specifies how many variants to generate. A larger number of variants will produce more prompts and therefore the likelihood of finding the best prompt for the use case.
86
86
-**Base Prompt**: Since this tool generates variants of the provided base prompt, a strong base prompt can set up the tool to provide the maximum value for your case. Please review the guidelines at Prompt engineering techniques with [Azure OpenAI](/azure/ai-services/openai/concepts/advanced-prompt-engineering).
87
87
88
-
## Evaluation of Auto-Generate Prompt Variants
88
+
## Evaluation of auto-generate prompt variants
89
89
90
90
### Evaluation methods
91
91
92
-
The Auto-Generate Prompt Variants feature been testing by the internal development team, targeting fit for purpose and harm mitigation.
92
+
The auto-generate prompt variants feature been testing by the internal development team, targeting fit for purpose and harm mitigation.
93
93
94
94
### Evaluation results
95
95
96
96
Evaluation of harm management showed staunch support for the combination of system prompt and Azure Open AI content management policies in actively safe-guarding responses. Additional opportunities to minimize the chance and risk of harms can be found in the Microsoft documentation: [Azure OpenAI Service abuse monitoring](/azure/ai-services/openai/concepts/abuse-monitoring) and [Azure OpenAI Service content filtering](/azure/ai-services/openai/concepts/content-filter).
97
97
98
98
Fit for purpose testing supported the quality of generated prompts from creative purposes (poetry) and chat-bot agents. The reader is cautioned from drawing sweeping conclusions given the breadth of possible base prompt and potential use cases. As previously mentioned, please use evaluations appropriate to the required use cases and ensure a human reviewer is part of the process.
99
99
100
-
## Evaluating and integrating Auto-Generate Prompt Variants for your use
100
+
## Evaluating and integrating auto-generate prompt variants for your use
101
101
102
-
The performance of the Auto-Generate Prompt Variants feature will vary depending on the base prompt and use case in it is used. True usage of the generated prompts will depend on a combination of the many elements of the system in which the prompt is used.
102
+
The performance of the auto-generate prompt variants feature will vary depending on the base prompt and use case in it is used. True usage of the generated prompts will depend on a combination of the many elements of the system in which the prompt is used.
103
103
104
-
To ensure optimal performance in their scenarios, customers should conduct their own evaluations of the solutions they implement using Auto-Generate Prompt Variants. Customers should, generally, follow an evaluation process that:
104
+
To ensure optimal performance in their scenarios, customers should conduct their own evaluations of the solutions they implement using auto-generate prompt variants. Customers should, generally, follow an evaluation process that:
105
105
106
106
- Uses internal stakeholders to evaluate any generated prompt.
107
107
- Uses internal stakeholders to evaluate results of any system which uses a generated prompt.
@@ -113,6 +113,6 @@ To ensure optimal performance in their scenarios, customers should conduct their
113
113
-[Microsoft responsible AI resources](https://www.microsoft.com/ai/responsible-ai-resources)
114
114
-[Microsoft Azure Learning courses on responsible AI](/training/paths/responsible-ai-business-principles/)
115
115
116
-
## Learn more about Auto-Generate Prompt Variants
116
+
## Learn more about auto-generate prompt variants
117
117
-[What is prompt flow](./overview-what-is-prompt-flow.md)
0 commit comments