-
Notifications
You must be signed in to change notification settings - Fork 10.4k
[AIG] How to add feedback documentatation #17438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 4 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
0f9d15c
How to add feedback documentatation
daisyfaithauma c63bb66
grammar edit
daisyfaithauma 3c63167
Update add-human-feedback.mdx
kathayl 20dd23f
Added a Note
daisyfaithauma 62cfbf7
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma 0b5ff38
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma b37c865
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma 29f4a0b
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma 43885f2
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma 95f77f6
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma b01a23e
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma 5e4c7da
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma fdedd84
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma 033e1b2
Update src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
daisyfaithauma File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
65 changes: 65 additions & 0 deletions
65
src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| --- | ||
| pcx_content_type: how-to | ||
| title: Add Human Feedback | ||
| sidebar: | ||
| order: 3 | ||
| badge: | ||
| text: Beta | ||
| --- | ||
|
|
||
| Human feedback is a valuable metric to assess the performance of your AI models. By incorporating human feedback, you can gain deeper insights into how the model's responses are perceived and how well it performs from a user-centric perspective. This feedback can then be used in evaluations to calculate performance metrics, driving optimization and ultimately enhancing the reliability, accuracy, and efficiency of your AI application. | ||
|
|
||
| Human feedback measures the performance of your dataset based on direct human input. The metric is calculated as the percentage of positive feedback (thumbs up) given on logs, which are annotated in the Logs tab of the Cloudflare dashboard. This feedback helps refine model performance by considering real-world evaluations of its output. | ||
|
|
||
| This tutorial will guide you through the process of adding human feedback to your evaluations in AI Gateway. | ||
|
|
||
| ## 1. Log into the dashboard | ||
|
|
||
| 1. Log into the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account. | ||
| 2. Navigate to the AI Gateway section. | ||
daisyfaithauma marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## 2. Access the Logs tab | ||
|
|
||
| 1. In the dashboard, go to the Logs tab under the AI Gateway section. | ||
daisyfaithauma marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| 2. The Logs tab displays all logs associated with your datasets. These logs show key information, including: | ||
| - Timestamp: When the interaction occurred. | ||
| - Status: Whether the request was successful, cached, or failed. | ||
| - Model: The model used in the request. | ||
| - Tokens: The number of tokens consumed by the response. | ||
| - Cost: The cost based on token usage. | ||
| - Duration: The time taken to complete the response. | ||
| - Feedback: Where you can provide human feedback on each log. | ||
|
|
||
| ## 3. Provide human feedback | ||
|
|
||
| 1. In the Logs tab, click on the log entry you want to review. This expands the log, allowing you to see more detailed information. | ||
daisyfaithauma marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| 2. In the expanded log, you can view additional details such as: | ||
| - The user prompt. | ||
| - The model's response. | ||
daisyfaithauma marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - HTTP response details. | ||
| - Endpoint information. | ||
| 3. You will see two icons: | ||
| - Thumbs up: Indicates positive feedback. | ||
| - Thumbs down: Indicates negative feedback. | ||
| 4. Click either the thumbs up or thumbs down icon based on how you rate the model's response for that particular log entry. | ||
daisyfaithauma marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## 4. Evaluate human feedback | ||
|
|
||
| After providing feedback on your logs, it becomes part of the evaluation process. | ||
daisyfaithauma marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| When you run an evaluation (as outlined in the [Set Up Evaluations](/ai-gateway/evaluations/set-up-evaluations/) guide), the human feedback metric will be calculated based on the percentage of logs that received thumbs-up feedback. | ||
|
|
||
| :::note[Note] | ||
|
|
||
| You need to select human feedback as an evaluator to receive its metrics. | ||
|
|
||
| ::: | ||
|
|
||
| ## 5. Review results | ||
|
|
||
| After running the evaluation,review results on the Evaluations tab. | ||
daisyfaithauma marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| You will be able to see the model's performance based on cost, speed, and now human feedback, represented as the percentage of positive feedback (thumbs up). | ||
daisyfaithauma marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| The human feedback score is displayed as a percentage, showing how manyy of the dataset's responses were rated positively. | ||
daisyfaithauma marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| For more information on running evaluations, refer to the documentation [Set Up Evaluations](/ai-gateway/evaluations/set-up-evaluations/). | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.