Skip to content

Commit b9b51d6

Browse files
committed
docs: Add LangSmith Prompt Hub repetitions feature documentation
1 parent cc934bd commit b9b51d6

File tree

2 files changed

+24
-1
lines changed

2 files changed

+24
-1
lines changed

journals/2025_11_26.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,5 @@
33
- [Access Amazon Bedrock foundation models - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html#model-access-permissions)
44
- > To modify access to Amazon Bedrock foundation models, you first need to attach an identity-based IAM policy with the following [AWS Marketplace actions](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsmarketplace.html#awsmarketplace-actions-as-permissions) to the IAM role that allows access to Amazon Bedrock.
55
- > **When you first invoke an Amazon Bedrock serverless model** served from AWS Marketplace in an account, **Bedrock attempts to automatically enable the model for your account**. **For this auto-enablement to work, AWS Marketplace permissions are required**.
6-
- > If **you can't assume AWS Marketplace permission**, someone with AWS Marketplace permissions **must enable the model for the account as a one-time step (either manually or via auto-enablement)**. Once enabled, **all users in the account can invoke the model without needing AWS Marketplace permissions**. Users don't need AWS Marketplace subscription permissions to invoke models after they've been enabled. These permissions are only required the first time a model is being used in an account.
6+
- > If **you can't assume AWS Marketplace permission**, someone with AWS Marketplace permissions **must enable the model for the account as a one-time step (either manually or via auto-enablement)**. Once enabled, **all users in the account can invoke the model without needing AWS Marketplace permissions**. Users don't need AWS Marketplace subscription permissions to invoke models after they've been enabled. These permissions are only required the first time a model is being used in an account.
7+
- [[LangSmith/PromptHub/How To/Use Repetitions to Reduce Variability]]
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
- # LangSmith Prompt Hub - Using Repetitions to Reduce Variability
2+
- ## Overview
3+
- LangSmith Prompt Hub supports running the same prompt multiple times to reduce variability in results
4+
- This feature is available in **Evaluate Mode** when testing prompts
5+
- ## How It Works
6+
- **Repetitions Setting:** Configure the number of times to run the same prompt
7+
- Default appears to be configurable (shown as "5" in the UI)
8+
- Controlled via a slider interface in the Prompt Hub evaluation settings
9+
- **Purpose:** Running the same prompt multiple times helps:
10+
- Reduce variability in results
11+
- Get more consistent evaluation metrics
12+
- Better understand the stability of prompt outputs
13+
- ## Usage
14+
- Access through the **Evaluate Mode** interface in LangSmith Prompt Hub
15+
- The Repetitions setting appears as a card/panel in the evaluation configuration
16+
- Adjust the number of repetitions using the slider control
17+
- The "Learn more" link provides additional documentation
18+
- ## Benefits
19+
- **Statistical Reliability:** Multiple runs provide better statistical confidence in results
20+
- **Variance Reduction:** Helps identify if prompt outputs are consistent or highly variable
21+
- **Better Evaluation:** More data points lead to more reliable evaluation metrics
22+

0 commit comments

Comments
 (0)