Skip to content

update amazon bedrock configuration example #1116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 2, 2025
Merged

update amazon bedrock configuration example #1116

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 20 additions & 14 deletions docs/cody/enterprise/model-config-examples.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -695,34 +695,34 @@ In the configuration above,
"displayName": "Anthropic models through AWS Bedrock",
"serverSideConfig": {
"type": "awsBedrock",
"accessToken": "<base64 encoded IAM username:password",
"accessToken": "<ACCESS_KEY_ID>:<SECRET_ACCESS_KEY>",
"endpoint": "<VPC Endpoint URL>",
"region": "us-west-2"
}
}
],
"modelOverrides": [
{
"modelRef": "aws-bedrock::2024-02-29::claude-3-sonnet",
"displayName": "Claude 3 Sonnet",
"modelName": "claude-3-sonnet",
"modelRef": "aws-bedrock::2025-02-19::claude-3-7-sonnet",
"displayName": "Claude 3.7 Sonnet",
"modelName": "anthropic.claude-3-7-sonnet-20250219-v1:0",
"serverSideConfig": {
"type": "awsBedrockProvisionedThroughput",
"arn": "<ARN>" // e.g., arn:aws:bedrock:us-west-2:537452198621:provisioned-model/57z3lgkt1cx2
},
"contextWindow": {
"maxInputTokens": 16000,
"maxOutputTokens": 4000
"maxInputTokens": 132000,
"maxOutputTokens": 8192
},
"capabilities": ["chat", "autocomplete"],
"category": "balanced",
"status": "stable"
},
],
"defaultModels": {
"chat": "aws-bedrock::2024-02-29::claude-3-sonnet",
"codeCompletion": "aws-bedrock::2024-02-29::claude-3-sonnet",
"fastChat": "aws-bedrock::2024-02-29::claude-3-sonnet"
"chat": "aws-bedrock::2025-02-19::claude-3-7-sonnet",
"codeCompletion": "aws-bedrock::2025-02-19::claude-3-7-sonnet",
"fastChat": "aws-bedrock::2025-02-19::claude-3-7-sonnet"
},
}
```
Expand All @@ -731,7 +731,7 @@ In the configuration described above,

- Set up a provider override for Amazon Bedrock, routing requests for this provider directly to the specified endpoint, bypassing Cody Gateway
- Add the `"aws-bedrock::2024-02-29::claude-3-sonnet"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above
- Note: Since the model in the example uses provisioned throughput, specify the ARN in the `serverSideConfig.arn` field of the model override.
- Since the model in the example uses [Amazon Bedrock provisioned throughput](https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html), specify the ARN in the `serverSideConfig.arn` field of the model override.

Provider override `serverSideConfig` fields:

Expand All @@ -740,18 +740,24 @@ Provider override `serverSideConfig` fields:
| `type` | Must be `"awsBedrock"`. |
| `accessToken` | Leave empty to rely on instance role bindings or other AWS configurations in the frontend service. Use `<ACCESS_KEY_ID>:<SECRET_ACCESS_KEY>` for direct credential configuration, or `<ACCESS_KEY_ID>:<SECRET_ACCESS_KEY>:<SESSION_TOKEN>` if a session token is also required. |
| `endpoint` | For pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint. For provisioned throughput, set it to the provisioned VPC endpoint for the bedrock-runtime API (e.g., `https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com`). |
| `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary container cannot access environment variables from the host OS. |
| `region` | The region to use when configuring API clients. The `AWS_REGION` Environment variable must also be configured in the `sourcegraph-frontend` container to match. |

Provisioned throughput for Amazon Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side configuration type. Refer to the [Model Overrides](/cody/enterprise/model-configuration#model-overrides) section for more details.

<Callout type="note">
If using [IAM roles for EC2 / instance role binding](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html),
you may need to increase the [HttpPutResponseHopLimit
](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_InstanceMetadataOptionsRequest.html#:~:text=HttpPutResponseHopLimit) instance metadata option to a higher value (e.g., 2) to ensure that the metadata service can be accessed from the frontend container running in the EC2 instance. See [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-IMDS-existing-instances.html) for instructions.
</Callout>

<Callout type="warning">
We only recommend configuring AWS Bedrock to use an accessToken for
authentication. Specifying no accessToken (e.g. to use [IAM roles for EC2 /
instance role
binding](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html))
is not currently recommended (there is a known performance bug with this
method which will prevent autocomplete from working correctly. (internal
issue: PRIME-662)
is not currently recommended. There is a known performance bug with this
method which will prevent autocomplete from working correctly (internal
issue: CORE-819)
</Callout>

</Accordion>