feat(ai-proxy): add AWS Bedrock Converse-API Driver by tysoekong · Pull Request #13354 · Kong/kong

tysoekong · 2024-07-09T21:57:25Z

Summary

Supersedes #13054 which was completely broken.

Adds AWS Bedrock "Converse API" support to Kong AI Gateway.

Checklist

The Pull Request has tests
A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

AG-14

subnetmarco · 2024-07-17T08:12:06Z

@fffonion let's start the review process on this large PR.

kong/llm/drivers/bedrock.lua

kong/llm/init.lua

kong/tools/aws_stream.lua

jschmid1

1/n

kong/clustering/compat/checkers.lua

kong/llm/drivers/shared.lua

kong/plugins/ai-proxy/handler.lua

kong/llm/drivers/bedrock.lua

kong/llm/drivers/gemini.lua

kong/llm/drivers/shared.lua

kong/llm/schemas/init.lua

tysoekong · 2024-07-26T16:18:14Z

@fffonion I done all your second round comments

kong/plugins/ai-proxy/handler.lua

kong/tools/aws_stream.lua

fffonion · 2024-07-29T07:55:16Z

LGTM, please make sure comments from other reviewers are addressed : )

Co-authored-by: Keery Nie <windmgc@gmail.com>

team-gateway-bot · 2024-07-30T07:48:40Z

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13354-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13354-to-master-to-upstream
git checkout -b cherry-pick-13354-to-master-to-upstream
ancref=$(git merge-base 61e2c761d7ec7757552b54851a992b5a8740f4be 8184ee0faaee6adf6ff71de0f8f5ce88fbfa326e)
git cherry-pick -x $ancref..8184ee0faaee6adf6ff71de0f8f5ce88fbfa326e

fffonion · 2024-07-30T07:49:07Z

cc @ttyS0e I will do the cherrypicks to EE because there are couple of PRs that needs to cooridnate for conflict.

fffonion · 2024-07-30T09:01:45Z

https://github.com/Kong/kong-ee/pull/9824

Supersedes #13054 which was completely broken. Adds AWS Bedrock "Converse API" support to Kong AI Gateway. AG-14 Co-authored-by: Jack Tysoe <91137069+tysoekong@users.noreply.github.com>

pull-request-size bot added the size/XL label Jul 9, 2024

github-actions bot added schema-change-noteworthy cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee plugins/ai-proxy plugins/ai-request-transformer plugins/ai-response-transformer labels Jul 9, 2024

github-actions bot assigned tysoekong Jul 9, 2024

fffonion reviewed Jul 22, 2024

View reviewed changes

kong/llm/drivers/bedrock.lua Show resolved Hide resolved

kong/llm/drivers/bedrock.lua Outdated Show resolved Hide resolved

kong/llm/init.lua Show resolved Hide resolved

kong/llm/init.lua Outdated Show resolved Hide resolved

kong/tools/aws_stream.lua Show resolved Hide resolved

tysoekong force-pushed the feat/ai_proxy_aws_bedrock branch from 2c5883a to 1a541a3 Compare July 24, 2024 15:30

pull-request-size bot added size/XXL and removed size/XL labels Jul 24, 2024

tysoekong force-pushed the feat/ai_proxy_aws_bedrock branch 3 times, most recently from fa52c16 to f499d3f Compare July 25, 2024 13:02

tysoekong changed the title ~~Feat/ai proxy aws bedrock~~ feat: AI Proxy, AWS Bedrock Converse API Driver Jul 25, 2024

github-actions bot added the core/clustering label Jul 25, 2024

tysoekong changed the title ~~feat: AI Proxy, AWS Bedrock Converse API Driver~~ feat: AI Proxy, AWS Bedrock Converse AI Driver Jul 25, 2024

jschmid1 reviewed Jul 25, 2024

View reviewed changes

kong/clustering/compat/checkers.lua Outdated Show resolved Hide resolved

kong/clustering/compat/checkers.lua Outdated Show resolved Hide resolved

kong/llm/drivers/shared.lua Outdated Show resolved Hide resolved

tysoekong force-pushed the feat/ai_proxy_aws_bedrock branch from 9d864e7 to f704158 Compare July 25, 2024 19:12

tysoekong changed the title ~~feat: AI Proxy, AWS Bedrock Converse AI Driver~~ feat: AI Proxy, AWS Bedrock Converse-API Driver Jul 25, 2024

windmgc reviewed Jul 26, 2024

View reviewed changes

kong/plugins/ai-proxy/handler.lua Outdated Show resolved Hide resolved

kong/plugins/ai-proxy/handler.lua Outdated Show resolved Hide resolved

kong/plugins/ai-proxy/handler.lua Outdated Show resolved Hide resolved

kong/plugins/ai-proxy/handler.lua Outdated Show resolved Hide resolved

fffonion reviewed Jul 26, 2024

View reviewed changes

kong/llm/drivers/bedrock.lua Outdated Show resolved Hide resolved

kong/llm/drivers/gemini.lua Outdated Show resolved Hide resolved

kong/llm/drivers/shared.lua Outdated Show resolved Hide resolved

kong/llm/schemas/init.lua Outdated Show resolved Hide resolved

windmgc reviewed Jul 29, 2024

View reviewed changes

kong/plugins/ai-proxy/handler.lua Outdated Show resolved Hide resolved

oowl reviewed Jul 29, 2024

View reviewed changes

kong/tools/aws_stream.lua Outdated Show resolved Hide resolved

fffonion changed the title ~~feat: AI Proxy, AWS Bedrock Converse-API Driver~~ feat(ai-proxy): add AWS Bedrock Converse-API Driver Jul 29, 2024

fffonion approved these changes Jul 29, 2024

View reviewed changes

bedrock changelog

25c59b9

tysoekong and others added 19 commits July 29, 2024 15:38

add complex provider signing support to ai-transformer plugin functions

2d04d68

add bedrock provider options to shared schema

e21d531

add aws provider to keybastion loose table

14e6fba

adjust sdk to use new streaming parser format

1942f0a

add bedrock format test scaffolding

810891e

add bedrock driver class itself

c037480

add bedrock drivers to rockspec

0cb31f2

fix(ai-proxy)(general): body_filter should not run on failure

9252142

add aws streaming format parser test

7275453

feat(ai-proxy): bedrock unit tests

152b98c

fix(ai-proxy): fix gemini streaming; fix gemini analytics

d044735

fix(ai-proxy): bedrock: cluster compat

154d31d

Update bedrock.lua

5e00426

fix(clustering): faster comparisons for unsupported AI providers

da4fcab

fix(clustering): faster comparisons for unsupported AI providers

6826842

fix(ai-proxy): bedrock: refactor auth chain

dcd1d21

lint

8d4ded7

fix(ai-proxy): (bedrock, gemini): comments rollup

334df7f

Update kong/plugins/ai-proxy/handler.lua

8184ee0

Co-authored-by: Keery Nie <windmgc@gmail.com>

tysoekong force-pushed the feat/ai_proxy_aws_bedrock branch from 1b85373 to 8184ee0 Compare July 29, 2024 14:38

windmgc approved these changes Jul 30, 2024

View reviewed changes

fffonion merged commit 264341d into master Jul 30, 2024

fffonion deleted the feat/ai_proxy_aws_bedrock branch July 30, 2024 07:48

github-actions bot added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Jul 30, 2024

AndyZhang0707 removed the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Aug 1, 2024

randmonkey mentioned this pull request Aug 5, 2024

Failures in integration tests against Kong nightly image Kong/kubernetes-ingress-controller#6370

Closed

1 task

Conversation

tysoekong commented Jul 9, 2024

Summary

Checklist

Issue reference

Uh oh!

subnetmarco commented Jul 17, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jschmid1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tysoekong commented Jul 26, 2024

Uh oh!

Uh oh!

Uh oh!

fffonion commented Jul 29, 2024

Uh oh!

team-gateway-bot commented Jul 30, 2024

Uh oh!

fffonion commented Jul 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fffonion commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

fffonion commented Jul 30, 2024 •

edited

Loading