Skip to content

installation-deployments.md

Latest commit

 

History

History
486 lines (375 loc) · 15.5 KB

installation-deployments.md

File metadata and controls

486 lines (375 loc) · 15.5 KB

How to Deploy the Gateway?

  1. Managed Deployment by Portkey for quick setup without infrastructure concerns
  2. Local Deployment for complete control & customization
  3. Enterprise Deployment for advanced features and dedicated support

Managed Deployment

Portkey runs this same Gateway on our API and processes billions of tokens daily. Portkey's API is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.

Sign up for the free developer plan here or discuss here for enterprise deployments.

Check out the API docs here.

Local Deployment

  1. Do NPM or Bun Install
  2. Run a Node.js Server
  3. Deploy on App Stack
  4. Deploy on Cloudflare Workers
  5. Deploy using Docker
  6. Deploy using Docker Compose
  7. Deploy on Replit
  8. Deploy on Zeabur
  9. Deploy with Supabase Functions
  10. Deploy using Fastly

Node

$ npx @portkey-ai/gateway

Bun

$ bunx @portkey-ai/gateway

NodeJS Server

  1. Clone the Repository
git clone https://github.com/portkey-ai/gateway
  1. Install the NPM Dependencies
cd gateway
npm i
npm run build
  1. Run the Server
node build/start-server.js

Deploy to App Stack

F5 Distributed Cloud

  1. Create an App Stack Site

  2. Retrieve the global kubeconfig

export DISTRIBUTED_CLOUD_TENANT=mytenantname
# find tenant id in the F5 Distributed Cloud GUI at
# Account -> Account Settings -> Tenant Overview -> Tenant ID
export DISTRIBUTED_CLOUD_TENANT_ID=mytenantnamewithextensionfoundintheconsole
# create an API token in the F5 Distributed Cloud GUI at
# Account -> Account Settings -> Credentials -> Add Credentials 
# set Credential Type to API Token, not API Certificate
export DISTRIBUTED_CLOUD_API_TOKEN=myapitoken
export DISTRIBUTED_CLOUD_SITE_NAME=appstacksitename
export DISTRIBUTED_CLOUD_NAMESPACE=mydistributedcloudnamespace
export DISTRIBUTED_CLOUD_APP_STACK_NAMESPACE=portkeyai
export DISTRIBUTED_CLOUD_APP_STACK_SITE=myappstacksite
export DISTRIBUTED_CLOUD_SERVICE_NAME=portkeyai
# adjust the expiry date to a time no more than 90 days in the future
export KUBECONFIG_CERT_EXPIRE_DATE="2021-09-14T09:02:25.547659194Z"
export PORTKEY_GATEWAY_FQDN=the.host.nameof.theservice
export PORTKEY_PROVIDER=openai
export PORTKEY_PROVIDER_AUTH_TOKEN=authorizationtoken

curl --location --request POST 'https://$DISTRIBUTED_CLOUD_TENANT.console.ves.volterra.io/api/web/namespaces/system/sites/$DISTRIBUTED_CLOUD_SITE_NAME/global-kubeconfigs' \
--header 'Authorization: APIToken $DISTRIBUTED_CLOUD_API_TOKEN' \
--header 'Access-Control-Allow-Origin: *' \
--header 'x-volterra-apigw-tenant: $DISTRIBUTED_CLOUD_TENANT'\
--data-raw '{"expirationTimestamp":"$KUBECONFIG_CERT_EXPIRE_DATE"}'

Save the response in a YAML file for later use.
more detailed instructions for retrieving the App Stack kubeconfig file

  1. Copy the deployment YAML
wget https://raw.githubusercontent.com/Portkey-AI/gateway/main/deployment.yaml
  1. Apply the manifest
export KUBECONFIG=path/to/downloaded/global/kubeconfig/in/step/two
# apply the file downloaded in step 3
kubectl apply -f deployment.yaml
  1. Create Origin Pool
# create origin pool
curl --request POST \
  --url https://$DISTRIBUTED_CLOUD_TENANT.console.ves.volterra.io/api/config/namespaces/$DISTRIBUTED_CLOUD_NAMESPACE/origin_pools \
  --header 'authorization: APIToken $DISTRIBUTED_CLOUD_API_TOKEN' \
  --header 'content-type: application/json' \
  --data '{"metadata": {"name": "$DISTRIBUTED_CLOUD_SERVICE_NAME","namespace": "$DISTRIBUTED_CLOUD_NAMESPACE","labels": {},"annotations": {},"description": "","disable": false},"spec": {"origin_servers": [{"k8s_service": {"service_name": "$DISTRIBUTED_CLOUD_SERVICE_NAME.$DISTRIBUTED_CLOUD_APP_STACK_NAMESPACE","site_locator": {"site": {"tenant": "$DISTRIBUTED_CLOUD_TENANT_ID","namespace": "system","name": "$DISTRIBUTED_CLOUD_APP_STACK_SITE"}},"inside_network": {}},"labels": {}}],"no_tls": {},"port": 8787,"same_as_endpoint_port": {},"healthcheck": [],"loadbalancer_algorithm": "LB_OVERRIDE","endpoint_selection": "LOCAL_PREFERRED","advanced_options": null}}'

or use the UI

  1. Create an HTTP Load Balancer, including header injection of Portkey provider and credentials
curl --request POST \
  --url https://$DISTRIBUTED_CLOUD_TENANT.console.ves.volterra.io/api/config/namespaces/$DISTRIBUTED_CLOUD_NAMESPACE/http_loadbalancers \
  --header 'authorization: APIToken $DISTRIBUTED_CLOUD_API_TOKEN' \
  --header 'content-type: application/json' \
  --data '{"metadata": {"name": "$DISTRIBUTED_CLOUD_SERVICE_NAME","namespace": "$DISTRIBUTED_CLOUD_NAMESPACE","labels": {},"annotations": {},"description": "","disable": false},"spec": {"domains": ["$PORTKEY_GATEWAY_FQDN"],"https_auto_cert": {"http_redirect": true,"add_hsts": false,"tls_config": {"default_security": {}},"no_mtls": {},"default_header": {},"enable_path_normalize": {},"port": 443,"non_default_loadbalancer": {},"header_transformation_type": {"default_header_transformation": {}},"connection_idle_timeout": 120000,"http_protocol_options": {"http_protocol_enable_v1_v2": {}}},"advertise_on_public_default_vip": {},"default_route_pools": [{"pool": {"tenant": "$DISTRIBUTED_CLOUD_TENANT_ID","namespace": "$DISTRIBUTED_CLOUD_NAMESPACE","name": "$DISTRIBUTED_CLOUD_SERVICE_NAME"},"weight": 1,"priority": 1,"endpoint_subsets": {}}],"origin_server_subset_rule_list": null,"routes": [],"cors_policy": null,"disable_waf": {},"add_location": true,"no_challenge": {},"more_option": {"request_headers_to_add": [{"name": "x-portkey-provider","value": "$PORTKEY_PROVIDER","append": false},{"name": "Authorization","value": "Bearer $PORTKEY_PROVIDER_AUTH_TOKEN","append": false}],"request_headers_to_remove": [],"response_headers_to_add": [],"response_headers_to_remove": [],"max_request_header_size": 60,"buffer_policy": null,"compression_params": null,"custom_errors": {},"javascript_info": null,"jwt": [],"idle_timeout": 30000,"disable_default_error_pages": false,"cookies_to_modify": []},"user_id_client_ip": {},"disable_rate_limit": {},"malicious_user_mitigation": null,"waf_exclusion_rules": [],"data_guard_rules": [],"blocked_clients": [],"trusted_clients": [],"api_protection_rules": null,"ddos_mitigation_rules": [],"service_policies_from_namespace": {},"round_robin": {},"disable_trust_client_ip_headers": {},"disable_ddos_detection": {},"disable_malicious_user_detection": {},"disable_api_discovery": {},"disable_bot_defense": {},"disable_api_definition": {},"disable_ip_reputation": {},"disable_client_side_defense": {},"csrf_policy": null,"graphql_rules": [],"protected_cookies": [],"host_name": "","dns_info": [],"internet_vip_info": [],"system_default_timeouts": {},"jwt_validation": null,"disable_threat_intelligence": {},"l7_ddos_action_default": {},}}'

or use the UI

  1. Test the service
curl --request POST \
  --url https://$PORTKEY_GATEWAY_FQDN/v1/chat/completions \
  --header 'content-type: application/json' \
  --data '{"messages": [{"role": "user","content": "Say this might be a test."}],"max_tokens": 20,"model": "gpt-4"}'

in addition to the response headers, you should get a response body like

{
  "id": "chatcmpl-abcde......09876",
  "object": "chat.completion",
  "created": "0123456789",
  "model": "gpt-4-0321",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This might be a test."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 6,
    "total_tokens": 20
  },
  "system_fingerprint": null
}

Cloudflare Workers

  1. Clone the Repository
git clone https://github.com/portkey-ai/gateway
  1. Install the NPM Dependencies
cd gateway
npm install
  1. Deploy (using Wrangler CLI)
npm run deploy

For more details, refer to Cloudflare Workers official.

Docker

Run through the latest Docker Hub image:

docker run --rm  -p 8787:8787 portkeyai/gateway:latest

For more information on the Docker image, check here


Docker Compose

  1. Download Compose File from the Repository:
wget "https://raw.githubusercontent.com/Portkey-AI/gateway/main/docker-compose.yaml"
  1. Run:
docker compose up -d

The service is now running and listening on port 8787

For more details, refer to Docker Compose official.

AWS EC2

  1. Copy the AWS CloudFormation template from below:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: VPC where the EC2 instance will be launched
  SubnetId:
    Type: AWS::EC2::Subnet::Id
    Description: Subnet where the EC2 instance will be launched
  InstanceType:
    Type: String
    Default: t2.micro
    AllowedValues:
      - t2.micro
      - t2.small
      - t2.medium
      - t3.micro
      - t3.small
    Description: EC2 instance type

Resources:
  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !FindInMap [RegionMap, !Ref "AWS::Region", AMI]
      InstanceType: !Ref InstanceType
      SecurityGroupIds:
        - !Ref InstanceSecurityGroup
      SubnetId: !Ref SubnetId
      UserData:
        Fn::Base64: |
          #!/bin/bash
          sudo yum update -y
          sudo yum install -y amazon-linux-extras
          sudo amazon-linux-extras enable docker
          sudo yum install -y docker
          sudo systemctl start docker
          sudo systemctl enable docker
          sudo docker run -p 8787:8787 -d portkeyai/gateway:latest
      Tags:
        - Key: Name
          Value: PortkeyGateway

  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for Portkey Gateway
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 8787
          ToPort: 8787
          CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
        - IpProtocol: -1
          FromPort: -1
          ToPort: -1
          CidrIp: 0.0.0.0/0

Mappings:
  RegionMap:
    Metadata:
      Name: amzn2-ami-hvm-2.0.20250220.0-x86_64-gp2
      Owner: amazon
      CreationDate: 2025-02-20T22:38:11.000Z
    eu-west-1:
      AMI: ami-049b732d3f35a4f44
    ca-central-1:
      AMI: ami-06816da431adb7634
    eu-west-2:
      AMI: ami-0eebf19cec0b40d10
    us-east-2:
      AMI: ami-0e7b3e7766d24a6ff
    eu-west-3:
      AMI: ami-004f2229fb9afa698
    eu-north-1:
      AMI: ami-08fbe5a8c8061068f
    us-west-1:
      AMI: ami-01891d4f3898759b2
    ap-northeast-3:
      AMI: ami-0316e0efae0ce53d2
    us-east-1:
      AMI: ami-0ace34e9f53c91c5d
    ap-northeast-2:
      AMI: ami-0891aeb92f786d7a2
    sa-east-1:
      AMI: ami-081d377a25d396ece
    us-west-2:
      AMI: ami-04c0ab8f1251f1600
    ap-northeast-1:
      AMI: ami-00561c77487da40c1
    ap-south-1:
      AMI: ami-0f4f6fd19fad11737
    ap-southeast-2:
      AMI: ami-044b50caba366ec3a
    ap-southeast-1:
      AMI: ami-0301dd2fb476c9850
    eu-central-1:
      AMI: ami-014eb100f18a84d89


Outputs:
  PortkeyGatewayURL:
    Description: URL to access Portkey Gateway
    Value: !Sub http://${EC2Instance.PublicDnsName}:8787

  1. Create a new stack in the AWS CloudFormation console with the template above(you can upload in your S3 or directly upload the template).

  2. Fill the following parameters:

  • VpcId: The VPC ID of the VPC where the EC2 instance will be launched
  • SubnetId: The Subnet ID of the Subnet where the EC2 instance will be launched
  • InstanceType: The instance type of the EC2 instance

  1. Create the stack and wait for it to be created.

  2. Once the stack is created, you can access the Portkey Gateway URL from the Outputs section.

Replit

Deploy on Replit

Zeabur

Deploy on Zeabur

Supabase Functions

  1. Clone the Repository:
git clone https://github.com/portkey-ai/gateway
  1. Set up a Supabase Account and create a new project (in browser).
  2. Install the Supabase CLI, login using supabase login and Initialize, follow guides
  3. Write Your Supabase Function in:
supabase/functions/your-function-file/index.ts
  1. Deploy the Function (for windows add npx in start):
supabase functions deploy your-function-file
  1. Now, its ready to Test, Monitor and Manage Functions (Using url from supabase dashboard):
    • Similar to https://your-project-id.supabase.co/functions/v1/your-function-file

Refer to Supabase's official documentation for more details.


Fastly

  1. Create / login to your fastly account (in browser).

  2. Clone the Gateway Repository:

git clone https://github.com/portkey-ai/gateway
  1. Install Fastly’s CLI:
npm install fastly-cli
  1. Log in to Fastly:
fastly login
  1. Create a New Fastly Service:
fastly compute init

  1. Write Your Edge Function Code in src/main.rs or src/index.js

  2. Set up the token locally (token is located on Account > User(left bottom most) > API Token)

fastly profile create

Then follow the steps there to add token.

  1. Deploy to Fastly
fastly compute publish
  1. Now, its ready to Test, Monitor and Manage Functions (Using url from fastly dashboard):
    • Similar to your-new-project.edgecompute.app

For more details, refer to Fastly’s official.


Vercel

  • Docs to be written, please help!
  • Make sure you've deployed and tested the same.

AWS Lambda

  • Docs to be written, please help!
  • Make sure you've deployed and tested the same.

Lambda edge

  • Docs to be written, please help!
  • Make sure you've deployed and tested the same.

Enterprise Deployment

Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.

✅  Secure Key Management - for role-based access control and tracking
✅  Simple & Semantic Caching - to serve repeat queries faster & save costs
✅  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
✅  PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
✅  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
✅  Professional Support - along with feature prioritization

Schedule a call to discuss enterprise deployments