Google Vertex AI litellm proxy yaml config

[michaelbarrett-hicksons]

Can anyone assist in why I can't get litellm proxy, Ollama and Google Vertex AI working?
I've retrieved the 1 hr gcloud-auth-token and can connect via Postman.
I also found this documentation these 2 pages of documentation, but can't get it too connect:

1. https://docs.litellm.ai/docs/providers/vertex#usage-with-litellm-proxy-server.
2. https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-gemini-using-openai-library#environment-variables

Any guidance would be appreciated.

model_list:
  - model_name: gemini-1.5
    litellm_params:
      model: vertex_ai/gemini-1.5-pro
      vertex_project: "my-project-id-here" 
      vertex_location: "my-location-here" 
      
litellm_settings:
  drop_params: True
  set_verbose: True
  api_base: https://my-location-here-aiplatform.googleapis.com/v1beta1/projects/my-project-id-here/locations/my-location-here/endpoints/openapi/chat/completions
  api_key: Bearer gcloud-auth-login-token

[ishaan-jaff]

hi @michaelbarrett-hicksons

• you need to provide vertex_credentials like this where vertex_credentials is a path to your service account json.

model_list:
  - model_name: gemini-vision
    litellm_params:
      model: vertex_ai/gemini-1.5-pro
      api_base: https://exampleopenaiendpoint-production.up.railway.app/v1/projects/adroit-crow-413218/locations/us-central1/publishers/google/models/gemini-1.0-pro-vision-001
      vertex_project: "adroit-crow-413218"
      vertex_location: "us-central1"
      vertex_credentials: "/Users/ishaanjaffer/Downloads/adroit-crow-413218-a956eef1a2a8.json"

If this does not solve your problem can we hop on a call ? I want to help unblock you
my cal for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
my linkedin if you prefer DMs: https://www.linkedin.com/in/reffajnaahsi/

[priamai]

Hi there,
I am struggling to make it work with a deployment such as like this:

  - model_name: phi3-mini-4k
    litellm_params:
      model: vertex_ai/phi3-mini-4k-instruct
      api_base: https://us-central1-aiplatform.googleapis.com/v1/projects/306767508xxx/locations/us-central1/endpoints/7590132176188866yyy:predict
      vertex_project: 306767508xxx
      vertex_location: us-central1

Deployed via model garden:
https://console.cloud.google.com/vertex-ai/publishers/microsoft/model-garden/phi3