How to use Vertex AI Llama 405B API (whilst it is still free)

[michaelbarrett-hicksons]

Firstly, become a massive fan of LibreChat - to the point I've connected our Azure GPT4o, Google Gemini 1.5 Flash & Pro plus numerous Ollama offline open source models (Phi, Llama3, Gemma2) seamlessly.

However, I'm stuck with how I can connect (or maybe I can't?) the Llama 3.1 405b API (currently free) from Google / Vertex AI.

Note: I have a valid, billable Google Cloud / Vertex AI account already.
I have an existing PROJECT_ID.

The instructions, which work with Postman etc are as follows:

Try Llama 3.1 API Service
To use Llama 3.1 API service with the command line interface (CLI), do the following:

1. Open Cloud Shell or a local terminal window with the gcloud CLI installed.
2. Configure environment variables by entering the following. Replace YOUR_PROJECT_ID with the ID of your Google Cloud project.

            ENDPOINT=[us-central1-aiplatform.googleapis.com](http://us-central1-aiplatform.googleapis.com/)
            REGION=us-central1
            PROJECT_ID="YOUR_PROJECT_ID"

3. Send a prompt request by entering the following curl command:

curl \
  -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" https://${ENDPOINT}/v1beta1/projects/${PROJECT_ID}/locations/${REGION}/endpoints/openapi/chat/completions \
  -d '{"model":"meta/llama3-405b-instruct-maas", "stream":true, "messages":[{"role": "user", "content": "Summer travel plan to Paris"}]}'

Is there anyway to integrate this into Libre? (Even if it does mean I have to get a new bearer token every 60 mins)

Many thanks for any advice or guidance.

[abhranil26]

i was wondering the same