Skip to content

Commit fa8e2d4

Browse files
cybermaggedoncybermaggedon
authored
Update Minikube docs (#14)
1 parent 0d11852 commit fa8e2d4

File tree

6 files changed

+1314
-403
lines changed

6 files changed

+1314
-403
lines changed

‎_includes/llm-configuration-details-compose.md‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
<details>
2+
<summary>Specific guidance for Azure</summary>
3+
<div markdown="1">
4+
There are 2 hosted model options for Azure:
5+
6+
- Machine Learning Services (MLS)
7+
- Cognitive Services (CS)
8+
9+
TrustGraph's *Azure* is for integration with MLS. *Azure OpenAI* is for
10+
integration with CS. If you are using the *Azure* / MLS integration, you
11+
should make sure you know your model endpoint, and the token granted for the
12+
endpoint, and configure these values thus:
13+
```
14+
export AZURE_ENDPOINT=https://ENDPOINT.API.HOST.GOES.HERE/
15+
export AZURE_TOKEN=TOKEN-GOES-HERE
16+
```
17+
If you are using the *Azure OpenAI* / CS integration, you should make sure
18+
you know your model endpoint, the token and configure them thus:
19+
```
20+
export AZURE_ENDPOINT=https://ENDPOINT.API.HOST.GOES.HERE/
21+
export AZURE_TOKEN=TOKEN-GOES-HERE
22+
```
23+
</div>
24+
</details>
25+
26+
<details>
27+
<summary>Specific guidance for AWS Bedrock</summary>
28+
<div markdown="1">
29+
To use Bedrock, you need to have AWS credentials provisioned.
30+
The easiest way is to create an IAM user, and create credentials for this
31+
user. When you provision the user, you will be asked to give the user
32+
permissions. To allow Bedrock access, the `AmazonBedrockFullAccess`
33+
role should be added.
34+
35+
You would then provision credentials which would give you an *access key ID*
36+
and a *secret access key*. You should pick the identifier of an
37+
AWS region to connect to e.g. `eu-west-2`. In order to prepare to deploy,
38+
you should set three environment variables using the information.
39+
40+
```
41+
export AWS_ACCESS_KEY_ID=ID-KEY-HERE
42+
export AWS_SECRET_ACCESS_KEY=TOKEN-GOES-HERE
43+
export AWS_DEFAULT_REGION=AWS-REGION-HERE
44+
```
45+
46+
Note: You should be very careful with AWS cloud credentials provisioned
47+
this way: if lost or leaked this provides a malicious person access to the
48+
AWS resources you gave this user.
49+
</div>
50+
</details>
51+
52+
<details>
53+
<summary>Specific guidance for Anthropic Claude</summary>
54+
<div markdown="1">
55+
To use Anthropic's Claude models directly, sign up for API access at
56+
[console.anthropic.com](https://console.anthropic.com/). Create an API key
57+
from the dashboard. Set the key as an environment variable:
58+
59+
```
60+
export CLAUDE_KEY=sk-ant-api03-xxxxx
61+
```
62+
</div>
63+
</details>
64+
65+
<details>
66+
<summary>Specific guidance for Cohere</summary>
67+
<div markdown="1">
68+
To use Cohere's models, sign up at [cohere.com](https://cohere.com/) and
69+
create an API key from your dashboard. Set the key as an environment variable:
70+
71+
```
72+
export COHERE_KEY=your-cohere-api-key-here
73+
```
74+
</div>
75+
</details>
76+
77+
<details>
78+
<summary>Specific guidance for Google AI Studio</summary>
79+
<div markdown="1">
80+
To use Google's Gemini models via AI Studio, visit
81+
[aistudio.google.com](https://aistudio.google.com/) and generate an API key.
82+
Set the key as an environment variable:
83+
84+
```
85+
export GOOGLE_AI_STUDIO_KEY=your-api-key-here
86+
```
87+
</div>
88+
</details>
89+
90+
<details>
91+
<summary>Specific guidance for Llamafile / llama.cpp server</summary>
92+
<div markdown="1">
93+
If running a llamafile or llama.cpp server locally, configure the URL to point
94+
to your server. The URL must include the `/v1` path:
95+
96+
```
97+
export LLAMAFILE_URL=http://your-server-host:port/v1
98+
```
99+
100+
If running on the same host as your containers, use `host.containers.internal`
101+
as the hostname (e.g., `http://host.containers.internal:7000/v1`).
102+
103+
See also: [Container networking and self-hosted models](container-networking)
104+
</div>
105+
</details>
106+
107+
<details>
108+
<summary>Specific guidance for LMStudio</summary>
109+
<div markdown="1">
110+
If running LMStudio locally, configure the URL to point to your LMStudio server.
111+
LMStudio typically runs on port 1234:
112+
113+
```
114+
export LMSTUDIO_URL=http://your-server-host:1234
115+
```
116+
117+
If running on the same host as your containers, use `host.containers.internal`
118+
as the hostname (e.g., `http://host.containers.internal:1234`).
119+
120+
See also: [Container networking and self-hosted models](container-networking)
121+
</div>
122+
</details>
123+
124+
<details>
125+
<summary>Specific guidance for Mistral AI</summary>
126+
<div markdown="1">
127+
To use Mistral's API, sign up at [console.mistral.ai](https://console.mistral.ai/)
128+
and create an API key. Set the key as an environment variable:
129+
130+
```
131+
export MISTRAL_TOKEN=your-mistral-api-key-here
132+
```
133+
</div>
134+
</details>
135+
136+
<details>
137+
<summary>Specific guidance for Ollama</summary>
138+
<div markdown="1">
139+
If running Ollama locally, configure the URL to point to your Ollama server.
140+
Ollama typically runs on port 11434:
141+
142+
```
143+
export OLLAMA_HOST=http://your-server-host:11434
144+
```
145+
146+
If running on the same host as your containers, use `host.containers.internal`
147+
as the hostname (e.g., `http://host.containers.internal:11434`).
148+
149+
See also: [Container networking and self-hosted models](container-networking)
150+
</div>
151+
</details>
152+
153+
<details>
154+
<summary>Specific guidance for OpenAI</summary>
155+
<div markdown="1">
156+
To use OpenAI's API, sign up at [platform.openai.com](https://platform.openai.com/)
157+
and create an API key. Set the key as an environment variable:
158+
159+
```
160+
export OPENAI_TOKEN=your-openai-api-key-here
161+
```
162+
163+
Many other services provide OpenAI-compatible APIs. You can use these by setting
164+
the `OPENAI_BASE_URL` environment variable to point to the alternative service:
165+
166+
```
167+
export OPENAI_BASE_URL=http://your-server-host:8000/v1
168+
```
169+
</div>
170+
</details>
171+
172+
<details>
173+
<summary>Specific guidance for Google Cloud VertexAI</summary>
174+
<div markdown="1">
175+
To use Google Cloud VertexAI, you need to create a service account with
176+
appropriate permissions and download its credentials file.
177+
178+
1. In Google Cloud Console, create a service account
179+
2. Grant the service account permissions to invoke VertexAI models (e.g.,
180+
`Vertex AI User` role - use minimal permissions, not admin roles)
181+
3. Create and download a JSON key file for the service account
182+
4. Save the key file as `vertexai/private.json` in your deployment directory
183+
184+
{: .warning }
185+
**Important**: Service account credentials provide access to your Google Cloud
186+
resources. Never commit `private.json` to version control. Use minimal
187+
permissions - grant only what's needed for VertexAI model invocation, not
188+
administrator roles.
189+
190+
After placing the file, you may need to adjust file permissions as described
191+
earlier in the configuration unpacking section:
192+
193+
```sh
194+
chmod 644 vertexai/private.json
195+
```
196+
197+
On SELinux systems, also run:
198+
199+
```sh
200+
sudo chcon -Rt svirt_sandbox_file_t vertexai/
201+
```
202+
</div>
203+
</details>
204+
205+
<details>
206+
<summary>Specific guidance for vLLM</summary>
207+
<div markdown="1">
208+
If running vLLM locally, configure the URL to point to your vLLM server.
209+
The URL should include the `/v1` path:
210+
211+
```
212+
export VLLM_URL=http://your-server-host:port/v1
213+
```
214+
215+
If running on the same host as your containers, use `host.containers.internal`
216+
as the hostname (e.g., `http://host.containers.internal:8000/v1`).
217+
218+
See also: [Container networking and self-hosted models](container-networking)
219+
</div>
220+
</details>

0 commit comments

Comments
 (0)