Skip to content

Commit 8c7ed49

Browse files
siddharthsambharia-portkeysiddharthsambharia-portkey
committed
v1
1 parent 9a3940e commit 8c7ed49

File tree

2 files changed

+257
-0
lines changed

2 files changed

+257
-0
lines changed

‎docs.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,7 @@
294294
"integrations/llms/jina-ai",
295295
"integrations/llms/lambda",
296296
"integrations/llms/lemon-fox",
297+
"integrations/llms/lepton",
297298
"integrations/llms/lingyi-01.ai",
298299
"integrations/llms/mistral-ai",
299300
"integrations/llms/monster-api",

‎integrations/llms/lepton.mdx

Lines changed: 256 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,256 @@
1+
---
2+
title: "Lepton AI"
3+
---
4+
5+
Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including [Lepton AI APIs](https://www.lepton.ai/docs/guides).
6+
7+
With Portkey, you can take advantage of features like fast AI gateway access, observability, prompt management, and more, all while ensuring the secure management of your LLM API keys through a [virtual key](/product/ai-gateway/virtual-keys) system.
8+
<Note>
9+
Provider Slug. `lepton`
10+
</Note>
11+
12+
## Portkey SDK Integration with Lepton AI Models
13+
14+
Portkey provides a consistent API to interact with models from various providers. To integrate Lepton AI with Portkey:
15+
16+
### 1\. Install the Portkey SDK
17+
18+
Add the Portkey SDK to your application to interact with Lepton AI's API through Portkey's gateway.
19+
20+
<Tabs>
21+
<Tab title="NodeJS">
22+
```sh
23+
npm install --save portkey-ai
24+
```
25+
</Tab>
26+
<Tab title="Python">
27+
```sh
28+
pip install portkey-ai
29+
```
30+
</Tab>
31+
</Tabs>
32+
33+
### 2\. Initialize Portkey with the Virtual Key
34+
35+
To use Lepton AI with Portkey, [get your API key from Lepton AI](https://console.lepton.ai/), then add it to Portkey to create the virtual key.
36+
37+
<Tabs>
38+
<Tab title="NodeJS SDK">
39+
```js
40+
import Portkey from 'portkey-ai'
41+
42+
const portkey = new Portkey({
43+
apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
44+
virtualKey: "VIRTUAL_KEY" // Your Lepton AI Virtual Key
45+
})
46+
```
47+
</Tab>
48+
<Tab title="Python SDK">
49+
```python
50+
from portkey_ai import Portkey
51+
52+
portkey = Portkey(
53+
api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
54+
virtual_key="VIRTUAL_KEY" # Replace with your virtual key for Lepton
55+
)
56+
```
57+
</Tab>
58+
</Tabs>
59+
60+
### 3\. Invoke Chat Completions with Lepton AI
61+
62+
Use the Portkey instance to send requests to Lepton AI. You can also override the virtual key directly in the API call if needed.
63+
64+
<Tabs>
65+
<Tab title="NodeJS SDK">
66+
```js
67+
const chatCompletion = await portkey.chat.completions.create({
68+
messages: [{ role: 'user', content: 'Say this is a test' }],
69+
model: 'llama-3-8b-sft-v1',
70+
});
71+
72+
console.log(chatCompletion.choices);
73+
```
74+
</Tab>
75+
<Tab title="Python SDK">
76+
```python
77+
completion = portkey.chat.completions.create(
78+
messages= [{ "role": 'user', "content": 'Say this is a test' }],
79+
model= 'llama-3-8b-sft-v1'
80+
)
81+
82+
print(completion)
83+
```
84+
</Tab>
85+
<Tab title="cURL">
86+
```sh
87+
curl --location 'https://api.portkey.ai/v1/chat/completions' \
88+
-H 'Content-Type: application/json' \
89+
-H 'x-portkey-api-key: PORTKEY_API_KEY' \
90+
-H 'x-portkey-virtual-key: VIRTUAL_KEY' \
91+
--data '{
92+
"model": "llama-3-8b-sft-v1",
93+
"messages": [
94+
{
95+
"role": "user",
96+
"content": "Say this is a test"
97+
}
98+
]
99+
}'
100+
```
101+
</Tab>
102+
</Tabs>
103+
104+
## Text Completions
105+
106+
For applications that require the traditional completions API, you can use Lepton AI's completions endpoint.
107+
108+
<Tabs>
109+
<Tab title="NodeJS SDK">
110+
```js
111+
const completion = await portkey.completions.create({
112+
prompt: 'Write a poem about AI',
113+
model: 'llama-3-8b-sft-v1',
114+
max_tokens: 250
115+
});
116+
117+
console.log(completion.choices);
118+
```
119+
</Tab>
120+
<Tab title="Python SDK">
121+
```python
122+
completion = portkey.completions.create(
123+
prompt="Write a poem about AI",
124+
model="llama-3-8b-sft-v1",
125+
max_tokens=250
126+
)
127+
128+
print(completion)
129+
```
130+
</Tab>
131+
</Tabs>
132+
133+
## Speech-to-Text (Transcription)
134+
135+
Lepton AI provides speech-to-text capabilities through Portkey's unified API:
136+
137+
<Tabs>
138+
<Tab title="NodeJS SDK">
139+
```js
140+
import fs from 'fs';
141+
142+
const transcription = await portkey.audio.transcriptions.create({
143+
file: fs.createReadStream('audio.mp3'),
144+
model: 'whisper-large-v3',
145+
});
146+
147+
console.log(transcription.text);
148+
```
149+
</Tab>
150+
<Tab title="Python SDK">
151+
```python
152+
with open("audio.mp3", "rb") as audio_file:
153+
transcription = portkey.audio.transcriptions.create(
154+
file=audio_file,
155+
model="whisper-large-v3"
156+
)
157+
158+
print(transcription.text)
159+
```
160+
</Tab>
161+
</Tabs>
162+
163+
## Advanced Features
164+
165+
### Streaming Responses
166+
167+
Lepton AI supports streaming responses to provide real-time generation:
168+
169+
<Tabs>
170+
<Tab title="NodeJS SDK">
171+
```js
172+
const stream = await portkey.chat.completions.create({
173+
messages: [{ role: 'user', content: 'Write a story about a robot' }],
174+
model: 'llama-3-8b-sft-v1',
175+
stream: true,
176+
});
177+
178+
for await (const chunk of stream) {
179+
process.stdout.write(chunk.choices[0]?.delta?.content || '');
180+
}
181+
```
182+
</Tab>
183+
<Tab title="Python SDK">
184+
```python
185+
stream = portkey.chat.completions.create(
186+
messages=[{"role": "user", "content": "Write a story about a robot"}],
187+
model="llama-3-8b-sft-v1",
188+
stream=True
189+
)
190+
191+
for chunk in stream:
192+
if chunk.choices[0].delta.content:
193+
print(chunk.choices[0].delta.content, end="")
194+
```
195+
</Tab>
196+
</Tabs>
197+
198+
### Lepton-Specific Parameters
199+
200+
Lepton AI models support several unique parameters for advanced control:
201+
202+
- `length_penalty`: Controls the length of generated text (default: 1)
203+
- `repetition_penalty`: Reduces repetitive patterns in generation (default: 1)
204+
- `dry_multiplier`: Controls monotonicity of text (default: 0)
205+
- `top_k`: Number of highest probability tokens to consider (default: 50)
206+
- `min_p`: Filters out tokens below a probability threshold (default: 0)
207+
208+
Example with custom parameters:
209+
210+
```python
211+
completion = portkey.chat.completions.create(
212+
messages=[{"role": "user", "content": "Write a creative story"}],
213+
model="llama-3-8b-sft-v1",
214+
temperature=0.8,
215+
top_p=0.95,
216+
repetition_penalty=1.2,
217+
length_penalty=1.1,
218+
top_k=40,
219+
min_p=0.05
220+
)
221+
```
222+
223+
### Audio Support
224+
225+
Lepton AI offers specialized audio features:
226+
227+
```python
228+
completion = portkey.chat.completions.create(
229+
messages=[{"role": "user", "content": "Generate a spoken response"}],
230+
model="llama-3-8b-sft-v1",
231+
require_audio=True,
232+
tts_preset_id="jessica", # Voice ID
233+
tts_audio_format="mp3", # Output format
234+
tts_audio_bitrate=64 # Quality setting
235+
)
236+
```
237+
238+
## Managing Lepton AI Prompts
239+
240+
You can manage all prompts to Lepton AI in the [Prompt Library](/product/prompt-library). All the current models of Lepton AI are supported and you can easily start testing different prompts.
241+
242+
Once you're ready with your prompt, you can use the `portkey.prompts.completions.create` interface to use the prompt in your application.
243+
244+
## Next Steps
245+
246+
The complete list of features supported in the SDK are available on the link below.
247+
248+
<Card title="SDK" href="/api-reference/portkey-sdk-client">
249+
</Card>
250+
251+
You'll find more information in the relevant sections:
252+
253+
1. [Add metadata to your requests](/product/observability/metadata)
254+
2. [Add gateway configs to your Lepton AI requests](/product/ai-gateway/configs)
255+
3. [Tracing Lepton AI requests](/product/observability/traces)
256+
4. [Setup a fallback from OpenAI to Lepton AI APIs](/product/ai-gateway/fallbacks)

0 commit comments

Comments
 (0)