You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+25-14Lines changed: 25 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,26 @@
1
1
# ResilientLLM
2
2
3
-
A simple but robust LLM integration layer designed to ensure reliable, seamless interactions across multiple APIs by intelligently handling failures and rate limits.
3
+
A minimalist but robust LLM integration layer designed to ensure reliable, seamless interactions across multiple LLM providers by intelligently handling failures and rate limits.
4
+
5
+
## Why Use ResilentLLM
6
+
7
+
This library solves challenges in building production-ready AI Agents due to
8
+
9
+
- Unstable network conditions
10
+
- Inconsistent error handling
11
+
- Unpredictable LLM API rate limit errors
12
+
13
+
### Key Features
14
+
15
+
-**Rate limiting**: You don’t need to calculate tokens or manage rate limits yourself
16
+
-**Token estimation**: The number of LLM tokens is estimated for each request and enforced.
17
+
-**Retries, backoff, and circuit breaker**: All are handled internally by the `ResilientOperation`.
18
+
19
+
## Installation
20
+
21
+
```bash
22
+
npm i resilient-llm
23
+
```
4
24
5
25
## Quickstart
6
26
@@ -16,7 +36,7 @@ const llm = new ResilientLLM({
16
36
requestsPerMinute:60, // Limit to 60 requests per minute
17
37
llmTokensPerMinute:90000// Limit to 90,000 LLM tokens per minute
18
38
},
19
-
retries:3, // Number of times to retry if req. fails for reasons possible to fix by retry
39
+
retries:3, // Number of times to retry when req. fails and only if it is possible to fix by retry
20
40
backoffFactor:2// Increase delay between retries by this factor
21
41
});
22
42
@@ -35,17 +55,7 @@ const conversationHistory = [
35
55
})();
36
56
```
37
57
38
-
---
39
-
40
-
### Key Points
41
-
42
-
-**Rate limiting is automatic**: You don’t need to pass token counts or manage rate limits yourself.
43
-
-**Token estimation**: The number of LLM tokens is estimated for each request and enforced.
44
-
-**Retries, backoff, and circuit breaker**: All are handled internally by the `ResilientOperation`.
ResilientLLM is a resilient, unified LLM interface featuring circuit breaker, token bucket rate limiting, caching, and adaptive retry with dynamic backoff support.
@@ -76,7 +87,7 @@ The final solution was to extract tiny LLM orchestration class out of all my AI
76
87
This library solves my challenges in building production-ready AI Agents such as:
77
88
- unstable network conditions
78
89
- inconsistent error handling
79
-
- unpredictable LLM API rate limit errrors
90
+
- unpredictable LLM API rate limit errors
80
91
81
92
This library aims to solve the same challenges for you by providing a resilient layer that intelligently manages failures and rate limits, enabling you (developers) to integrate LLMs confidently and effortlessly at scale.
0 commit comments