Skip to content

Commit c56cc43

Browse files
mudlermudler
authored
feat(memory): add knowledgebase and memory management from LocalRecall (#424)
* feat: integrate knowledge base management Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactorings Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
1 parent bc567ef commit c56cc43

21 files changed

+1926
-127
lines changed

‎README.md‎

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ LocalAGI ensures your data stays exactly where you want it—on your hardware. N
3333
- 🤖 **Advanced Agent Teaming**: Instantly create cooperative agent teams from a single prompt.
3434
- 📡 **Connectors**: Built-in integrations with Discord, Slack, Telegram, GitHub Issues, and IRC.
3535
- 🛠 **Comprehensive REST API**: Seamless integration into your workflows. Every agent created will support OpenAI Responses API out of the box.
36-
- 📚 **Short & Long-Term Memory**: Powered by [LocalRecall](https://github.com/mudler/LocalRecall).
36+
- 📚 **Short & Long-Term Memory**: Built-in knowledge base (RAG) for collections, file uploads, and semantic search. Manage collections in the Web UI under **Knowledge base**; agents with "Knowledge base" enabled use it automatically (implementation uses [LocalRecall](https://github.com/mudler/LocalRecall) libraries).
3737
- 🧠 **Planning & Reasoning**: Agents intelligently plan, reason, and adapt.
3838
- 🔄 **Periodic Tasks**: Schedule tasks with cron-like syntax.
3939
- 💾 **Memory Management**: Control memory usage with options for long-term and summary memory.
@@ -108,7 +108,7 @@ Still having issues? see this Youtube video: https://youtu.be/HtVwIxW3ePg
108108
</td>
109109
<td width="50%" valign="top">
110110
<h3><a href="https://github.com/mudler/LocalRecall">LocalRecall</a></h3>
111-
<p>A REST-ful API and knowledge base management system that provides persistent memory and storage capabilities for AI agents.</p>
111+
<p>A REST-ful API and knowledge base management system. LocalAGI embeds this functionality: the Web UI includes a <strong>Knowledge base</strong> section and the same collections API, so you no longer need to run LocalRecall separately.</p>
112112
</td>
113113
</tr>
114114
</table>
@@ -239,11 +239,13 @@ LocalAGI supports environment configurations. Note that these environment variab
239239
| `LOCALAGI_LLM_API_KEY` | API authentication |
240240
| `LOCALAGI_TIMEOUT` | Request timeout settings |
241241
| `LOCALAGI_STATE_DIR` | Where state gets stored |
242-
| `LOCALAGI_LOCALRAG_URL` | LocalRecall connection |
242+
| `LOCALAGI_BASE_URL` | Optional base URL for the app (only relevant when using an external LocalRAG URL; not used for built-in knowledge base) |
243243
| `LOCALAGI_ENABLE_CONVERSATIONS_LOGGING` | Toggle conversation logs |
244244
| `LOCALAGI_API_KEYS` | A comma separated list of api keys used for authentication |
245245
| `LOCALAGI_CUSTOM_ACTIONS_DIR` | Directory containing custom Go action files to be automatically loaded |
246246

247+
For the built-in knowledge base, optional env (defaults use `LOCALAGI_STATE_DIR`): `COLLECTION_DB_PATH`, `FILE_ASSETS`, `VECTOR_ENGINE` (e.g. `chromem`, `postgres`), `EMBEDDING_MODEL`, `DATABASE_URL` (when `VECTOR_ENGINE=postgres`).
248+
247249
Skills are stored in a fixed `skills` subdirectory under `LOCALAGI_STATE_DIR` (e.g. `/pool/skills` in Docker). Git repo config for skills lives in that directory. No extra environment variables are required.
248250

249251
## Installation Options
@@ -339,15 +341,16 @@ import (
339341
"github.com/mudler/LocalAGI/core/types"
340342
)
341343

342-
// Create a new agent pool
344+
// Create a new agent pool (call pool.SetRAGProvider(...) for knowledge base; see main.go)
343345
pool, err := state.NewAgentPool(
344346
"default-model", // default model name
345347
"default-multimodal-model", // default multimodal model
346-
"image-model", // image generation model
348+
"transcription-model", // default transcription model
349+
"en", // default transcription language
350+
"tts-model", // default TTS model
347351
"http://localhost:8080", // API URL
348-
"your-api-key", // API key
349-
"./state", // state directory
350-
"http://localhost:8081", // LocalRAG API URL
352+
"your-api-key", // API key
353+
"./state", // state directory
351354
func(config *AgentConfig) func(ctx context.Context, pool *AgentPool) []types.Action {
352355
// Define available actions for agents
353356
return func(ctx context.Context, pool *AgentPool) []types.Action {
@@ -374,8 +377,9 @@ pool, err := state.NewAgentPool(
374377
// Add your custom filters here
375378
}
376379
},
377-
"10m", // timeout
378-
true, // enable conversation logs
380+
"10m", // timeout
381+
true, // enable conversation logs
382+
nil, // skills service (optional)
379383
)
380384

381385
// Create a new agent in the pool
@@ -741,7 +745,7 @@ export LOCALAGI_MODEL=gemma-3-4b-it-qat
741745
export LOCALAGI_MULTIMODAL_MODEL=moondream2-20250414
742746
export LOCALAGI_IMAGE_MODEL=sd-1.5-ggml
743747
export LOCALAGI_LLM_API_URL=http://localai:8080
744-
export LOCALAGI_LOCALRAG_URL=http://localrecall:8080
748+
# Knowledge base is built-in; no separate LocalRecall service needed
745749
export LOCALAGI_STATE_DIR=./pool
746750
export LOCALAGI_TIMEOUT=5m
747751
export LOCALAGI_ENABLE_CONVERSATIONS_LOGGING=false
@@ -1045,7 +1049,7 @@ LocalAGI supports environment configurations. Note that these environment variab
10451049
| `LOCALAGI_LLM_API_KEY` | API authentication |
10461050
| `LOCALAGI_TIMEOUT` | Request timeout settings |
10471051
| `LOCALAGI_STATE_DIR` | Where state gets stored |
1048-
| `LOCALAGI_LOCALRAG_URL` | LocalRecall connection |
1052+
| `LOCALAGI_BASE_URL` | Optional base URL for built-in knowledge base (default `http://localhost:3000`) |
10491053
| `LOCALAGI_SSHBOX_URL` | LocalAGI SSHBox URL, e.g. user:pass@ip:port |
10501054
| `LOCALAGI_ENABLE_CONVERSATIONS_LOGGING` | Toggle conversation logs |
10511055
| `LOCALAGI_API_KEYS` | A comma separated list of api keys used for authentication |

‎core/state/compaction.go‎

Lines changed: 32 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,33 @@ import (
1515
"github.com/sashabaranov/go-openai"
1616
)
1717

18+
// KBCompactionClient is the interface used by compaction. It can be implemented by the HTTP RAG client adapter or by the in-process collection adapter.
19+
type KBCompactionClient interface {
20+
Collection() string
21+
ListEntries() ([]string, error)
22+
GetEntryContent(entry string) (content string, chunkCount int, err error)
23+
Store(filePath string) error
24+
DeleteEntry(entry string) error
25+
}
26+
27+
// wrappedClientCompactionAdapter adapts *localrag.WrappedClient to KBCompactionClient.
28+
type wrappedClientCompactionAdapter struct {
29+
*localrag.WrappedClient
30+
}
31+
32+
func (a *wrappedClientCompactionAdapter) ListEntries() ([]string, error) {
33+
return a.Client.ListEntries(a.Collection())
34+
}
35+
36+
func (a *wrappedClientCompactionAdapter) Store(filePath string) error {
37+
return a.Client.Store(a.Collection(), filePath)
38+
}
39+
40+
func (a *wrappedClientCompactionAdapter) DeleteEntry(entry string) error {
41+
_, err := a.Client.DeleteEntry(a.Collection(), entry)
42+
return err
43+
}
44+
1845
// datePrefixRegex matches YYYY-MM-DD at the start of a filename (e.g. 2006-01-02-15-04-05-hash.txt).
1946
var datePrefixRegex = regexp.MustCompile(`^(\d{4}-\d{2}-\d{2})`)
2047

@@ -102,9 +129,9 @@ func (s *openAISummarizer) Summarize(ctx context.Context, content string) (strin
102129
}
103130

104131
// RunCompaction runs one compaction pass: list entries, group by period, for each group fetch content, optionally summarize, store result, delete originals.
105-
func RunCompaction(ctx context.Context, client *localrag.WrappedClient, period string, summarize bool, apiURL, apiKey, model string) error {
132+
func RunCompaction(ctx context.Context, client KBCompactionClient, period string, summarize bool, apiURL, apiKey, model string) error {
106133
collection := client.Collection()
107-
entries, err := client.Client.ListEntries(collection)
134+
entries, err := client.ListEntries()
108135
if err != nil {
109136
return fmt.Errorf("list entries: %w", err)
110137
}
@@ -164,15 +191,15 @@ func RunCompaction(ctx context.Context, client *localrag.WrappedClient, period s
164191
xlog.Warn("compaction: write temp file failed", "error", err)
165192
continue
166193
}
167-
if err := client.Client.Store(collection, tmpPath); err != nil {
194+
if err := client.Store(tmpPath); err != nil {
168195
os.RemoveAll(tmpDir)
169196
xlog.Warn("compaction: store failed", "key", key, "error", err)
170197
continue
171198
}
172199
os.RemoveAll(tmpDir)
173200

174201
for _, entry := range groupEntries {
175-
if _, err := client.Client.DeleteEntry(collection, entry); err != nil {
202+
if err := client.DeleteEntry(entry); err != nil {
176203
xlog.Warn("compaction: delete entry failed", "entry", entry, "error", err)
177204
}
178205
}
@@ -182,7 +209,7 @@ func RunCompaction(ctx context.Context, client *localrag.WrappedClient, period s
182209
}
183210

184211
// runCompactionTicker runs compaction on a schedule (daily/weekly/monthly). It stops when ctx is done.
185-
func runCompactionTicker(ctx context.Context, client *localrag.WrappedClient, config *AgentConfig, apiURL, apiKey, model string) {
212+
func runCompactionTicker(ctx context.Context, client KBCompactionClient, config *AgentConfig, apiURL, apiKey, model string) {
186213
// Run first compaction immediately on startup
187214
if err := RunCompaction(ctx, client, config.KBCompactionInterval, config.KBCompactionSummarize, apiURL, apiKey, model); err != nil {
188215
xlog.Warn("compaction ticker initial run failed", "collection", client.Collection(), "error", err)

‎core/state/pool.go‎

Lines changed: 46 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,26 @@ type SkillsProvider interface {
2727
GetMCPSession(ctx context.Context) (*mcp.ClientSession, error)
2828
}
2929

30+
// RAGProvider returns a RAGDB and optional compaction client for a collection (e.g. agent name).
31+
// effectiveRAGURL/Key are pool/agent defaults; implementation may use them (HTTP) or ignore them (embedded).
32+
type RAGProvider func(collectionName, effectiveRAGURL, effectiveRAGKey string) (RAGDB, KBCompactionClient, bool)
33+
34+
// NewHTTPRAGProvider returns a RAGProvider that uses the LocalRAG HTTP API. When effective URL/key are empty, baseURL/baseKey are used.
35+
func NewHTTPRAGProvider(baseURL, baseKey string) RAGProvider {
36+
return func(collectionName, effectiveURL, effectiveKey string) (RAGDB, KBCompactionClient, bool) {
37+
url := effectiveURL
38+
if url == "" {
39+
url = baseURL
40+
}
41+
key := effectiveKey
42+
if key == "" {
43+
key = baseKey
44+
}
45+
wc := localrag.NewWrappedClient(url, key, collectionName)
46+
return wc, &wrappedClientCompactionAdapter{WrappedClient: wc}, true
47+
}
48+
}
49+
3050
type AgentPool struct {
3151
sync.Mutex
3252
file string
@@ -37,7 +57,8 @@ type AgentPool struct {
3757
agentStatus map[string]*Status
3858
apiURL, defaultModel, defaultMultimodalModel, defaultTTSModel string
3959
defaultTranscriptionModel, defaultTranscriptionLanguage string
40-
localRAGAPI, localRAGKey, apiKey string
60+
apiKey string
61+
ragProvider RAGProvider
4162
availableActions func(*AgentConfig) func(ctx context.Context, pool *AgentPool) []types.Action
4263
connectors func(*AgentConfig) []Connector
4364
dynamicPrompt func(*AgentConfig) func(ctx context.Context, pool *AgentPool) []DynamicPrompt
@@ -47,6 +68,13 @@ type AgentPool struct {
4768
skillsService SkillsProvider
4869
}
4970

71+
// SetRAGProvider sets the single RAG provider (HTTP or embedded). Must be called after pool creation.
72+
func (a *AgentPool) SetRAGProvider(fn RAGProvider) {
73+
a.Lock()
74+
defer a.Unlock()
75+
a.ragProvider = fn
76+
}
77+
5078
type Status struct {
5179
ActionResults []types.ActionState
5280
}
@@ -79,7 +107,6 @@ func loadPoolFromFile(path string) (*AgentPoolData, error) {
79107

80108
func NewAgentPool(
81109
defaultModel, defaultMultimodalModel, defaultTranscriptionModel, defaultTranscriptionLanguage, defaultTTSModel, apiURL, apiKey, directory string,
82-
LocalRAGAPI string,
83110
availableActions func(*AgentConfig) func(ctx context.Context, pool *AgentPool) []types.Action,
84111
connectors func(*AgentConfig) []Connector,
85112
promptBlocks func(*AgentConfig) func(ctx context.Context, pool *AgentPool) []DynamicPrompt,
@@ -108,7 +135,6 @@ func NewAgentPool(
108135
defaultTranscriptionModel: defaultTranscriptionModel,
109136
defaultTranscriptionLanguage: defaultTranscriptionLanguage,
110137
defaultTTSModel: defaultTTSModel,
111-
localRAGAPI: LocalRAGAPI,
112138
apiKey: apiKey,
113139
agents: make(map[string]*Agent),
114140
pool: make(map[string]AgentConfig),
@@ -143,7 +169,6 @@ func NewAgentPool(
143169
agentStatus: map[string]*Status{},
144170
pool: *poolData,
145171
connectors: connectors,
146-
localRAGAPI: LocalRAGAPI,
147172
dynamicPrompt: promptBlocks,
148173
filters: filters,
149174
availableActions: availableActions,
@@ -303,14 +328,8 @@ func (a *AgentPool) startAgentWithConfig(name, pooldir string, config *AgentConf
303328
} else {
304329
config.APIKey = a.apiKey
305330
}
306-
effectiveLocalRAGAPI := a.localRAGAPI
307-
if config.LocalRAGURL != "" {
308-
effectiveLocalRAGAPI = config.LocalRAGURL
309-
}
310-
effectiveLocalRAGKey := a.localRAGKey
311-
if config.LocalRAGAPIKey != "" {
312-
effectiveLocalRAGKey = config.LocalRAGAPIKey
313-
}
331+
effectiveLocalRAGAPI := config.LocalRAGURL
332+
effectiveLocalRAGKey := config.LocalRAGAPIKey
314333

315334
connectors := a.connectors(config)
316335
promptBlocks := a.dynamicPrompt(config)(ctx, a)
@@ -510,26 +529,27 @@ func (a *AgentPool) startAgentWithConfig(name, pooldir string, config *AgentConf
510529
}
511530
}
512531

513-
var ragClient *localrag.WrappedClient
514-
if config.EnableKnowledgeBase {
515-
ragClient = localrag.NewWrappedClient(effectiveLocalRAGAPI, effectiveLocalRAGKey, name)
516-
opts = append(opts, WithRAGDB(ragClient), EnableKnowledgeBase)
517-
// Set KB auto search option (defaults to true for backward compatibility)
518-
// For backward compatibility: if both new KB fields are false (zero values),
519-
// assume this is an old config and default KBAutoSearch to true
532+
var ragDB RAGDB
533+
var compactionClient KBCompactionClient
534+
if config.EnableKnowledgeBase && a.ragProvider != nil {
535+
if db, comp, ok := a.ragProvider(name, effectiveLocalRAGAPI, effectiveLocalRAGKey); ok && db != nil {
536+
ragDB = db
537+
compactionClient = comp
538+
}
539+
}
540+
if ragDB != nil {
541+
opts = append(opts, WithRAGDB(ragDB), EnableKnowledgeBase)
520542
kbAutoSearch := config.KBAutoSearch
521543
if !config.KBAutoSearch && !config.KBAsTools {
522-
// Both new fields are false, likely an old config - default to true for backward compatibility
523544
kbAutoSearch = true
524545
}
525546
opts = append(opts, WithKBAutoSearch(kbAutoSearch))
526-
// Inject KB wrapper actions if enabled
527-
if config.KBAsTools && ragClient != nil {
547+
if config.KBAsTools {
528548
kbResults := config.KnowledgeBaseResults
529549
if kbResults <= 0 {
530-
kbResults = 5 // Default
550+
kbResults = 5
531551
}
532-
searchAction, addAction := NewKBWrapperActions(ragClient, kbResults)
552+
searchAction, addAction := NewKBWrapperActions(ragDB, kbResults)
533553
opts = append(opts, WithActions(searchAction, addAction))
534554
}
535555
}
@@ -582,8 +602,8 @@ func (a *AgentPool) startAgentWithConfig(name, pooldir string, config *AgentConf
582602
}
583603
}()
584604

585-
if config.EnableKnowledgeBase && config.EnableKBCompaction && ragClient != nil {
586-
go runCompactionTicker(ctx, ragClient, config, effectiveAPIURL, effectiveAPIKey, model)
605+
if config.EnableKnowledgeBase && config.EnableKBCompaction && compactionClient != nil {
606+
go runCompactionTicker(ctx, compactionClient, config, effectiveAPIURL, effectiveAPIKey, model)
587607
}
588608

589609
xlog.Info("Starting connectors", "name", name, "config", config)

‎docker-compose.amd.yaml‎

Lines changed: 4 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -11,25 +11,15 @@ services:
1111
- /dev/dri
1212
- /dev/kfd
1313

14-
dind:
15-
extends:
16-
file: docker-compose.yaml
17-
service: dind
18-
19-
localrecall-postgres:
14+
postgres:
2015
extends:
2116
file: docker-compose.yaml
22-
service: localrecall-postgres
17+
service: postgres
2318

24-
localrecall:
25-
extends:
26-
file: docker-compose.yaml
27-
service: localrecall
28-
29-
localrecall-healthcheck:
19+
dind:
3020
extends:
3121
file: docker-compose.yaml
32-
service: localrecall-healthcheck
22+
service: dind
3323

3424
localagi:
3525
extends:

‎docker-compose.intel.yaml‎

Lines changed: 4 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -12,25 +12,15 @@ services:
1212
- /dev/dri/card1
1313
- /dev/dri/renderD129
1414

15-
dind:
16-
extends:
17-
file: docker-compose.yaml
18-
service: dind
19-
20-
localrecall-postgres:
15+
postgres:
2116
extends:
2217
file: docker-compose.yaml
23-
service: localrecall-postgres
18+
service: postgres
2419

25-
localrecall:
26-
extends:
27-
file: docker-compose.yaml
28-
service: localrecall
29-
30-
localrecall-healthcheck:
20+
dind:
3121
extends:
3222
file: docker-compose.yaml
33-
service: localrecall-healthcheck
23+
service: dind
3424

3525
localagi:
3626
extends:

‎docker-compose.nvidia.yaml‎

Lines changed: 4 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -17,25 +17,15 @@ services:
1717
count: 1
1818
capabilities: [gpu]
1919

20-
dind:
21-
extends:
22-
file: docker-compose.yaml
23-
service: dind
24-
25-
localrecall-postgres:
20+
postgres:
2621
extends:
2722
file: docker-compose.yaml
28-
service: localrecall-postgres
23+
service: postgres
2924

30-
localrecall:
31-
extends:
32-
file: docker-compose.yaml
33-
service: localrecall
34-
35-
localrecall-healthcheck:
25+
dind:
3626
extends:
3727
file: docker-compose.yaml
38-
service: localrecall-healthcheck
28+
service: dind
3929

4030
localagi:
4131
extends:

0 commit comments

Comments
 (0)