feat: API1.1

.gitignore

@@ -15,6 +15,7 @@ evaluation/.env
 !evaluation/configs-example/*.json
 evaluation/configs/*
 **tree_textual_memory_locomo**
+**script.py**
 .env
 evaluation/scripts/personamem
 

README.md

@@ -54,22 +54,20 @@
 
 ## 📈 Performance Benchmark
 
-MemOS demonstrates significant improvements over baseline memory solutions in multiple reasoning tasks.
+MemOS demonstrates significant improvements over baseline memory solutions in multiple memory tasks,
+showcasing its capabilities in **information extraction**, **temporal and cross-session reasoning**, and **personalized preference responses**.
 
-| Model       | Avg. Score | Multi-Hop | Open Domain | Single-Hop | Temporal Reasoning |
-|-------------|------------|-----------|-------------|------------|---------------------|
-| **OpenAI**  | 0.5275     | 0.6028    | 0.3299      | 0.6183     | 0.2825              |
-| **MemOS**   | **0.7331** | **0.6430** | **0.5521**   | **0.7844** | **0.7321**          |
-| **Improvement** | **+38.98%** | **+6.67%** | **+67.35%** | **+26.86%** | **+159.15%**       |
+| Model           | LOCOMO      | LongMemEval | PrefEval-10 | PersonaMem  |
+|-----------------|-------------|-------------|-------------|-------------|
+| **GPT-4o-mini** | 52.75       | 55.4        | 2.8         | 43.46       |
+| **MemOS**       | **75.80**   | **77.80**   | **71.90**   | **61.17**   |
+| **Improvement** | **+43.70%** | **+40.43%** | **+2568%**  | **+40.75%** |
 
-> 💡 **Temporal reasoning accuracy improved by 159% compared to the OpenAI baseline.**
-
-### Details of End-to-End Evaluation on LOCOMO
-
-> [!NOTE]
-> Comparison of LLM Judge Scores across five major tasks in the LOCOMO benchmark. Each bar shows the mean evaluation score judged by LLMs for a given method-task pair, with standard deviation as error bars. MemOS-0630 consistently outperforms baseline methods (LangMem, Zep, OpenAI, Mem0) across all task types, especially in multi-hop and temporal reasoning scenarios.
-
-<img src="https://statics.memtensor.com.cn/memos/score_all_end2end.jpg" alt="END2END SCORE">
+### Detailed Evaluation Results
+- We use gpt-4o-mini as the processing and judging LLM and bge-m3 as embedding model in MemOS evaluation.
+- The evaluation was conducted under conditions that align various settings as closely as possible. Reproduce the results with our scripts at [`evaluation`](./evaluation).
+- Check the full search and response details at huggingface https://huggingface.co/datasets/MemTensor/MemOS_eval_result.
+> 💡 **MemOS outperforms all other methods (Mem0, Zep, Memobase, SuperMemory et al.) across all benchmarks!**
 
 ## ✨ Key Features
 
@@ -83,6 +81,27 @@ MemOS demonstrates significant improvements over baseline memory solutions in mu
 
 ## 🚀 Getting Started
 
+### ⭐️ MemOS online API
+The easiest way to use MemOS. Equip your agent with memory **in minutes**!
+
+Sign up and get started on[`MemOS dashboard`](https://memos-dashboard.openmem.net/cn/quickstart/?source=landing).
+
+
+### Self-Hosted Server
+1. Get the repository.
+```bash
+git clone https://github.com/MemTensor/MemOS.git
+cd MemOS
+pip install -r ./docker/requirements.txt
+```
+
+2. Configure `docker/.env.example` and copy to `MemOS/.env`
+3. Start the service.
+```bash
+uvicorn memos.api.server_api:app --host 0.0.0.0 --port 8001 --workers 8
+```
+
+### Local SDK
 Here's a quick example of how to create a **`MemCube`**, load it from a directory, access its memories, and save it.
 
 ```python
@@ -104,7 +123,7 @@ for item in mem_cube.act_mem.get_all():
 mem_cube.dump("tmp/mem_cube")
 ```
 
-What about **`MOS`** (Memory Operating System)? It's a higher-level orchestration layer that manages multiple MemCubes and provides a unified API for memory operations. Here's a quick example of how to use MOS:
+**`MOS`** (Memory Operating System) is a higher-level orchestration layer that manages multiple MemCubes and provides a unified API for memory operations. Here's a quick example of how to use MOS:
 
 ```python
 from memos.configs.mem_os import MOSConfig

docker/.env.example

@@ -1,29 +1,60 @@
 # MemOS Environment Variables Configuration
+TZ=Asia/Shanghai
 
-# Path to memory storage (e.g. /tmp/data_test)
-MOS_CUBE_PATH=
+MOS_CUBE_PATH="/tmp/data_test"            # Path to memory storage (e.g. /tmp/data_test)
+MOS_ENABLE_DEFAULT_CUBE_CONFIG="true"     # Enable default cube config (true/false)
 
 # OpenAI Configuration
-OPENAI_API_KEY=            # Your OpenAI API key
-OPENAI_API_BASE=           # OpenAI API base URL (default: https://api.openai.com/v1)
+OPENAI_API_KEY="sk-xxx"                   # Your OpenAI API key
+OPENAI_API_BASE="http://xxx"              # OpenAI API base URL (default: https://api.openai.com/v1)
 
-# MemOS Feature Toggles
-MOS_ENABLE_DEFAULT_CUBE_CONFIG=   # Enable default cube config (true/false)
-MOS_ENABLE_SCHEDULER=             # Enable background scheduler (true/false)
+# MemOS Chat Model Configuration
+MOS_CHAT_MODEL=gpt-4o-mini
+MOS_CHAT_TEMPERATURE=0.8
+MOS_MAX_TOKENS=8000
+MOS_TOP_P=0.9
+MOS_TOP_K=50
+MOS_CHAT_MODEL_PROVIDER=openai
 
-# Neo4j Configuration
-NEO4J_URI=               # Neo4j connection URI (e.g. bolt://localhost:7687)
-NEO4J_USER=              # Neo4j username
-NEO4J_PASSWORD=          # Neo4j password
-MOS_NEO4J_SHARED_DB=     # Shared Neo4j database name (if using multi-db)
+# graph db
+# neo4j
+NEO4J_BACKEND=xxx
+NEO4J_URI=bolt://xxx
+NEO4J_USER=xxx
+NEO4J_PASSWORD=xxx
+MOS_NEO4J_SHARED_DB=xxx
+NEO4J_DB_NAME=xxx
+
+# tetxmem reog
+MOS_ENABLE_REORGANIZE=false
 
 # MemOS User Configuration
-MOS_USER_ID=             # Unique user ID
-MOS_SESSION_ID=          # Session ID for current chat
-MOS_MAX_TURNS_WINDOW=    # Max number of turns to keep in memory
+MOS_USER_ID=root
+MOS_SESSION_ID=default_session
+MOS_MAX_TURNS_WINDOW=20
+
+# MemRader Configuration
+MEMRADER_MODEL=gpt-4o-mini
+MEMRADER_API_KEY=sk-xxx
+MEMRADER_API_BASE=http://xxx:3000/v1
+MEMRADER_MAX_TOKENS=5000
+
+#embedding & rerank
+EMBEDDING_DIMENSION=1024
+MOS_EMBEDDER_BACKEND=universal_api
+MOS_EMBEDDER_MODEL=bge-m3
+MOS_EMBEDDER_API_BASE=http://xxx
+MOS_EMBEDDER_API_KEY=EMPTY
+MOS_RERANKER_BACKEND=http_bge
+MOS_RERANKER_URL=http://xxx
+# Ollama Configuration (for embeddings)
+#OLLAMA_API_BASE=http://xxx
 
-# Ollama Configuration (for local embedding models)
-OLLAMA_API_BASE=         # Ollama API base URL (e.g. http://localhost:11434)
+# milvus for pref mem
+MILVUS_URI=http://xxx
+MILVUS_USER_NAME=xxx
+MILVUS_PASSWORD=xxx
 
-# Embedding Configuration
-MOS_EMBEDDER_BACKEND=    # Embedding backend: openai, ollama, etc.
+# pref mem
+ENABLE_PREFERENCE_MEMORY=true
+RETURN_ORIGINAL_PREF_MEM=true

docker/requirements.txt

@@ -157,4 +157,4 @@ volcengine-python-sdk==4.0.6
 watchfiles==1.1.0
 websockets==15.0.1
 xlrd==2.0.2
-xlsxwriter==3.2.5
+xlsxwriter==3.2.5

docs/openapi.json

@@ -884,7 +884,7 @@
             "type": "string",
             "title": "Session Id",
             "description": "Session ID for the MOS. This is used to distinguish between different dialogue",
-            "default": "0ce84b9c-0615-4b9d-83dd-fba50537d5d3"
+            "default": "41bb5e18-252d-4948-918c-07d82aa47086"
           },
           "chat_model": {
             "$ref": "#/components/schemas/LLMConfigFactory",
@@ -939,6 +939,12 @@
             "description": "Enable parametric memory for the MemChat",
             "default": false
           },
+          "enable_preference_memory": {
+            "type": "boolean",
+            "title": "Enable Preference Memory",
+            "description": "Enable preference memory for the MemChat",
+            "default": false
+          },
           "enable_mem_scheduler": {
             "type": "boolean",
             "title": "Enable Mem Scheduler",

evaluation/.env-example

@@ -3,39 +3,22 @@ MODEL="gpt-4o-mini"
 OPENAI_API_KEY="sk-***REDACTED***"
 OPENAI_BASE_URL="http://***.***.***.***:3000/v1"
 
-MEM0_API_KEY="m0-***REDACTED***"
-
-ZEP_API_KEY="z_***REDACTED***"
 
 # response model
 CHAT_MODEL="gpt-4o-mini"
 CHAT_MODEL_BASE_URL="http://***.***.***.***:3000/v1"
 CHAT_MODEL_API_KEY="sk-***REDACTED***"
 
+# memos
 MEMOS_KEY="Token mpg-xxxxx"
-MEMOS_URL="https://apigw-pre.memtensor.cn/api/openmem/v1"
-PRE_SPLIT_CHUNK=false  # pre split chunk in client end
-
-MEMOBASE_API_KEY="xxxxx"
-MEMOBASE_PROJECT_URL="http://xxx.xxx.xxx.xxx:8019"
-
-# Configuration Only For Scheduler
-# RabbitMQ Configuration
-MEMSCHEDULER_RABBITMQ_HOST_NAME=rabbitmq-cn-***.cn-***.amqp-32.net.mq.amqp.aliyuncs.com
-MEMSCHEDULER_RABBITMQ_USER_NAME=***
-MEMSCHEDULER_RABBITMQ_PASSWORD=***
-MEMSCHEDULER_RABBITMQ_VIRTUAL_HOST=memos
-MEMSCHEDULER_RABBITMQ_ERASE_ON_CONNECT=true
-MEMSCHEDULER_RABBITMQ_PORT=5672
+MEMOS_URL="http://127.0.0.1:8001"
+MEMOS_ONLINE_URL="https://memos.memtensor.cn/api/openmem/v1"
 
-# OpenAI Configuration
-MEMSCHEDULER_OPENAI_API_KEY=sk-***
-MEMSCHEDULER_OPENAI_BASE_URL=http://***.***.***.***:3000/v1
-MEMSCHEDULER_OPENAI_DEFAULT_MODEL=gpt-4o-mini
+# other memory agents
+MEM0_API_KEY="m0-xxx"
+ZEP_API_KEY="z_xxx"
+MEMU_API_KEY="mu_xxx"
+SUPERMEMORY_API_KEY="sm_xxx"
+MEMOBASE_API_KEY="xxx"
+MEMOBASE_PROJECT_URL="http://***.***.***.***:8019"
 
-# Graph DB Configuration
-MEMSCHEDULER_GRAPHDBAUTH_URI=bolt://localhost:7687
-MEMSCHEDULER_GRAPHDBAUTH_USER=neo4j
-MEMSCHEDULER_GRAPHDBAUTH_PASSWORD=***
-MEMSCHEDULER_GRAPHDBAUTH_DB_NAME=neo4j
-MEMSCHEDULER_GRAPHDBAUTH_AUTO_CREATE=true

evaluation/README.md

@@ -1,6 +1,6 @@
 # Evaluation Memory Framework
 
-This repository provides tools and scripts for evaluating the LoCoMo dataset using various models and APIs.
+This repository provides tools and scripts for evaluating the `LoCoMo`, `LongMemEval`, `PrefEval`, `personaMem` dataset using various models and APIs.
 
 ## Installation
 
@@ -16,16 +16,35 @@ This repository provides tools and scripts for evaluating the LoCoMo dataset usi
    ```
 
 ## Configuration
+Copy the `.env-example` file to `.env`, and fill in the required environment variables according to your environment and API keys.
 
-1. Copy the `.env-example` file to `.env`, and fill in the required environment variables according to your environment and API keys.
+## Setup MemOS
+### local server
+```bash
+# modify {project_dir}/.env file and start server
+uvicorn memos.api.server_api:app --host 0.0.0.0 --port 8001 --workers 8
+
+# configure {project_dir}/evaluation/.env file
+MEMOS_URL="http://127.0.0.1:8001"
+```
+### online service
+```bash
+# get your api key at https://memos-dashboard.openmem.net/cn/quickstart/
+# configure {project_dir}/evaluation/.env file
+MEMOS_KEY="Token mpg-xxxxx"
+MEMOS_ONLINE_URL="https://memos.memtensor.cn/api/openmem/v1"
+
+```
 
-2. Copy the `configs-example/` directory to a new directory named `configs/`, and modify the configuration files inside it as needed. This directory contains model and API-specific settings.
+## Supported frameworks
+We support `memos-api` and `memos-api-online` in our scripts.
+And give unofficial implementations for the following memory frameworks:`zep`, `mem0`, `memobase`, `supermemory`, `memu`.
 
 
 ## Evaluation Scripts
 
 ### LoCoMo Evaluation
-⚙️ To evaluate the **LoCoMo** dataset using one of the supported memory frameworks — `memos`, `mem0`, or `zep` — run the following [script](./scripts/run_locomo_eval.sh):
+⚙️ To evaluate the **LoCoMo** dataset using one of the supported memory frameworks — run the following [script](./scripts/run_locomo_eval.sh):
 
 ```bash
 # Edit the configuration in ./scripts/run_locomo_eval.sh
@@ -45,10 +64,21 @@ First prepare the dataset `longmemeval_s` from https://huggingface.co/datasets/x
 ./scripts/run_lme_eval.sh
 ```
 
-### prefEval Evaluation
+### PrefEval Evaluation
+Downloading benchmark_dataset/filtered_inter_turns.json from https://github.com/amazon-science/PrefEval/blob/main/benchmark_dataset/filtered_inter_turns.json and save it as `./data/prefeval/filtered_inter_turns.json`.
+To evaluate the **Prefeval** dataset — run the following [script](./scripts/run_prefeval_eval.sh):
 
-### personaMem Evaluation
+```bash
+# Edit the configuration in ./scripts/run_prefeval_eval.sh
+# Specify the model and memory backend you want to use (e.g., mem0, zep, etc.)
+./scripts/run_prefeval_eval.sh
+```
+
+### PersonaMem Evaluation
 get `questions_32k.csv` and `shared_contexts_32k.jsonl` from https://huggingface.co/datasets/bowen-upenn/PersonaMem and save them at `data/personamem/`
 ```bash
+# Edit the configuration in ./scripts/run_pm_eval.sh
+# Specify the model and memory backend you want to use (e.g., mem0, zep, etc.)
+# If you want to use MIRIX, edit the the configuration in ./scripts/personamem/config.yaml
 ./scripts/run_pm_eval.sh
 ```