diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml index b1d91624a..31c3ca656 100644 --- a/.github/ISSUE_TEMPLATE/bug-report.yml +++ b/.github/ISSUE_TEMPLATE/bug-report.yml @@ -1,38 +1,64 @@ -name: "\U0001F41B Bug / help" -description: Create a report to help us improve MemOS +name: "\U0001F41B Bug Report" +description: Report a bug to help us improve MemOS | 报告错误以帮助我们改进 MemOS +title: "fix: " labels: ["bug", "pending"] body: + - type: checkboxes + id: checklist + attributes: + label: Pre-submission checklist | 提交前检查 + options: + - label: I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题，确认此问题尚未被提及 + required: true + - label: I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在 + required: true + - label: This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的，而不是一般软件问题 + required: true + - type: textarea - id: system-info + id: description + attributes: + label: "Bug Description | 问题描述" + placeholder: "Describe what happened and what you expected to happen" validations: required: true - attributes: - label: System Info - description: | - Please share your system info with us. You can run the command **pip show MemoryOS** and copy-paste its output below. - 请提供您的系统信息。您可以在命令行运行 **pip show MemoryOS** 并将其输出复制到该文本框中。 - - placeholder: MemoryOS version, platform, python version, ... - type: textarea id: reproduction + attributes: + label: "How to Reproduce | 如何重现" + placeholder: | + 1. Import/run '...' + 2. Call function '...' + 3. See error validations: required: true - attributes: - label: Reproduction - description: | - Please provide entry arguments, error messages and stack traces that reproduces the problem. - 请提供入口参数，错误日志以及异常堆栈以便于我们复现问题。 - value: | - ```text - Put your message here. - ``` + - type: textarea + id: environment + attributes: + label: "Environment | 环境信息" + placeholder: | + - Python version: + - Operating System: + - MemOS version: (run `pip show memoryos`) + validations: + required: true - type: textarea id: others validations: required: false attributes: - label: Others + label: "Additional Context | 其他信息" + + - type: checkboxes + id: contribution + attributes: + label: Willingness to Implement | 实现意愿 + options: + - label: I'm willing to implement this myself | 我愿意自己解决 + required: false + - label: I would like someone else to implement this | 我希望其他人来解决 + required: false diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 000000000..8b3604a00 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,14 @@ +blank_issues_enabled: false +contact_links: + - name: "\U0001F527 GitHub Pull Requests" + url: https://github.com/MemTensor/MemOS/pulls + about: Contribute code improvements via Pull Requests | 通过 Pull Requests 贡献代码改进 + - name: "\U0001F4AC GitHub Discussions" + url: https://github.com/MemTensor/MemOS/discussions + about: Participate in our GitHub Discussions to ask questions or share ideas | 加入 GitHub Discussions，提出问题或分享想法 + - name: "\U0001F3AE Discord Server" + url: https://discord.gg/Txbx3gebZR + about: Join our Discord Server for real-time community chat | 加入我们的 Discord 服务器进行实时社区聊天 + - name: "\U0001F4F1 WeChat Group" + url: https://statics.memtensor.com.cn/memos/qr-code.png + about: Scan the QR code to join our WeChat group for more discussions | 扫描二维码加入我们的微信群，进行更多讨论 diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature-request.yml index ed17b4109..b441250fa 100644 --- a/.github/ISSUE_TEMPLATE/feature-request.yml +++ b/.github/ISSUE_TEMPLATE/feature-request.yml @@ -1,17 +1,19 @@ name: "\U0001F680 Feature request" -description: Submit a request for a new feature +description: Submit a request for a new feature | 申请添加新功能 +title: "feat: " labels: ["enhancement", "pending"] body: + - type: checkboxes id: checklist attributes: - label: Pre-submission checklist + label: Pre-submission checklist | 提交前检查 options: - - label: I have searched existing issues and this feature hasn't been requested before | 我已搜索现有问题，确认此功能尚未被请求 + - label: I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题，确认此问题尚未被提及 required: true - - label: I have read the project documentation and confirmed this feature doesn't already exist | 我已阅读项目文档并确认此功能尚未存在 + - label: I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在 required: true - - label: This feature request is specific to MemOS and not a general software issue | 该功能请求是针对 MemOS 的，而不是一般软件问题 + - label: This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的，而不是一般软件问题 required: true - type: textarea @@ -19,7 +21,7 @@ body: validations: required: true attributes: - label: Problem Statement + label: Problem Statement | 问题陈述 placeholder: | Describe the problem you're trying to solve... Example: "As a developer using MemOS, I find it difficult to..." @@ -27,9 +29,9 @@ body: - type: checkboxes id: contribution attributes: - label: Implementation Contribution + label: Willingness to Implement | 实现意愿 options: - - label: I'm willing to implement this feature myself | 我愿意自己实现此功能 + - label: I'm willing to implement this myself | 我愿意自己解决 required: false - - label: I would like someone else to implement this | 我希望其他人来实现此功能 + - label: I would like someone else to implement this | 我希望其他人来解决 required: false diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index f532a0c42..4c2d6ca16 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -3,19 +3,23 @@ Summary: (summary) Fix: #(issue) +Docs Issue/PR: (docs-issue-or-pr-link) + Reviewer: @(reviewer) ## Checklist: @@ -23,6 +27,6 @@ Reviewer: @(reviewer) - [ ] I have performed a self-review of my own code | 我已自行检查了自己的代码 - [ ] I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释 - [ ] I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常 -- [ ] I have added necessary documentation (if applicable) | 我已添加必要的文档（如果适用） +- [ ] I have created related documentation issue/PR in [MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) (if applicable) | 我已在 [MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) 中创建了相关的文档 issue/PR（如果适用） - [ ] I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR（如果适用） - [ ] I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人 diff --git a/.github/workflows/python-tests.yml b/.github/workflows/python-tests.yml index 83ff35572..152aac1e5 100644 --- a/.github/workflows/python-tests.yml +++ b/.github/workflows/python-tests.yml @@ -26,6 +26,7 @@ jobs: os: - "ubuntu-latest" - "windows-latest" + - "macos-13" - "macos-14" - "macos-15" # Ref: https://docs.github.com/en/actions/how-tos/writing-workflows/choosing-where-your-workflow-runs/choosing-the-runner-for-a-job @@ -46,13 +47,54 @@ jobs: with: python-version: ${{ matrix.python-version }} cache: 'poetry' - - name: Install dependencies + + # Dependency and building tests + - name: Install main dependencies + run: | + poetry install --no-root --no-interaction + - name: Check no top-level optional dependencies + run: | + poetry run python scripts/check_dependencies.py + - name: Build sdist and wheel + run: poetry build + - name: Test wheel installation on Windows + if: startsWith(matrix.os, 'windows') + run: | + Get-ChildItem dist/*.whl | ForEach-Object { pip install $_.FullName } + pip uninstall -y memoryos + - name: Test wheel installation on Linux / Mac + if: ${{ !startsWith(matrix.os, 'windows') }} + run: | + pip install dist/*.whl + pip uninstall -y memoryos + - name: Test sdist installation on Windows + if: startsWith(matrix.os, 'windows') run: | - poetry install --no-interaction --with dev --with test - - name: Test with ruff + Get-ChildItem dist/*.tar.gz | ForEach-Object { pip install $_.FullName } + pip uninstall -y memoryos + - name: Test sdist installation on Linux / Mac + if: ${{ !startsWith(matrix.os, 'windows') }} + run: | + pip install dist/*.tar.gz + pip uninstall -y memoryos + + # Ruff checks + - name: Install test group dependencies + run: | + poetry install --no-interaction --with test + - name: Ruff checks run: | poetry run ruff check poetry run ruff format --check - - name: Test with pytest + + # PyTest checks + - name: Install all extra dependencies + # macos-13 doesn't support torch==2.7.1 + # So, pytest won't work + if: ${{ !startsWith(matrix.os, 'macos-13') }} + run: | + poetry install --no-interaction --extras all + - name: PyTest unit tests + if: ${{ !startsWith(matrix.os, 'macos-13') }} run: | - poetry run pytest tests -vv + poetry run pytest tests -vv --durations=10 diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml new file mode 100644 index 000000000..b398497de --- /dev/null +++ b/.github/workflows/stale.yml @@ -0,0 +1,27 @@ +name: "Mark stale issues and PRs" + +on: + schedule: + - cron: '0 2 * * *' # Runs every day at 2 AM UTC + +permissions: + issues: write + pull-requests: write + +jobs: + stale: + runs-on: ubuntu-latest + steps: + - uses: actions/stale@v9 + with: + stale-issue-message: 'This issue has been automatically marked as stale due to inactivity.' + stale-pr-message: 'This PR has been automatically marked as stale due to inactivity.' + close-issue-message: 'This issue has been automatically closed due to inactivity.' + close-pr-message: 'This PR has been automatically closed due to inactivity.' + days-before-stale: 30 # Days of inactivity before marking as stale + days-before-close: 7 # Days of inactivity before closing stale issues/PRs + stale-issue-label: 'stale' + stale-pr-label: 'stale' + exempt-issue-labels: 'do not close' + exempt-pr-labels: 'do not close' + remove-stale-when-updated: true diff --git a/.gitignore b/.gitignore index abdbb4a87..184181828 100644 --- a/.gitignore +++ b/.gitignore @@ -6,10 +6,10 @@ tmp/ **/tmp_data/ # evaluation data -evaluation/data/langmemeval evaluation/*tmp/ evaluation/results evaluation/.env +!evaluation/configs-example/*.json evaluation/configs/* **tree_textual_memory_locomo** .env @@ -186,6 +186,9 @@ dmypy.json # Cython debug symbols cython_debug/ +# auth file +*_auth.yaml + # PyCharm # JetBrains specific template is maintained in a separate JetBrains.gitignore that can # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore diff --git a/Makefile b/Makefile index c250316ca..57ede5838 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ .PHONY: test install: - poetry install --with dev --with test + poetry install --extras all --with dev --with test poetry run pre-commit install --install-hooks clean: @@ -24,4 +24,4 @@ serve: poetry run uvicorn memos.api.start_api:app openapi: - poetry run python scripts/export_openapi.py --output docs/openapi.json + poetry run memos export_openapi --output docs/openapi.json diff --git a/README.md b/README.md index 00af88745..906663416 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,9 @@ Supported Python versions + + Supported Platforms + Documentation @@ -138,34 +141,37 @@ For more detailed examples, please check out the [`examples`](./examples) direct ## 📦 Installation -> [!WARNING] -> MemOS is compatible with Linux, Windows, and macOS. -> -> However, if you're using macOS, please note that there may be dependency issues that are difficult to resolve. -> -> For example, compatibility with macOS 13 Ventura is currently challenging. - ### Install via pip ```bash pip install MemoryOS ``` -### Development Install +### Optional Dependencies -To contribute to MemOS, clone the repository and install it in editable mode: +MemOS provides several optional dependency groups for different features. You can install them based on your needs. + +| Feature | Package Name | +| --------------------- | ------------------------- | +| Tree Memory | `MemoryOS[tree-mem]` | +| Memory Reader | `MemoryOS[mem-reader]` | +| Memory Scheduler | `MemoryOS[mem-scheduler]` | + +Example installation commands: ```bash -git clone https://github.com/MemTensor/MemOS.git -cd MemOS -make install +pip install MemoryOS[tree-mem] +pip install MemoryOS[tree-mem,mem-reader] +pip install MemoryOS[mem-scheduler] +pip install MemoryOS[tree-mem,mem-reader,mem-scheduler] ``` -### Optional Dependencies +### External Dependencies #### Ollama Support To use MemOS with [Ollama](https://ollama.com/), first install the Ollama CLI: + ```bash curl -fsSL https://ollama.com/install.sh | sh ``` @@ -174,6 +180,14 @@ curl -fsSL https://ollama.com/install.sh | sh To use functionalities based on the `transformers` library, ensure you have [PyTorch](https://pytorch.org/get-started/locally/) installed (CUDA version recommended for GPU acceleration). +#### Download Examples + +To download example code, data and configurations, run the following command: + +```bash +memos download_examples +``` + ## 💬 Community & Support Join our community to ask questions, share your projects, and connect with other developers. diff --git a/docker/.env.example b/docker/.env.example new file mode 100644 index 000000000..33f7ae853 --- /dev/null +++ b/docker/.env.example @@ -0,0 +1,29 @@ +# MemOS Environment Variables Configuration + +# Path to memory storage (e.g. /tmp/data_test) +MOS_CUBE_PATH= + +# OpenAI Configuration +OPENAI_API_KEY= # Your OpenAI API key +OPENAI_API_BASE= # OpenAI API base URL (default: https://api.openai.com/v1) + +# MemOS Feature Toggles +MOS_ENABLE_DEFAULT_CUBE_CONFIG= # Enable default cube config (true/false) +MOS_ENABLE_SCHEDULER= # Enable background scheduler (true/false) + +# Neo4j Configuration +NEO4J_URI= # Neo4j connection URI (e.g. bolt://localhost:7687) +NEO4J_USER= # Neo4j username +NEO4J_PASSWORD= # Neo4j password +MOS_NEO4J_SHARED_DB= # Shared Neo4j database name (if using multi-db) + +# MemOS User Configuration +MOS_USER_ID= # Unique user ID +MOS_SESSION_ID= # Session ID for current chat +MOS_MAX_TURNS_WINDOW= # Max number of turns to keep in memory + +# Ollama Configuration (for local embedding models) +OLLAMA_API_BASE= # Ollama API base URL (e.g. http://localhost:11434) + +# Embedding Configuration +MOS_EMBEDDER_BACKEND= # Embedding backend: openai, ollama, etc. diff --git a/docker/Dockerfile b/docker/Dockerfile new file mode 100644 index 000000000..ea158b111 --- /dev/null +++ b/docker/Dockerfile @@ -0,0 +1,25 @@ +FROM python:3.11-slim + +RUN apt-get update && apt-get install -y \ + gcc \ + g++ \ + build-essential \ + libffi-dev \ + python3-dev \ + curl \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app + +ENV HF_ENDPOINT=https://hf-mirror.com + +COPY requirements.txt . +RUN pip install --upgrade pip && pip install --no-cache-dir -r requirements.txt + +RUN pip install chonkie + +COPY ../. . +ENV PYTHONPATH=/app/src + +EXPOSE 8000 +CMD ["uvicorn", "memos.api.product_api:app", "--host", "0.0.0.0", "--port", "8000", "--reload"] diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml new file mode 100644 index 000000000..64d383a25 --- /dev/null +++ b/docker/docker-compose.yml @@ -0,0 +1,65 @@ +name: memos-dev + +services: + memos: + container_name: memos-api-server + build: + context: .. + dockerfile: Dockerfile + ports: + - "8000:8000" + env_file: + - ../.env + depends_on: + - neo4j + - qdrant + environment: + - PYTHONPATH=/app/src + volumes: + - .:/app + networks: + - memos_network + + neo4j: + image: neo4j:5.26.4 + container_name: neo4j-server + ports: + - "7474:7474" # HTTP + - "7687:7687" # Bolt + healthcheck: + test: wget http://localhost:7687 || exit 1 + interval: 1s + timeout: 10s + retries: 20 + start_period: 3s + environment: + NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes" + NEO4J_AUTH: "neo4j/12345678" + volumes: + - neo4j_data:/data + - neo4j_logs:/logs + networks: + - memos_network + + qdrant: + image: qdrant/qdrant:v1.15.0 + container_name: qdrant-server + ports: + - "6333:6333" # REST API + - "6334:6334" # gRPC API + volumes: + - ./qdrant_data:/qdrant/storage + environment: + QDRANT__SERVICE__GRPC_PORT: 6334 + QDRANT__SERVICE__HTTP_PORT: 6333 + restart: unless-stopped + networks: + - memos_network + +volumes: + neo4j_data: + neo4j_logs: + +networks: + memos_network: + driver: bridge diff --git a/docker/requirements.txt b/docker/requirements.txt new file mode 100644 index 000000000..596e94b47 --- /dev/null +++ b/docker/requirements.txt @@ -0,0 +1,92 @@ +annotated-types==0.7.0 ; python_version >= "3.10" and python_version < "4.0" +anyio==4.9.0 ; python_version >= "3.10" and python_version < "4.0" +attrs==25.3.0 ; python_version >= "3.10" and python_version < "4.0" +authlib==1.6.0 ; python_version >= "3.10" and python_version < "4.0" +certifi==2025.7.14 ; python_version >= "3.10" and python_version < "4.0" +cffi==1.17.1 ; python_version >= "3.10" and python_version < "4.0" and platform_python_implementation != "PyPy" +charset-normalizer==3.4.2 ; python_version >= "3.10" and python_version < "4.0" +click==8.2.1 ; python_version >= "3.10" and python_version < "4.0" +colorama==0.4.6 ; python_version >= "3.10" and python_version < "4.0" and (platform_system == "Windows" or sys_platform == "win32") +cryptography==45.0.5 ; python_version >= "3.10" and python_version < "4.0" +cyclopts==3.22.2 ; python_version >= "3.10" and python_version < "4.0" +distro==1.9.0 ; python_version >= "3.10" and python_version < "4.0" +dnspython==2.7.0 ; python_version >= "3.10" and python_version < "4.0" +docstring-parser==0.16 ; python_version >= "3.10" and python_version < "4.0" +docutils==0.21.2 ; python_version >= "3.10" and python_version < "4.0" +email-validator==2.2.0 ; python_version >= "3.10" and python_version < "4.0" +exceptiongroup==1.3.0 ; python_version >= "3.10" and python_version < "4.0" +fastapi-cli==0.0.8 ; python_version >= "3.10" and python_version < "4.0" +fastapi-cloud-cli==0.1.4 ; python_version >= "3.10" and python_version < "4.0" +fastapi==0.115.14 ; python_version >= "3.10" and python_version < "4.0" +fastmcp==2.10.5 ; python_version >= "3.10" and python_version < "4.0" +filelock==3.18.0 ; python_version >= "3.10" and python_version < "4.0" +fsspec==2025.7.0 ; python_version >= "3.10" and python_version < "4.0" +greenlet==3.2.3 ; python_version >= "3.10" and python_version < "3.14" and (platform_machine == "aarch64" or platform_machine == "ppc64le" or platform_machine == "x86_64" or platform_machine == "amd64" or platform_machine == "AMD64" or platform_machine == "win32" or platform_machine == "WIN32") +h11==0.16.0 ; python_version >= "3.10" and python_version < "4.0" +hf-xet==1.1.5 ; python_version >= "3.10" and python_version < "4.0" and (platform_machine == "x86_64" or platform_machine == "amd64" or platform_machine == "arm64" or platform_machine == "aarch64") +httpcore==1.0.9 ; python_version >= "3.10" and python_version < "4.0" +httptools==0.6.4 ; python_version >= "3.10" and python_version < "4.0" +httpx-sse==0.4.1 ; python_version >= "3.10" and python_version < "4.0" +httpx==0.28.1 ; python_version >= "3.10" and python_version < "4.0" +huggingface-hub==0.33.4 ; python_version >= "3.10" and python_version < "4.0" +idna==3.10 ; python_version >= "3.10" and python_version < "4.0" +itsdangerous==2.2.0 ; python_version >= "3.10" and python_version < "4.0" +jinja2==3.1.6 ; python_version >= "3.10" and python_version < "4.0" +jiter==0.10.0 ; python_version >= "3.10" and python_version < "4.0" +joblib==1.5.1 ; python_version >= "3.10" and python_version < "4.0" +jsonschema-specifications==2025.4.1 ; python_version >= "3.10" and python_version < "4.0" +jsonschema==4.24.1 ; python_version >= "3.10" and python_version < "4.0" +markdown-it-py==3.0.0 ; python_version >= "3.10" and python_version < "4.0" +markupsafe==3.0.2 ; python_version >= "3.10" and python_version < "4.0" +mcp==1.12.0 ; python_version >= "3.10" and python_version < "4.0" +mdurl==0.1.2 ; python_version >= "3.10" and python_version < "4.0" +numpy==2.2.6 ; python_version == "3.10" +numpy==2.3.1 ; python_version >= "3.11" and python_version < "4.0" +ollama==0.4.9 ; python_version >= "3.10" and python_version < "4.0" +openai==1.97.0 ; python_version >= "3.10" and python_version < "4.0" +openapi-pydantic==0.5.1 ; python_version >= "3.10" and python_version < "4.0" +orjson==3.11.0 ; python_version >= "3.10" and python_version < "4.0" +packaging==25.0 ; python_version >= "3.10" and python_version < "4.0" +pycparser==2.22 ; python_version >= "3.10" and python_version < "4.0" and platform_python_implementation != "PyPy" +pydantic-core==2.33.2 ; python_version >= "3.10" and python_version < "4.0" +pydantic-extra-types==2.10.5 ; python_version >= "3.10" and python_version < "4.0" +pydantic-settings==2.10.1 ; python_version >= "3.10" and python_version < "4.0" +pydantic==2.11.7 ; python_version >= "3.10" and python_version < "4.0" +pygments==2.19.2 ; python_version >= "3.10" and python_version < "4.0" +pyperclip==1.9.0 ; python_version >= "3.10" and python_version < "4.0" +python-dotenv==1.1.1 ; python_version >= "3.10" and python_version < "4.0" +python-multipart==0.0.20 ; python_version >= "3.10" and python_version < "4.0" +pywin32==311 ; python_version >= "3.10" and python_version < "4.0" and (platform_system == "Windows" or sys_platform == "win32") +pyyaml==6.0.2 ; python_version >= "3.10" and python_version < "4.0" +referencing==0.36.2 ; python_version >= "3.10" and python_version < "4.0" +regex==2024.11.6 ; python_version >= "3.10" and python_version < "4.0" +requests==2.32.4 ; python_version >= "3.10" and python_version < "4.0" +rich-rst==1.3.1 ; python_version >= "3.10" and python_version < "4.0" +rich-toolkit==0.14.8 ; python_version >= "3.10" and python_version < "4.0" +rich==14.0.0 ; python_version >= "3.10" and python_version < "4.0" +rignore==0.6.2 ; python_version >= "3.10" and python_version < "4.0" +rpds-py==0.26.0 ; python_version >= "3.10" and python_version < "4.0" +safetensors==0.5.3 ; python_version >= "3.10" and python_version < "4.0" +scikit-learn==1.7.0 ; python_version >= "3.10" and python_version < "4.0" +scipy==1.15.3 ; python_version == "3.10" +scipy==1.16.0 ; python_version >= "3.11" and python_version < "4.0" +sentry-sdk==2.33.0 ; python_version >= "3.10" and python_version < "4.0" +shellingham==1.5.4 ; python_version >= "3.10" and python_version < "4.0" +sniffio==1.3.1 ; python_version >= "3.10" and python_version < "4.0" +sqlalchemy==2.0.41 ; python_version >= "3.10" and python_version < "4.0" +sse-starlette==2.4.1 ; python_version >= "3.10" and python_version < "4.0" +starlette==0.46.2 ; python_version >= "3.10" and python_version < "4.0" +tenacity==9.1.2 ; python_version >= "3.10" and python_version < "4.0" +threadpoolctl==3.6.0 ; python_version >= "3.10" and python_version < "4.0" +tokenizers==0.21.2 ; python_version >= "3.10" and python_version < "4.0" +tqdm==4.67.1 ; python_version >= "3.10" and python_version < "4.0" +transformers==4.53.2 ; python_version >= "3.10" and python_version < "4.0" +typer==0.16.0 ; python_version >= "3.10" and python_version < "4.0" +typing-extensions==4.14.1 ; python_version >= "3.10" and python_version < "4.0" +typing-inspection==0.4.1 ; python_version >= "3.10" and python_version < "4.0" +ujson==5.10.0 ; python_version >= "3.10" and python_version < "4.0" +urllib3==2.5.0 ; python_version >= "3.10" and python_version < "4.0" +uvicorn==0.35.0 ; python_version >= "3.10" and python_version < "4.0" +uvloop==0.21.0 ; python_version >= "3.10" and python_version < "4.0" and platform_python_implementation != "PyPy" and sys_platform != "win32" and sys_platform != "cygwin" +watchfiles==1.1.0 ; python_version >= "3.10" and python_version < "4.0" +websockets==15.0.1 ; python_version >= "3.10" and python_version < "4.0" diff --git a/docs/openapi.json b/docs/openapi.json index 52b6980ac..15e834d04 100644 --- a/docs/openapi.json +++ b/docs/openapi.json @@ -884,7 +884,7 @@ "type": "string", "title": "Session Id", "description": "Session ID for the MOS. This is used to distinguish between different dialogue", - "default": "3d88949f-cbe1-4244-a2e1-d346e8b76ca0" + "default": "a47d75a0-5ee8-473f-86c4-3f09073fd59f" }, "chat_model": { "$ref": "#/components/schemas/LLMConfigFactory", diff --git a/evaluation/README.md b/evaluation/README.md index 39188aea4..19da665ad 100644 --- a/evaluation/README.md +++ b/evaluation/README.md @@ -12,7 +12,7 @@ This repository provides tools and scripts for evaluating the LoCoMo dataset usi 2. Install the required dependencies: ```bash - poetry install --with eval + poetry install --extras all --with eval ``` ## Configuration diff --git a/evaluation/configs-example/mem_cube_config.json b/evaluation/configs-example/mem_cube_config.json new file mode 100644 index 000000000..d609d27b0 --- /dev/null +++ b/evaluation/configs-example/mem_cube_config.json @@ -0,0 +1,51 @@ +{ + "user_id": "__USER_ID__", + "cube_id": "__USER_ID__", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": { + "backend": "openai", + "config": { + "model_name_or_path": "gpt-4o-mini", + "temperature": 0.8, + "max_tokens": 1024, + "top_p": 0.9, + "top_k": 50, + "api_key": "sk-***REDACTED***", + "api_base": "http://***.***.***.***:3000/v1" + } + }, + "dispatcher_llm": { + "backend": "openai", + "config": { + "model_name_or_path": "gpt-4o-mini", + "temperature": 0.8, + "max_tokens": 1024, + "top_p": 0.9, + "top_k": 50, + "api_key": "sk-***REDACTED***", + "api_base": "http://***.***.***.***:3000/v1" + } + }, + "graph_db": { + "backend": "neo4j", + "config": { + "uri": "bolt://***.***.***.***:7687", + "user": "***REDACTED***", + "password": "***REDACTED***", + "db_name": "__DB_NAME__", + "auto_create": true + } + }, + "embedder": { + "backend": "ollama", + "config": { + "model_name_or_path": "nomic-embed-text:latest" + } + } + } + }, + "act_mem": {}, + "para_mem": {} +} diff --git a/evaluation/configs-example/mos_memos_config.json b/evaluation/configs-example/mos_memos_config.json new file mode 100644 index 000000000..b7f2767b7 --- /dev/null +++ b/evaluation/configs-example/mos_memos_config.json @@ -0,0 +1,51 @@ +{ + "user_id": "root", + "chat_model": { + "backend": "openai", + "config": { + "model_name_or_path": "gpt-4o-mini", + "api_key": "sk-***REDACTED***", + "api_base": "http://***.***.***.***:3000/v1", + "temperature": 0.1, + "remove_think_prefix": true, + "max_tokens": 4096 + } + }, + "mem_reader": { + "backend": "simple_struct", + "config": { + "llm": { + "backend": "openai", + "config": { + "model_name_or_path": "gpt-4o-mini", + "temperature": 0.8, + "max_tokens": 1024, + "top_p": 0.9, + "top_k": 50, + "api_key": "sk-***REDACTED***", + "api_base": "http://***.***.***.***:3000/v1" + } + }, + "embedder": { + "backend": "ollama", + "config": { + "model_name_or_path": "nomic-embed-text:latest" + } + }, + "chunker": { + "backend": "sentence", + "config": { + "tokenizer_or_token_counter": "gpt2", + "chunk_size": 512, + "chunk_overlap": 128, + "min_sentences_per_chunk": 1 + } + } + } + }, + "max_turns_window": 30, + "top_k": "__TOP_K__", + "enable_textual_memory": true, + "enable_activation_memory": false, + "enable_parametric_memory": false +} diff --git a/evaluation/data/longmemeval/.gitkeep b/evaluation/data/longmemeval/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/evaluation/scripts/locomo/locomo_eval.py b/evaluation/scripts/locomo/locomo_eval.py index b5b478426..c6adbd61c 100644 --- a/evaluation/scripts/locomo/locomo_eval.py +++ b/evaluation/scripts/locomo/locomo_eval.py @@ -53,7 +53,7 @@ async def locomo_grader(llm_client, question: str, gold_answer: str, response: s """ accuracy_prompt = f""" - Your task is to label an answer to a question as ’CORRECT’ or ’WRONG’. You williolw23 be given the following data: + Your task is to label an answer to a question as ’CORRECT’ or ’WRONG’. You will be given the following data: (1) a question (posed by one user to another user), (2) a ’gold’ (ground truth) answer, (3) a generated answer diff --git a/evaluation/scripts/locomo/openai_memory_locomo_eval_guide.md b/evaluation/scripts/locomo/openai_memory_locomo_eval_guide.md index c7b5a7e3f..dc92bd5cc 100644 --- a/evaluation/scripts/locomo/openai_memory_locomo_eval_guide.md +++ b/evaluation/scripts/locomo/openai_memory_locomo_eval_guide.md @@ -81,7 +81,7 @@ Can you please extract relevant information from this conversation and create me * Click on **Manage** in the memory confirmation to view the newly generated memories. * Create a new local `.txt` file with the same name as the input file (e.g., `0-D1.txt`). * Copy each memory entry from ChatGPT and paste it into the new file, with each memory on a new line. -5. **Reset Memories for the Next Conversation:** +5. **Reset Memories for the Next Conversation:** * Once all sessions for a conversation are complete, it is essential to **delete all memories to ensure a clean state for the next conversation**. Navigate to Settings -> Personalization -> Manage and click Delete all. **Example Memory Output (`0-D9.txt`):** diff --git a/evaluation/scripts/longmemeval/lme_eval.py b/evaluation/scripts/longmemeval/lme_eval.py new file mode 100644 index 000000000..2d54a5acf --- /dev/null +++ b/evaluation/scripts/longmemeval/lme_eval.py @@ -0,0 +1,378 @@ +import argparse +import asyncio +import concurrent.futures +import json +import logging +import os +import sys + +import nltk +import numpy as np +import transformers + +from bert_score import score as bert_score +from dotenv import load_dotenv +from nltk.translate.bleu_score import SmoothingFunction, sentence_bleu +from nltk.translate.meteor_score import meteor_score +from openai import OpenAI +from pydantic import BaseModel, Field +from rouge_score import rouge_scorer +from scipy.spatial.distance import cosine +from sentence_transformers import SentenceTransformer +from tqdm import tqdm + + +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from utils.prompts import LME_JUDGE_MODEL_TEMPLATE + + +logging.basicConfig(level=logging.CRITICAL) +transformers.logging.set_verbosity_error() + +# Download necessary NLTK resources +try: + nltk.download("wordnet", quiet=True) + nltk.download("punkt", quiet=True) + print("NLTK resources downloaded successfully.") +except Exception as e: + print(f"Warning: Failed to download NLTK resources: {e}") + +try: + sentence_model_name = "Qwen/Qwen3-Embedding-0.6B" + sentence_model = SentenceTransformer(sentence_model_name) + print(f"SentenceTransformer model : {sentence_model_name} loaded successfully.") +except Exception as e: + print(f"Failed to load SentenceTransformer model: {e}") + sentence_model = None + + +class LLMGrade(BaseModel): + llm_judgment: str = Field(description="CORRECT or WRONG") + llm_reasoning: str = Field(description="Explain why the answer is correct or incorrect.") + + +def calculate_rouge_scores(golden_answer, response): + metrics = {"rouge1_f": 0.0, "rouge2_f": 0.0, "rougeL_f": 0.0} + try: + scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True) + rouge_scores = scorer.score(golden_answer, response) + metrics["rouge1_f"] = rouge_scores["rouge1"].fmeasure + metrics["rouge2_f"] = rouge_scores["rouge2"].fmeasure + metrics["rougeL_f"] = rouge_scores["rougeL"].fmeasure + except Exception as e: + print(f"Failed to calculate ROUGE scores: {e}") + return metrics + + +def calculate_bleu_scores(gold_tokens, response_tokens): + metrics = {"bleu1": 0.0, "bleu2": 0.0, "bleu3": 0.0, "bleu4": 0.0} + + try: + smoothing = SmoothingFunction().method1 + weights = [(1, 0, 0, 0), (0.5, 0.5, 0, 0), (0.33, 0.33, 0.33, 0), (0.25, 0.25, 0.25, 0.25)] + + for i, weight in enumerate(weights, 1): + metrics[f"bleu{i}"] = sentence_bleu( + [gold_tokens], response_tokens, weights=weight, smoothing_function=smoothing + ) + except ZeroDivisionError: + pass + except Exception as e: + print(f"Failed to calculate BLEU scores: {e}") + + return metrics + + +def calculate_meteor_score(gold_tokens, response_tokens): + try: + return meteor_score([gold_tokens], response_tokens) + except Exception as e: + print(f"Failed to calculate METEOR score: {e}") + return 0.0 + + +def calculate_semantic_similarity(golden_answer, response): + global sentence_model + + try: + if sentence_model is None: + sentence_model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B") + + gold_embedding = sentence_model.encode([golden_answer], show_progress_bar=False)[0] + response_embedding = sentence_model.encode([response], show_progress_bar=False)[0] + return 1 - cosine(gold_embedding, response_embedding) + except Exception as e: + print(f"Failed to calculate semantic similarity: {e}") + return 0.0 + + +def calculate_f1_score(gold_tokens, response_tokens): + try: + gold_set = set(gold_tokens) + response_set = set(response_tokens) + + if len(gold_set) == 0 or len(response_set) == 0: + return 0.0 + + precision = len(gold_set.intersection(response_set)) / len(response_set) + recall = len(gold_set.intersection(response_set)) / len(gold_set) + + if precision + recall > 0: + return 2 * precision * recall / (precision + recall) + return 0.0 + except Exception as e: + print(f"Failed to calculate F1 score: {e}") + return 0.0 + + +def calculate_nlp_metrics(golden_answer, response, context, options=None): + if options is None: + options = ["lexical", "semantic"] + + golden_answer = str(golden_answer) if golden_answer is not None else "" + response = str(response) if response is not None else "" + context = str(context) if context is not None else "" + + metrics = {"context_tokens": len(nltk.word_tokenize(context)) if context else 0} + + if "lexical" in options: + gold_tokens = nltk.word_tokenize(golden_answer.lower()) + response_tokens = nltk.word_tokenize(response.lower()) + + metrics["lexical"] = {} + metrics["lexical"]["f1"] = calculate_f1_score(gold_tokens, response_tokens) + metrics["lexical"].update(calculate_rouge_scores(golden_answer, response)) + metrics["lexical"].update(calculate_bleu_scores(gold_tokens, response_tokens)) + metrics["lexical"]["meteor"] = calculate_meteor_score(gold_tokens, response_tokens) + + if "semantic" in options: + metrics["semantic"] = {} + metrics["semantic"]["similarity"] = calculate_semantic_similarity(golden_answer, response) + _, _, f1 = bert_score( + [golden_answer], [response], lang="en", rescale_with_baseline=True, verbose=False + ) + metrics["semantic"]["bert_f1"] = f1.item() if f1 is not None else 0.0 + + return metrics + + +def lme_grader(llm_client, question, golden_answer, response): + system_prompt = """You are an expert grader that determines if answers to questions match a gold standard answer""" + judge_prompt = LME_JUDGE_MODEL_TEMPLATE.format( + question=question, golden_answer=golden_answer, response=response + ) + + response = llm_client.chat.completions.create( + model="gpt-4o-mini", + messages=[ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": judge_prompt}, + ], + temperature=0, + ) + + message_content = response.choices[0].message.content + label = json.loads(message_content)["label"] + parsed = LLMGrade(llm_judgment=label, llm_reasoning="") + + return parsed.llm_judgment.strip().lower() == "correct" + + +async def process_qa( + user_id, response_data, llm_client, num_runs: int, nlp_options=None, executor=None +): + question = response_data.get("question") + golden_answer = response_data.get("golden_answer", "") + context = response_data.get("search_context", "") + response = response_data.get("answer", "") + + loop = asyncio.get_event_loop() + tasks = [ + loop.run_in_executor(executor, lme_grader, llm_client, question, golden_answer, response) + for _ in range(num_runs) + ] + judgments = await asyncio.gather(*tasks) + judgments_dict = {f"judgment_{i + 1}": j for i, j in enumerate(judgments)} + + nlp_metrics = calculate_nlp_metrics( + golden_answer=golden_answer, response=response, context=context, options=nlp_options + ) + + print("\n" + "=" * 80) + print(f"🔍 Processed User: \033[1m{user_id}\033[0m") + print("-" * 80) + print(f"❓ Question: \n {question}") + print("-" * 80) + print( + f"📖 Golden Answer: \n {golden_answer[:150]}..." + if len(str(golden_answer)) > 150 + else f"📖 Golden Answer: \n {golden_answer}" + ) + print("-" * 80) + print( + f"💬 LLM Response: \n {response[:150]}..." + if len(str(response)) > 150 + else f"💬 Answer: \n {response}" + ) + print("-" * 80) + + judgments_formatted = [] + for run, correct in judgments_dict.items(): + status = "\033[92m✓ CORRECT\033[0m" if correct else "\033[91m✗ WRONG\033[0m" + judgments_formatted.append(f"{run}: {status}") + + print(f"⚖️ Judgments: \n {', '.join(judgments_formatted)}") + print("=" * 80) + + graded_response = { + "user_id": user_id, + "category": response_data.get("category"), + "question": question, + "question_date": response_data.get("question_date"), + "golden_answer": response_data.get("golden_answer"), + "answer": response, + "llm_judgments": judgments_dict, + "nlp_metrics": nlp_metrics, + "response_duration_ms": response_data.get("response_duration_ms"), + "search_duration_ms": response_data.get("search_duration_ms"), + "total_duration_ms": response_data.get("response_duration_ms") + + response_data.get("search_duration_ms", 0), + } + return graded_response + + +def convert_numpy_types(obj): + if isinstance(obj, np.number): + return float(obj) + elif isinstance(obj, dict): + return {k: convert_numpy_types(v) for k, v in obj.items()} + elif isinstance(obj, list): + return [convert_numpy_types(i) for i in obj] + else: + return obj + + +def evaluate_accuracy(results, num_runs): + run_scores = [] + evaluated_count = 0 + + for i in range(1, num_runs + 1): + judgment_key = f"judgment_{i}" + correct, total = 0, 0 + for _, response in results.items(): + if judgment_key in response["llm_judgments"]: + total += 1 + if response["llm_judgments"][judgment_key]: + correct += 1 + if total > 0: + run_scores.append(correct / total) + evaluated_count += total + evaluated_count = evaluated_count // num_runs + return run_scores, evaluated_count + + +async def main(frame, version, nlp_options, num_runs=3, num_workers=5): + print(f"Starting evaluation for {frame} version {version}...") + + load_dotenv() + oai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"), base_url=os.getenv("OPENAI_BASE_URL")) + + response_path = f"results/lme/{frame}-{version}/{frame}_lme_responses.json" + judged_path = f"results/lme/{frame}-{version}/{frame}_lme_judged.json" + + with open(response_path) as file: + lme_responses = json.load(file) + + lme_eval_results = {} + error_count = 0 + + executor = concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) + tasks = [ + process_qa(user_id, response_data, oai_client, num_runs, nlp_options, executor) + for user_id, response_data in lme_responses.items() + ] + results = [] + pbar = tqdm(total=len(tasks), desc="Processing users") + for coro in asyncio.as_completed(tasks): + try: + result = await coro + user_id = result["user_id"] + lme_eval_results[user_id] = result + results.append(result) + except Exception as exc: + print(f"[ERROR] Processing user failed: {exc}") + error_count += 1 + pbar.update(1) + pbar.close() + executor.shutdown() + + run_scores, evaluated_count = evaluate_accuracy(lme_eval_results, num_runs) + + print("\n" + "=" * 80) + print("\033[1;36m📊 EVALUATION SUMMARY\033[0m".center(80)) + print("=" * 80) + + if evaluated_count > 0: + print( + f"📋 \033[1mEvaluated:\033[0m \033[93m{evaluated_count}\033[0m responses across \033[93m{num_runs}\033[0m runs" + ) + print( + f"🎯 \033[1mLLM-as-a-Judge Mean Accuracy:\033[0m \033[92m{np.mean(run_scores):.4f}\033[0m" + ) + print(f"🔍 \033[1mStandard Deviation:\033[0m \033[93m{np.std(run_scores):.4f}\033[0m") + + run_scores_formatted = [f"\033[94m{round(s, 4):.4f}\033[0m" for s in run_scores] + print(f"🔢 \033[1mIndividual run scores:\033[0m [{', '.join(run_scores_formatted)}]") + else: + print("\033[91m⚠️ No responses were evaluated. LLM-as-a-Judge score: N/A (0/0)\033[0m") + + if error_count > 0: + print(f"\033[91m⚠️ Encountered {error_count} errors during processing\033[0m") + + print("-" * 80) + + # Convert and save results + lme_eval_results = convert_numpy_types(lme_eval_results) + with open(judged_path, "w") as file: + json.dump(lme_eval_results, file, indent=4) + + print("\033[92m✅ Evaluation completed successfully!\033[0m") + print(f"📁 Results saved to: \033[1;94m{judged_path}\033[0m") + print("=" * 80 + "\n") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Evaluate LLM responses using LLM-as-a-Judge.") + parser.add_argument( + "--lib", + type=str, + choices=["mem0-local", "mem0-api"], + ) + parser.add_argument( + "--version", type=str, default="v1", help="Version of the evaluation framework." + ) + parser.add_argument( + "--options", + type=str, + nargs="+", + default=["lexical", "semantic"], + choices=["lexical", "semantic"], + help="NLP options to use for evaluation.", + ) + parser.add_argument( + "--num_runs", type=int, default=3, help="Number of runs for LLM-as-a-Judge evaluation." + ) + parser.add_argument( + "--workers", type=int, default=3, help="Number of runs for LLM-as-a-Judge evaluation." + ) + + args = parser.parse_args() + asyncio.run( + main( + frame=args.lib, + version=args.version, + nlp_options=args.options, + num_runs=args.num_runs, + num_workers=args.workers, + ) + ) diff --git a/evaluation/scripts/longmemeval/lme_ingestion.py b/evaluation/scripts/longmemeval/lme_ingestion.py new file mode 100644 index 000000000..aef8e076d --- /dev/null +++ b/evaluation/scripts/longmemeval/lme_ingestion.py @@ -0,0 +1,207 @@ +import argparse +import os +import sys + + +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from concurrent.futures import ThreadPoolExecutor, as_completed +from datetime import datetime, timezone + +import pandas as pd + +from tqdm import tqdm +from utils.client import mem0_client, memos_client, zep_client +from zep_cloud.types import Message + + +def ingest_session(session, date, user_id, session_id, frame, client): + messages = [] + if frame == "zep": + for idx, msg in enumerate(session): + print( + f"\033[90m[{frame}]\033[0m 💬 Session \033[1;94m{session_id}\033[0m: [\033[93m{idx + 1}/{len(session)}\033[0m] Ingesting message: \033[1m{msg['role']}\033[0m - \033[96m{msg['content'][:50]}...\033[0m at \033[92m{date.isoformat()}\033[0m" + ) + client.memory.add( + session_id=session_id, + messages=[ + Message( + role=msg["role"], + role_type=msg["role"], + content=msg["content"][:8000], + created_at=date.isoformat(), + ) + ], + ) + elif frame == "mem0-local" or frame == "mem0-api": + for idx, msg in enumerate(session): + messages.append({"role": msg["role"], "content": msg["content"][:8000]}) + print( + f"\033[90m[{frame}]\033[0m 📝 Session \033[1;94m{session_id}\033[0m: [\033[93m{idx + 1}/{len(session)}\033[0m] Reading message: \033[1m{msg['role']}\033[0m - \033[96m{msg['content'][:50]}...\033[0m at \033[92m{date.isoformat()}\033[0m" + ) + if frame == "mem0-local": + client.add( + messages=messages, user_id=user_id, run_id=session_id, timestamp=date.isoformat() + ) + elif frame == "mem0-api": + client.add( + messages=messages, + user_id=user_id, + session_id=session_id, + timestamp=int(date.timestamp()), + version="v2", + ) + print( + f"\033[90m[{frame}]\033[0m ✅ Session \033[1;94m{session_id}\033[0m: Ingested \033[93m{len(messages)}\033[0m messages at \033[92m{date.isoformat()}\033[0m" + ) + elif frame == "memos-local": + for idx, msg in enumerate(session): + messages.append( + { + "role": msg["role"], + "content": msg["content"][:8000], + "chat_time": date.isoformat(), + } + ) + print( + f"\033[90m[{frame}]\033[0m 📝 Session \033[1;94m{session_id}\033[0m: [\033[93m{idx + 1}/{len(session)}\033[0m] Reading message: \033[1m{msg['role']}\033[0m - \033[96m{msg['content'][:50]}...\033[0m at \033[92m{date.isoformat()}\033[0m" + ) + client.add(messages=messages, user_id=user_id) + print( + f"\033[90m[{frame}]\033[0m ✅ Session \033[1;94m{session_id}\033[0m: Ingested \033[93m{len(messages)}\033[0m messages at \033[92m{date.isoformat()}\033[0m" + ) + + +def ingest_conv(lme_df, version, conv_idx, frame, num_workers=2): + conversation = lme_df.iloc[conv_idx] + + sessions = conversation["haystack_sessions"] + dates = conversation["haystack_dates"] + + user_id = "lme_exper_user_" + str(conv_idx) + session_id = "lme_exper_session_" + str(conv_idx) + + print("\n" + "=" * 80) + print(f"🔄 \033[1;36mINGESTING CONVERSATION {conv_idx}\033[0m".center(80)) + print("=" * 80) + + if frame == "zep": + client = zep_client() + print("🔌 \033[1mUsing \033[94mZep client\033[0m \033[1mfor ingestion...\033[0m") + # Delete existing user and session if they exist + client.user.delete(user_id) + client.memory.delete(session_id) + print( + f"🗑️ Deleted existing user \033[93m{user_id}\033[0m and session \033[93m{session_id}\033[0m from Zep memory..." + ) + # Add user and session to Zep memory + client.user.add(user_id=user_id) + client.memory.add_session( + user_id=user_id, + session_id=session_id, + ) + print( + f"➕ Added user \033[93m{user_id}\033[0m and session \033[93m{session_id}\033[0m to Zep memory..." + ) + elif frame == "mem0-local": + client = mem0_client(mode="local") + print("🔌 \033[1mUsing \033[94mMem0 Local client\033[0m \033[1mfor ingestion...\033[0m") + # Delete existing memories for the user + client.delete_all(user_id=user_id) + print(f"🗑️ Deleted existing memories for user \033[93m{user_id}\033[0m...") + elif frame == "mem0-api": + client = mem0_client(mode="api") + print("🔌 \033[1mUsing \033[94mMem0 API client\033[0m \033[1mfor ingestion...\033[0m") + # Delete existing memories for the user + client.delete_all(user_id=user_id) + print(f"🗑️ Deleted existing memories for user \033[93m{user_id}\033[0m...") + elif frame == "memos-local": + client = memos_client( + mode="local", + db_name=f"lme_{frame}-{version}-{user_id.replace('_', '')}", + user_id=user_id, + top_k=20, + mem_cube_path=f"results/lme/{frame}-{version}/storages/{user_id}", + mem_cube_config_path="configs/mem_cube_config.json", + mem_os_config_path="configs/mos_memos_config.json", + addorsearch="add", + ) + print("🔌 \033[1mUsing \033[94mMemos Local client\033[0m \033[1mfor ingestion...\033[0m") + + with ThreadPoolExecutor(max_workers=num_workers) as executor: + futures = [] + + for idx, session in enumerate(sessions): + date = dates[idx] + " UTC" + date_format = "%Y/%m/%d (%a) %H:%M UTC" + date_string = datetime.strptime(date, date_format).replace(tzinfo=timezone.utc) + + future = executor.submit( + ingest_session, session, date_string, user_id, session_id, frame, client + ) + futures.append(future) + + if len(session) == 0: + print(f"\033[93m⚠️ Skipping empty session {idx} in conversation {conv_idx}\033[0m") + continue + + for future in tqdm( + as_completed(futures), total=len(futures), desc=f"📊 Ingesting user {conv_idx}" + ): + try: + future.result() + except Exception as e: + print(f"\033[91m❌ Error ingesting session: {e}\033[0m") + + print("=" * 80) + + +def main(frame, version, num_workers=2): + print("\n" + "=" * 80) + print(f"🚀 \033[1;36mLONGMEMEVAL INGESTION - {frame.upper()} v{version}\033[0m".center(80)) + print("=" * 80) + + lme_df = pd.read_json("data/longmemeval/longmemeval_s.json") + + print( + "📚 \033[1mLoaded LongMemeval dataset\033[0m from \033[94mdata/longmemeval/longmemeval_s.json\033[0m" + ) + num_multi_sessions = len(lme_df) + print(f"👥 Number of users: \033[93m{num_multi_sessions}\033[0m") + print("-" * 80) + + start_time = datetime.now() + for session_idx in range(num_multi_sessions): + ingest_conv(lme_df, version, session_idx, frame, num_workers=num_workers) + end_time = datetime.now() + elapsed_time = end_time - start_time + elapsed_time_str = str(elapsed_time).split(".")[0] + + print("\n" + "=" * 80) + print("✅ \033[1;32mINGESTION COMPLETE\033[0m".center(80)) + print("=" * 80) + print( + f"⏱️ Total time taken to ingest \033[93m{num_multi_sessions}\033[0m multi-sessions: \033[92m{elapsed_time_str}\033[0m" + ) + print( + f"🔄 Framework: \033[94m{frame}\033[0m | Version: \033[94m{version}\033[0m | Workers: \033[94m{num_workers}\033[0m" + ) + print("=" * 80 + "\n") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="LongMemeval Ingestion Script") + parser.add_argument( + "--lib", + type=str, + choices=["mem0-local", "mem0-api", "memos-local"], + ) + parser.add_argument( + "--version", type=str, default="v1", help="Version of the evaluation framework." + ) + parser.add_argument( + "--workers", type=int, default=3, help="Number of runs for LLM-as-a-Judge evaluation." + ) + + args = parser.parse_args() + + main(frame=args.lib, version=args.version, num_workers=args.workers) diff --git a/evaluation/scripts/longmemeval/lme_metric.py b/evaluation/scripts/longmemeval/lme_metric.py new file mode 100644 index 000000000..be285123f --- /dev/null +++ b/evaluation/scripts/longmemeval/lme_metric.py @@ -0,0 +1,273 @@ +import argparse +import json + +import numpy as np +import pandas as pd + + +def save_to_excel(results, output_path): + combined_data = [] + overall_row = {"category": "overall"} + overall_row["llm_judge_score"] = results["metrics"]["llm_judge_score"] + overall_row["llm_judge_std"] = results["metrics"]["llm_judge_std"] + for metric, value in results["metrics"]["lexical"].items(): + overall_row[metric] = value + for metric, value in results["metrics"]["semantic"].items(): + overall_row[metric] = value + overall_row["context_tokens"] = results["metrics"]["context_tokens"] + for metric, value in results["metrics"]["duration"].items(): + overall_row[metric] = value + combined_data.append(overall_row) + for _, scores in results["category_scores"].items(): + category_row = {"category": scores["category_name"]} + category_row["llm_judge_score"] = scores["llm_judge_score"] + category_row["llm_judge_std"] = scores["llm_judge_std"] + for metric, value in scores["lexical"].items(): + category_row[metric] = value + for metric, value in scores["semantic"].items(): + category_row[metric] = value + category_row["context_tokens"] = scores["context_tokens"] + for metric, value in scores["duration"].items(): + category_row[metric] = value + combined_data.append(category_row) + pd.DataFrame(combined_data).to_excel(output_path, sheet_name="Metrics", index=False) + print(f"Excel file saved to: {output_path}") + + +def calculate_scores(data, grade_path, output_path): + category_scores, category_question_count = {}, {} + overall_metrics = { + "lexical": { + m: [] + for m in [ + "f1", + "rouge1_f", + "rouge2_f", + "rougeL_f", + "bleu1", + "bleu2", + "bleu3", + "bleu4", + "meteor", + ] + }, + "semantic": {m: [] for m in ["bert_f1", "similarity"]}, + "context_tokens": [], + "duration": { + m: [] for m in ["response_duration_ms", "search_duration_ms", "total_duration_ms"] + }, + } + category_metrics, user_metrics = {}, {} + all_judgment_keys = set() + judgment_run_scores = {} + + for q in data.values(): + if "llm_judgments" in q: + all_judgment_keys.update(q["llm_judgments"].keys()) + for k in all_judgment_keys: + judgment_run_scores[k] = [] + + for _, (user, q) in enumerate(data.items()): + user_metrics[user] = { + "total": 0, + "llm_judge_score": 0, + "llm_judge_std": 0, + "judgment_run_scores": {k: [] for k in all_judgment_keys}, + "lexical": {m: [] for m in overall_metrics["lexical"]}, + "semantic": {m: [] for m in overall_metrics["semantic"]}, + "context_tokens": [], + "duration": {m: [] for m in overall_metrics["duration"]}, + } + if "llm_judgments" in q: + for k, v in q["llm_judgments"].items(): + score = 1 if v else 0 + judgment_run_scores[k].append(score) + user_metrics[user]["judgment_run_scores"][k].append(score) + cat = q["category"] + if cat not in category_scores: + category_scores[cat] = { + "total": 0, + "category_name": cat, + "judgment_run_scores": {k: [] for k in all_judgment_keys}, + } + category_metrics[cat] = { + "lexical": {m: [] for m in overall_metrics["lexical"]}, + "semantic": {m: [] for m in overall_metrics["semantic"]}, + "context_tokens": [], + "duration": {m: [] for m in overall_metrics["duration"]}, + } + category_question_count[cat] = 0 + category_scores[cat]["total"] += 1 + category_question_count[cat] += 1 + if "llm_judgments" in q: + for k, v in q["llm_judgments"].items(): + score = 1 if v else 0 + category_scores[cat]["judgment_run_scores"][k].append(score) + nlp = q.get("nlp_metrics", {}) + for m in overall_metrics["lexical"]: + v = nlp.get("lexical", {}).get(m) + if v is not None: + overall_metrics["lexical"][m].append(v) + category_metrics[cat]["lexical"][m].append(v) + user_metrics[user]["lexical"][m].append(v) + for m in overall_metrics["semantic"]: + v = nlp.get("semantic", {}).get(m) + if v is not None: + overall_metrics["semantic"][m].append(v) + category_metrics[cat]["semantic"][m].append(v) + user_metrics[user]["semantic"][m].append(v) + ct = nlp.get("context_tokens") + if ct is not None: + overall_metrics["context_tokens"].append(ct) + category_metrics[cat]["context_tokens"].append(ct) + user_metrics[user]["context_tokens"].append(ct) + for m in overall_metrics["duration"]: + v = q.get(m) + if v is not None: + overall_metrics["duration"][m].append(v) + category_metrics[cat]["duration"][m].append(v) + user_metrics[user]["duration"][m].append(v) + user_metrics[user]["total"] = 1 + judgment_avgs = [ + np.mean(scores) + for scores in user_metrics[user]["judgment_run_scores"].values() + if scores + ] + user_metrics[user]["llm_judge_score"] = np.mean(judgment_avgs) if judgment_avgs else 0.0 + user_metrics[user]["llm_judge_std"] = ( + np.std(judgment_avgs) if len(judgment_avgs) > 1 else 0.0 + ) + for group in ["lexical", "semantic"]: + for m in user_metrics[user][group]: + vals = user_metrics[user][group][m] + user_metrics[user][group][m] = np.mean(vals) if vals else 0.0 + user_metrics[user]["context_tokens"] = ( + np.mean(user_metrics[user]["context_tokens"]) + if user_metrics[user]["context_tokens"] + else 0.0 + ) + for m in list(user_metrics[user]["duration"].keys()): + vals = user_metrics[user]["duration"][m] + if vals: + user_metrics[user]["duration"][m] = np.mean(vals) + user_metrics[user]["duration"][f"{m}_p50"] = np.percentile(vals, 50) + user_metrics[user]["duration"][f"{m}_p95"] = np.percentile(vals, 95) + else: + user_metrics[user]["duration"][m] = 0.0 + user_metrics[user]["duration"][f"{m}_p50"] = 0.0 + user_metrics[user]["duration"][f"{m}_p95"] = 0.0 + + judgment_run_averages = [np.mean(scores) for scores in judgment_run_scores.values() if scores] + llm_judge_score = np.mean(judgment_run_averages) if judgment_run_averages else 0.0 + llm_judge_std = np.std(judgment_run_averages) if len(judgment_run_averages) > 1 else 0.0 + + category_overall_scores = {} + for cat, score_data in category_scores.items(): + cat_judgment_avgs = [ + np.mean(scores) for scores in score_data["judgment_run_scores"].values() if scores + ] + category_overall_scores[cat] = { + "category_name": score_data["category_name"], + "llm_judge_score": np.mean(cat_judgment_avgs) if cat_judgment_avgs else 0.0, + "llm_judge_std": np.std(cat_judgment_avgs) if len(cat_judgment_avgs) > 1 else 0.0, + "total": score_data["total"], + "lexical": {}, + "semantic": {}, + "duration": {}, + "context_tokens": 0.0, + } + for group in ["lexical", "semantic"]: + for m in category_metrics[cat][group]: + vals = category_metrics[cat][group][m] + category_overall_scores[cat][group][m] = np.mean(vals) if vals else 0.0 + category_overall_scores[cat]["context_tokens"] = ( + np.mean(category_metrics[cat]["context_tokens"]) + if category_metrics[cat]["context_tokens"] + else 0.0 + ) + for m in list(category_metrics[cat]["duration"].keys()): + vals = category_metrics[cat]["duration"][m] + if vals: + category_overall_scores[cat]["duration"][m] = np.mean(vals) + category_overall_scores[cat]["duration"][f"{m}_p50"] = np.percentile(vals, 50) + category_overall_scores[cat]["duration"][f"{m}_p95"] = np.percentile(vals, 95) + else: + category_overall_scores[cat]["duration"][m] = 0.0 + category_overall_scores[cat]["duration"][f"{m}_p50"] = 0.0 + category_overall_scores[cat]["duration"][f"{m}_p95"] = 0.0 + + overall_metric_averages = { + "llm_judge_score": llm_judge_score, + "llm_judge_std": llm_judge_std, + "lexical": {}, + "semantic": {}, + "context_tokens": 0.0, + "duration": {}, + } + for group in ["lexical", "semantic"]: + for m in overall_metrics[group]: + vals = overall_metrics[group][m] + overall_metric_averages[group][m] = np.mean(vals) if vals else 0.0 + overall_metric_averages["context_tokens"] = ( + np.mean(overall_metrics["context_tokens"]) if overall_metrics["context_tokens"] else 0.0 + ) + for m in list(overall_metrics["duration"].keys()): + vals = overall_metrics["duration"][m] + if vals: + overall_metric_averages["duration"][m] = np.mean(vals) + overall_metric_averages["duration"][f"{m}_p50"] = np.percentile(vals, 50) + overall_metric_averages["duration"][f"{m}_p95"] = np.percentile(vals, 95) + else: + overall_metric_averages["duration"][m] = 0.0 + overall_metric_averages["duration"][f"{m}_p50"] = 0.0 + overall_metric_averages["duration"][f"{m}_p95"] = 0.0 + + results = { + "metrics": overall_metric_averages, + "category_scores": category_overall_scores, + "user_scores": user_metrics, + } + with open(grade_path, "w") as outfile: + json.dump(results, outfile, indent=4) + save_to_excel(results, output_path) + + print("\n" + "=" * 80) + print("📊 \033[1;36mMETRIC CALCULATION SUMMARY\033[0m".center(80)) + print("=" * 80) + total = sum(results["category_scores"][cat]["total"] for cat in results["category_scores"]) + print( + f"🤖 \033[1mLLM-as-a-Judge score:\033[0m \033[92m{results['metrics']['llm_judge_score']:.4f}\033[0m ± \033[93m{results['metrics']['llm_judge_std']:.4f}\033[0m" + ) + print(f"📋 \033[1mTotal questions evaluated:\033[0m \033[93m{total}\033[0m") + print("-" * 80) + print("⏱️ \033[1mDuration Metrics (ms):\033[0m") + for m in ["response_duration_ms", "search_duration_ms", "total_duration_ms"]: + print( + f" \033[94m{m:<22}\033[0m (avg): \033[92m{results['metrics']['duration'][m]:.2f}\033[0m" + f" | (P50): \033[96m{results['metrics']['duration'][f'{m}_p50']:.2f}\033[0m" + f" | (P95): \033[91m{results['metrics']['duration'][f'{m}_p95']:.2f}\033[0m" + ) + print("-" * 80) + print(f"📁 \033[1mResults written to:\033[0m \033[1;94m{grade_path}\033[0m") + print(f"📊 \033[1mExcel report saved to:\033[0m \033[1;94m{output_path}\033[0m") + print("=" * 80 + "\n") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser("LongMemeval Analysis Eval Metric Script") + parser.add_argument( + "--lib", + type=str, + choices=["mem0-local", "mem0-api"], + ) + parser.add_argument( + "--version", type=str, default="v1", help="Version of the evaluation framework." + ) + args = parser.parse_args() + lib, version = args.lib, args.version + judged_path = f"results/lme/{lib}-{version}/{lib}_lme_judged.json" + grade_path = f"results/lme/{lib}-{version}/{lib}_lme_grades.json" + output_path = f"results/lme/{lib}-{version}/{lib}_lme_results.xlsx" + with open(judged_path) as file: + data = json.load(file) + calculate_scores(data, grade_path, output_path) diff --git a/evaluation/scripts/longmemeval/lme_responses.py b/evaluation/scripts/longmemeval/lme_responses.py new file mode 100644 index 000000000..9d5f8c1ab --- /dev/null +++ b/evaluation/scripts/longmemeval/lme_responses.py @@ -0,0 +1,158 @@ +import argparse +import json +import os +import sys + +from concurrent.futures import ThreadPoolExecutor, as_completed +from time import time + +from dotenv import load_dotenv +from openai import OpenAI +from tqdm import tqdm + + +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from utils.prompts import ANSWER_PROMPT + + +def lme_response(llm_client, context, question, question_date): + prompt = ANSWER_PROMPT.format( + question=question, + question_date=question_date, + context=context, + ) + + response = llm_client.chat.completions.create( + model=os.getenv("CHAT_MODEL"), + messages=[ + {"role": "system", "content": prompt}, + ], + temperature=0, + ) + result = response.choices[0].message.content or "" + + return result + + +def process_qa(user_id, search_result, llm_client): + start = time() + search_result = search_result[0] + question = search_result.get("question") + question_date = search_result.get("date") + context = search_result.get("search_context", "") + anwer = lme_response(llm_client, context, question, question_date) + + response_duration_ms = (time() - start) * 1000 + + print("\n" + "-" * 80) + print(f"🤖 Processed User: \033[1m{user_id}\033[0m") + print(f"⏱️ Duration: \033[92m{response_duration_ms:.2f} ms\033[0m") + print(f"❓ Question: \033[93m{question}\033[0m") + print( + f"💬 Answer: \033[96m{anwer[:150]}...\033[0m" + if len(anwer) > 150 + else f"💬 Answer: \033[96m{anwer}\033[0m" + ) + print("-" * 80) + + return { + "user_id": user_id, + "category": search_result.get("category"), + "question": question, + "answer": anwer, + "question_date": question_date, + "golden_answer": search_result.get("golden_answer"), + "response_duration_ms": response_duration_ms, + "search_context": context, + "search_duration_ms": search_result.get("search_duration_ms"), + "answer_evidences": search_result.get("answer_evidences", []), + } + + +def main(frame, version, num_workers=4): + print("\n" + "=" * 80) + print( + f"🚀 \033[1;36mLONGMEMEVAL RESPONSE GENERATION - {frame.upper()} v{version}\033[0m".center( + 80 + ) + ) + print("=" * 80) + + load_dotenv() + + oai_client = OpenAI( + api_key=os.getenv("CHAT_MODEL_API_KEY"), base_url=os.getenv("CHAT_MODEL_BASE_URL") + ) + + print( + f"🔌 \033[1mUsing OpenAI client with model:\033[0m \033[94m{os.getenv('CHAT_MODEL')}\033[0m" + ) + + search_path = f"results/lme/{frame}-{version}/{frame}_lme_search_results.json" + response_path = f"results/lme/{frame}-{version}/{frame}_lme_responses.json" + + print(f"📂 \033[1mLoading search results from:\033[0m \033[94m{search_path}\033[0m") + with open(search_path) as file: + lme_search_results = json.load(file) + print(f"📊 \033[1mFound\033[0m \033[93m{len(lme_search_results)}\033[0m users to process") + print(f"⚙️ \033[1mUsing\033[0m \033[93m{num_workers}\033[0m worker threads") + print("-" * 80) + + lme_responses = {} + start_time = time() + + with ThreadPoolExecutor(max_workers=num_workers) as executor: + future_to_user_id = {} + + for user_id, search_results in lme_search_results.items(): + future = executor.submit(process_qa, user_id, search_results, oai_client) + future_to_user_id[future] = user_id + + for future in tqdm( + as_completed(future_to_user_id), + total=len(future_to_user_id), + desc="📝 Generating responses", + ): + user_id = future_to_user_id[future] + try: + result = future.result() + lme_responses[user_id] = result + except Exception as exc: + print(f"\033[91m❌ Error processing user {user_id}: {exc}\033[0m") + + end_time = time() + elapsed_time = end_time - start_time + elapsed_sec = int(elapsed_time) + + print("\n" + "=" * 80) + print("✅ \033[1;32mRESPONSE GENERATION COMPLETE\033[0m".center(80)) + print("=" * 80) + print(f"⏱️ \033[1mTotal time:\033[0m \033[92m{elapsed_sec // 60}m {elapsed_sec % 60}s\033[0m") + print(f"📊 \033[1mProcessed:\033[0m \033[93m{len(lme_responses)}\033[0m users") + print( + f"🔄 \033[1mFramework:\033[0m \033[94m{frame}\033[0m | \033[1mVersion:\033[0m \033[94m{version}\033[0m" + ) + + with open(response_path, "w") as f: + json.dump(lme_responses, f, indent=4) + + print(f"📁 \033[1mResponses saved to:\033[0m \033[1;94m{response_path}\033[0m") + print("=" * 80 + "\n") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="LongMemeval Response Generation Script") + parser.add_argument( + "--lib", + type=str, + choices=["mem0-local", "mem0-api"], + ) + parser.add_argument( + "--version", type=str, default="v1", help="Version of the evaluation framework." + ) + parser.add_argument( + "--workers", type=int, default=3, help="Number of runs for LLM-as-a-Judge evaluation." + ) + + args = parser.parse_args() + main(frame=args.lib, version=args.version, num_workers=args.workers) diff --git a/evaluation/scripts/longmemeval/lme_search.py b/evaluation/scripts/longmemeval/lme_search.py new file mode 100644 index 000000000..0643c07ff --- /dev/null +++ b/evaluation/scripts/longmemeval/lme_search.py @@ -0,0 +1,298 @@ +import argparse +import json +import os +import sys + + +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from collections import defaultdict +from concurrent.futures import ThreadPoolExecutor, as_completed +from datetime import datetime +from time import time + +import pandas as pd + +from tqdm import tqdm +from utils.client import mem0_client, memos_client, zep_client +from utils.memos_filters import filter_memory_data +from utils.prompts import ( + MEM0_CONTEXT_TEMPLATE, + MEM0_GRAPH_CONTEXT_TEMPLATE, + MEMOS_CONTEXT_TEMPLATE, + ZEP_CONTEXT_TEMPLATE, +) + + +def zep_search(client, user_id, query, top_k=20): + start = time() + nodes_result = client.graph.search( + query=query, + user_id=user_id, + scope="nodes", + reranker="rrf", + limit=top_k, + ) + edges_result = client.graph.search( + query=query, + user_id=user_id, + scope="edges", + reranker="cross_encoder", + limit=top_k, + ) + + nodes = nodes_result.nodes + edges = edges_result.edges + + facts = [f" - {edge.fact} (event_time: {edge.valid_at})" for edge in edges] + entities = [f" - {node.name}: {node.summary}" for node in nodes] + context = ZEP_CONTEXT_TEMPLATE.format(facts="\n".join(facts), entities="\n".join(entities)) + + duration_ms = (time() - start) * 1000 + + return context, duration_ms + + +def mem0_search(client, user_id, query, top_k=20, enable_graph=False, frame="mem0-api"): + start = time() + + if frame == "mem0-local": + results = client.search( + query=query, + user_id=user_id, + top_k=top_k, + ) + search_memories = "\n".join( + [ + f" - {item['memory']} (date: {item['metadata']['timestamp']})" + for item in results["results"] + ] + ) + search_graph = ( + "\n".join( + [ + f" - 'source': {item.get('source', '?')} -> 'target': {item.get('destination', '?')} (relationship: {item.get('relationship', '?')})" + for item in results.get("relations", []) + ] + ) + if enable_graph + else "" + ) + + elif frame == "mem0-api": + results = client.search( + query=query, + user_id=user_id, + top_k=top_k, + version="v2", + output_format="v1.1", + enable_graph=enable_graph, + filters={"AND": [{"user_id": user_id}, {"run_id": "*"}]}, + ) + search_memories = "\n".join( + [f" - {item['memory']} (date: {item['created_at']})" for item in results["results"]] + ) + search_graph = ( + "\n".join( + [ + f" - 'source': {item.get('source', '?')} -> 'target': {item.get('target', '?')} (relationship: {item.get('relationship', '?')})" + for item in results.get("relations", []) + ] + ) + if enable_graph + else "" + ) + if enable_graph: + context = MEM0_GRAPH_CONTEXT_TEMPLATE.format( + user_id=user_id, memories=search_memories, relations=search_graph + ) + else: + context = MEM0_CONTEXT_TEMPLATE.format(user_id=user_id, memories=search_memories) + duration_ms = (time() - start) * 1000 + return context, duration_ms + + +def memos_search(client, user_id, query, frame="memos-local"): + start = time() + + results = client.search( + query=query, + user_id=user_id, + ) + + search_memories = filter_memory_data(results)["text_mem"][0]["memories"] + context = MEMOS_CONTEXT_TEMPLATE.format(user_id=user_id, memories=search_memories) + + duration_ms = (time() - start) * 1000 + return context, duration_ms + + +def process_user(lme_df, conv_idx, frame, version, top_k=20): + row = lme_df.iloc[conv_idx] + question = row["question"] + sessions = row["haystack_sessions"] + question_type = row["question_type"] + question_date = row["question_date"] + answer = row["answer"] + answer_session_ids = set(row["answer_session_ids"]) + haystack_session_ids = row["haystack_session_ids"] + user_id = f"lme_exper_user_{conv_idx!s}" + id_to_session = dict(zip(haystack_session_ids, sessions, strict=False)) + answer_sessions = [id_to_session[sid] for sid in answer_session_ids if sid in id_to_session] + answer_evidences = [] + + for session in answer_sessions: + for turn in session: + if turn.get("has_answer"): + data = turn.get("role") + " : " + turn.get("content") + answer_evidences.append(data) + + search_results = defaultdict(list) + print("\n" + "-" * 80) + print(f"🔎 \033[1;36m[{conv_idx + 1}/{len(lme_df)}] Processing conversation {conv_idx}\033[0m") + print(f"❓ Question: \033[93m{question}\033[0m") + print(f"📅 Date: \033[92m{question_date}\033[0m") + print(f"🏷️ Type: \033[94m{question_type}\033[0m") + print("-" * 80) + + existing_results, exists = load_existing_results(frame, version, conv_idx) + if exists: + print(f"♻️ \033[93mUsing existing results for conversation {conv_idx}\033[0m") + return existing_results + + if frame == "zep": + client = zep_client() + print("🔌 \033[1mUsing \033[94mZep client\033[0m \033[1mfor search...\033[0m") + context, duration_ms = zep_search(client, user_id, question) + + elif frame == "mem0-local": + client = mem0_client(mode="local") + print("🔌 \033[1mUsing \033[94mMem0 Local client\033[0m \033[1mfor search...\033[0m") + context, duration_ms = mem0_search(client, user_id, question, top_k=top_k, frame=frame) + elif frame == "mem0-api": + client = mem0_client(mode="api") + print("🔌 \033[1mUsing \033[94mMem0 API client\033[0m \033[1mfor search...\033[0m") + context, duration_ms = mem0_search(client, user_id, question, top_k=top_k, frame=frame) + elif frame == "memos-local": + client = memos_client( + mode="local", + db_name=f"lme_{frame}-{version}-{user_id.replace('_', '')}", + user_id=user_id, + top_k=20, + mem_cube_path=f"results/lme/{frame}-{version}/storages/{user_id}", + mem_cube_config_path="configs/mem_cube_config.json", + mem_os_config_path="configs/mos_memos_config.json", + addorsearch="search", + ) + print("🔌 \033[1mUsing \033[94mMemos Local client\033[0m \033[1mfor search...\033[0m") + context, duration_ms = memos_search(client, user_id, question, frame=frame) + + search_results[user_id].append( + { + "question": question, + "category": question_type, + "date": question_date, + "golden_answer": answer, + "answer_evidences": answer_evidences, + "search_context": context, + "search_duration_ms": duration_ms, + } + ) + + os.makedirs(f"results/lme/{frame}-{version}/tmp", exist_ok=True) + with open( + f"results/lme/{frame}-{version}/tmp/{frame}_lme_search_results_{conv_idx}.json", "w" + ) as f: + json.dump(search_results, f, indent=4) + print(f"💾 \033[92mSearch results for conversation {conv_idx} saved...\033[0m") + print("-" * 80) + + return search_results + + +def load_existing_results(frame, version, group_idx): + result_path = ( + f"results/locomo/{frame}-{version}/tmp/{frame}_locomo_search_results_{group_idx}.json" + ) + if os.path.exists(result_path): + try: + with open(result_path) as f: + return json.load(f), True + except Exception as e: + print(f"\033[91m❌ Error loading existing results for group {group_idx}: {e}\033[0m") + return {}, False + + +def main(frame, version, top_k=20, num_workers=2): + print("\n" + "=" * 80) + print(f"🔍 \033[1;36mLONGMEMEVAL SEARCH - {frame.upper()} v{version}\033[0m".center(80)) + print("=" * 80) + + lme_df = pd.read_json("data/longmemeval/longmemeval_s.json") + print( + "📚 \033[1mLoaded LongMemeval dataset\033[0m from \033[94mdata/longmemeval/longmemeval_s.json\033[0m" + ) + num_multi_sessions = len(lme_df) + print(f"👥 Number of users: \033[93m{num_multi_sessions}\033[0m") + print( + f"⚙️ Search parameters: top_k=\033[94m{top_k}\033[0m, workers=\033[94m{num_workers}\033[0m" + ) + print("-" * 80) + + all_search_results = defaultdict(list) + start_time = datetime.now() + + with ThreadPoolExecutor(max_workers=num_workers) as executor: + future_to_idx = { + executor.submit(process_user, lme_df, idx, frame, version, top_k): idx + for idx in range(num_multi_sessions) + } + + for future in tqdm( + as_completed(future_to_idx), total=num_multi_sessions, desc="📊 Processing users" + ): + idx = future_to_idx[future] + try: + search_results = future.result() + for user_id, results in search_results.items(): + all_search_results[user_id].extend(results) + except Exception as e: + print(f"\033[91m❌ Error processing user {idx}: {e}\033[0m") + + end_time = datetime.now() + elapsed_time = end_time - start_time + elapsed_time_str = str(elapsed_time).split(".")[0] + + print("\n" + "=" * 80) + print("✅ \033[1;32mSEARCH COMPLETE\033[0m".center(80)) + print("=" * 80) + print( + f"⏱️ Total time taken to search \033[93m{num_multi_sessions}\033[0m users: \033[92m{elapsed_time_str}\033[0m" + ) + print( + f"🔄 Framework: \033[94m{frame}\033[0m | Version: \033[94m{version}\033[0m | Workers: \033[94m{num_workers}\033[0m" + ) + + with open(f"results/lme/{frame}-{version}/{frame}_lme_search_results.json", "w") as f: + json.dump(dict(all_search_results), f, indent=4) + print( + f"📁 Results saved to: \033[1;94mresults/lme/{frame}-{version}/{frame}_lme_search_results.json\033[0m" + ) + print("=" * 80 + "\n") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="LongMemeval Search Script") + parser.add_argument("--lib", type=str, choices=["mem0-local", "mem0-api", "memos-local"]) + parser.add_argument( + "--version", type=str, default="v1", help="Version of the evaluation framework." + ) + parser.add_argument( + "--top_k", type=int, default=20, help="Number of top results to retrieve from the search." + ) + parser.add_argument( + "--workers", type=int, default=3, help="Number of runs for LLM-as-a-Judge evaluation." + ) + + args = parser.parse_args() + + main(frame=args.lib, version=args.version, top_k=args.top_k, num_workers=args.workers) diff --git a/evaluation/scripts/run_lme_eval.sh b/evaluation/scripts/run_lme_eval.sh new file mode 100755 index 000000000..96e430fd6 --- /dev/null +++ b/evaluation/scripts/run_lme_eval.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# Common parameters for all scripts +LIB="memos-local" +VERSION="071503" +WORKERS=10 +TOPK=20 + +echo "Running lme_ingestion.py..." +CUDA_VISIBLE_DEVICES=0 python scripts/longmemeval/lme_ingestion.py --lib $LIB --version $VERSION --workers $WORKERS +if [ $? -ne 0 ]; then + echo "Error running lme_ingestion.py" + exit 1 +fi + +echo "Running lme_search.py..." +CUDA_VISIBLE_DEVICES=0 python scripts/longmemeval/lme_search.py --lib $LIB --version $VERSION --top_k $TOPK --workers $WORKERS +if [ $? -ne 0 ]; then + echo "Error running lme_search.py" + exit 1 +fi + +echo "All scripts completed successfully!" diff --git a/evaluation/scripts/utils/__init__.py b/evaluation/scripts/utils/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/evaluation/scripts/utils/client.py b/evaluation/scripts/utils/client.py new file mode 100644 index 000000000..ddb144f6f --- /dev/null +++ b/evaluation/scripts/utils/client.py @@ -0,0 +1,198 @@ +import json +import os +import sys + +from dotenv import load_dotenv +from mem0 import MemoryClient +from zep_cloud.client import Zep +from zep_cloud.types import Message + + +sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from memos.configs.mem_cube import GeneralMemCubeConfig +from memos.configs.mem_os import MOSConfig +from memos.mem_cube.general import GeneralMemCube +from memos.mem_os.main import MOS +from utils.mem0_local import Mem0Client +from utils.memos_filters import filter_memory_data + + +load_dotenv() + + +def zep_client(): + """Initialize and return a Zep client instance.""" + api_key = os.getenv("ZEP_API_KEY") + zep = Zep(api_key=api_key) + + return zep + + +def mem0_client(mode="local"): + """Initialize and return a Mem0 client instance.""" + if mode == "local": + base_url = "http://localhost:9999" + mem0 = Mem0Client(base_url=base_url) + elif mode == "api": + mem0 = MemoryClient(api_key=os.getenv("MEM0_API_KEY")) + else: + raise ValueError("Invalid mode. Choose 'local' or 'cloud'.") + + return mem0 + + +def memos_client( + mode="local", + db_name=None, + user_id=None, + top_k=20, + mem_cube_path="./", + mem_cube_config_path="configs/lme_mem_cube_config.json", + mem_os_config_path="configs/mos_memos_config.json", + addorsearch="add", +): + """Initialize and return a Memos client instance.""" + if mode == "local": + with open(mem_os_config_path) as f: + mos_config_data = json.load(f) + mos_config_data["top_k"] = top_k + mos_config = MOSConfig(**mos_config_data) + memos = MOS(mos_config) + memos.create_user(user_id=user_id) + + if addorsearch == "add": + with open(mem_cube_config_path) as f: + mem_cube_config_data = json.load(f) + mem_cube_config_data["user_id"] = user_id + mem_cube_config_data["cube_id"] = user_id + mem_cube_config_data["text_mem"]["config"]["graph_db"]["config"]["db_name"] = ( + f"{db_name.replace('_', '')}" + ) + mem_cube_config = GeneralMemCubeConfig.model_validate(mem_cube_config_data) + mem_cube = GeneralMemCube(mem_cube_config) + + if not os.path.exists(mem_cube_path): + mem_cube.dump(mem_cube_path) + + memos.register_mem_cube( + mem_cube_name_or_path=mem_cube_path, + mem_cube_id=user_id, + user_id=user_id, + ) + + elif mode == "api": + pass + + return memos + + +if __name__ == "__main__": + # Example usage of the Zep client + zep = zep_client() + print("Zep client initialized successfully.") + + # Example of adding a session and a message to Zep memory + user_id = "user123" + session_id = "session123" + + zep.memory.add_session( + session_id=session_id, + user_id=user_id, + ) + + messages = [ + Message( + role="Jane", + role_type="user", + content="Who was Octavia Butler?", + ) + ] + new_episode = zep.memory.add( + session_id=session_id, + messages=messages, + ) + print("New episode added:", new_episode) + + # Example of searching for nodes and edges in Zep memory + nodes_result = zep.graph.search( + query="Octavia Butler", + user_id="user123", + scope="nodes", + reranker="rrf", + limit=10, + ).nodes + + edges_result = zep.graph.search( + query="Octavia Butler", + user_id="user123", + scope="edges", + reranker="cross_encoder", + limit=10, + ).edges + + print("Nodes found:", nodes_result) + print("Edges found:", edges_result) + + # Example usage of the Mem0 client + mem0 = mem0_client(mode="local") + print("Mem0 client initialized successfully.") + print("Adding memories...") + result = mem0.add( + messages=[ + {"role": "user", "content": "I like drinking coffee in the morning"}, + {"role": "user", "content": "I enjoy reading books at night"}, + ], + user_id="alice", + ) + print("Memory added:", result) + + print("Searching memories...") + search_result = mem0.search(query="coffee", user_id="alice", top_k=2) + print("Search results:", search_result) + + # Example usage of the Memos client + memos_a = memos_client( + mode="local", + db_name="session333", + user_id="dlice", + top_k=20, + mem_cube_path="./mem_cube_a", + mem_cube_config_path="configs/lme_mem_cube_config.json", + mem_os_config_path="configs/mos_memos_config.json", + ) + print("Memos a client initialized successfully.") + memos_b = memos_client( + mode="local", + db_name="session444", + user_id="alice", + top_k=20, + mem_cube_path="./mem_cube_b", + mem_cube_config_path="configs/lme_mem_cube_config.json", + mem_os_config_path="configs/mos_memos_config.json", + ) + print("Memos b client initialized successfully.") + + # Example of adding memories in Memos + memos_a.add( + messages=[ + {"role": "user", "content": "I like drinking coffee in the morning"}, + {"role": "user", "content": "I enjoy reading books at night"}, + ], + user_id="dlice", + ) + memos_b.add( + messages=[ + {"role": "user", "content": "I like playing football in the evening"}, + {"role": "user", "content": "I enjoy watching movies at night"}, + ], + user_id="alice", + ) + + # Example of searching memories in Memos + search_result_a = memos_a.search(query="coffee", user_id="dlice") + filtered_search_result_a = filter_memory_data(search_result_a)["text_mem"][0]["memories"] + print("Search results in Memos A:", filtered_search_result_a) + + search_result_b = memos_b.search(query="football", user_id="alice") + filtered_search_result_b = filter_memory_data(search_result_b)["text_mem"][0]["memories"] + print("Search results in Memos B:", filtered_search_result_b) diff --git a/evaluation/scripts/utils/mem0_local.py b/evaluation/scripts/utils/mem0_local.py new file mode 100644 index 000000000..62b9d905b --- /dev/null +++ b/evaluation/scripts/utils/mem0_local.py @@ -0,0 +1,191 @@ +from typing import Any + +import requests + + +class Mem0Client: + def __init__(self, base_url: str = "http://localhost:8000"): + self.base_url = base_url + + def add( + self, + messages: list[dict], + timestamp: str | None = None, + user_id: str | None = None, + agent_id: str | None = None, + run_id: str | None = None, + metadata: dict[str, Any] | None = None, + ): + """Create memories.""" + url = f"{self.base_url}/memories" + + if metadata is None: + metadata = {} + + if user_id is None and agent_id is None and run_id is None: + raise ValueError("At least one of user_id, agent_id, or run_id must be provided.") + + if user_id: + metadata["user_id"] = user_id + if agent_id: + metadata["agent_id"] = agent_id + if run_id: + metadata["run_id"] = run_id + + metadata["timestamp"] = timestamp + + data = { + "messages": messages, + "user_id": user_id, + "agent_id": agent_id, + "run_id": run_id, + "metadata": metadata, + } + + response = requests.post(url, json=data) + response.raise_for_status() + return response.json() + + def search( + self, + query: str, + user_id: str | None = None, + agent_id: str | None = None, + run_id: str | None = None, + filters: dict[str, Any] | None = None, + top_k: int = 10, + ): + """Search memories.""" + url = f"{self.base_url}/search" + + if filters is None: + filters = {} + + data = { + "query": query, + "user_id": user_id, + "agent_id": agent_id, + "run_id": run_id, + "filters": filters, + } + + response = requests.post(url, json=data) + response.raise_for_status() + + results = response.json().get("results", []) + top_k_results = results[:top_k] if len(results) > top_k else results + + relations = response.json().get("relations", []) + top_k_relations = relations[:top_k] if len(relations) > top_k else relations + + return {"results": top_k_results, "relations": top_k_relations} + + def get_all( + self, user_id: str | None = None, agent_id: str | None = None, run_id: str | None = None + ): + """Retrieve all memories.""" + url = f"{self.base_url}/memories" + + params = {} + if user_id: + params["user_id"] = user_id + if agent_id: + params["agent_id"] = agent_id + if run_id: + params["run_id"] = run_id + + response = requests.get(url, params=params) + response.raise_for_status() + return response.json() + + def get(self, memory_id: str): + """Retrieve a specific memory by ID.""" + url = f"{self.base_url}/memories/{memory_id}" + + response = requests.get(url) + response.raise_for_status() + return response.json() + + def delete(self, memory_id: str): + """Delete a specific memory by ID.""" + url = f"{self.base_url}/memories/{memory_id}" + + response = requests.delete(url) + response.raise_for_status() + return response.json() + + def delete_all( + self, user_id: str | None = None, agent_id: str | None = None, run_id: str | None = None + ): + """Delete all memories for a user, agent, or run.""" + url = f"{self.base_url}/memories" + + params = {} + if user_id: + params["user_id"] = user_id + if agent_id: + params["agent_id"] = agent_id + if run_id: + params["run_id"] = run_id + + response = requests.delete(url, params=params) + response.raise_for_status() + return response.json() + + def reset(self): + """Reset the memory store.""" + url = f"{self.base_url}/reset" + + response = requests.post(url) + response.raise_for_status() + return response.json() + + +if __name__ == "__main__": + client = Mem0Client(base_url="http://localhost:9999") + + # Example usage + print("Adding memories...") + add_result_a = client.add( + messages=[{"role": "user", "content": "I like drinking coffee in the morning"}], + user_id="alice", + ) + print(add_result_a) + + add_result_b = client.add( + messages=[{"role": "user", "content": "I enjoy reading books in the evening"}], + user_id="alice", + ) + print(add_result_b) + + print("\nSearching memories...") + search_result = client.search( + query="When did Melanie paint a sunrise?", user_id="alice", top_k=10 + ) + print(search_result) + print(len(search_result.get("results", []))) + + print("\nRetrieving all memories...") + all_memories = client.get_all(user_id="alice") + print(all_memories) + print(len(all_memories.get("results", []))) + + print("\nRetrieving a specific memory...") + if all_memories and "results" in all_memories and len(all_memories["results"]) > 0: + memory_id = all_memories["results"][0]["id"] + specific_memory = client.get(memory_id) + print(specific_memory) + + print("\nDeleting a specific memory...") + if all_memories and "results" in all_memories and len(all_memories["results"]) > 0: + memory_id = all_memories["results"][0]["id"] + delete_result = client.delete(memory_id) + print(delete_result) + + print("\nDeleting all memories for user 'alice'...") + delete_all_result = client.delete_all(user_id="alice") + print(delete_all_result) + + print("\nResetting the memory store...") + reset_result = client.reset() + print(reset_result) diff --git a/evaluation/scripts/utils/memos_filters.py b/evaluation/scripts/utils/memos_filters.py new file mode 100644 index 000000000..815f31436 --- /dev/null +++ b/evaluation/scripts/utils/memos_filters.py @@ -0,0 +1,51 @@ +def filter_memory_data(memories_data): + filtered_data = {} + for key, value in memories_data.items(): + if key == "text_mem": + filtered_data[key] = [] + for mem_group in value: + # Check if it's the new data structure (list of TextualMemoryItem objects) + if "memories" in mem_group and isinstance(mem_group["memories"], list): + # New data structure: directly a list of TextualMemoryItem objects + filtered_memories = [] + for memory_item in mem_group["memories"]: + # Create filtered dictionary + filtered_item = { + "id": memory_item.id, + "memory": memory_item.memory, + "metadata": {}, + } + # Filter metadata, excluding embedding + if hasattr(memory_item, "metadata") and memory_item.metadata: + for attr_name in dir(memory_item.metadata): + if not attr_name.startswith("_") and attr_name != "embedding": + attr_value = getattr(memory_item.metadata, attr_name) + if not callable(attr_value): + filtered_item["metadata"][attr_name] = attr_value + filtered_memories.append(filtered_item) + + filtered_group = { + "cube_id": mem_group.get("cube_id", ""), + "memories": filtered_memories, + } + filtered_data[key].append(filtered_group) + else: + # Old data structure: dictionary with nodes and edges + filtered_group = { + "memories": {"nodes": [], "edges": mem_group["memories"].get("edges", [])} + } + for node in mem_group["memories"].get("nodes", []): + filtered_node = { + "id": node.get("id"), + "memory": node.get("memory"), + "metadata": { + k: v + for k, v in node.get("metadata", {}).items() + if k != "embedding" + }, + } + filtered_group["memories"]["nodes"].append(filtered_node) + filtered_data[key].append(filtered_group) + else: + filtered_data[key] = value + return filtered_data diff --git a/evaluation/scripts/utils/prompts.py b/evaluation/scripts/utils/prompts.py new file mode 100644 index 000000000..6515619ec --- /dev/null +++ b/evaluation/scripts/utils/prompts.py @@ -0,0 +1,96 @@ +ANSWER_PROMPT = """ + You are an intelligent memory assistant tasked with retrieving accurate information from conversation memories. + + # CONTEXT: + You have access to memories from a conversation. These memories contain timestamped information that may be relevant to answering the question. + + # INSTRUCTIONS: + 1. Carefully analyze all provided memories. + 2. Pay special attention to the timestamps to determine the answer. + 3. If the question asks about a specific event or fact, look for direct evidence in the memories. + 4. The answer must be brief (under 10 words) and direct, with no extra description. + + # APPROACH (Think step by step): + 1. First, examine all memories that contain information related to the question. + 2. Examine the timestamps and content of these memories carefully. + 3. Look for explicit mentions of dates, times, locations, or events that answer the question. + 4. If the answer requires calculation (e.g., converting relative time references), show your work. + 5. Formulate a precise, concise answer based solely on the evidence in the memories. + 6. Double-check that your answer directly addresses the question asked. + 7. Ensure your final answer is specific and avoids vague time references. + + {context} + + Current Date: {question_date} + + Question: {question} + + Answer: + """ + +ZEP_CONTEXT_TEMPLATE = """ + FACTS and ENTITIES represent relevant context to the current conversation. + + # These are the most relevant facts for the conversation along with the datetime of the event that the fact refers to. + If a fact mentions something happening a week ago, then the datetime will be the date time of last week and not the datetime + of when the fact was stated. + Timestamps in memories represent the actual time the event occurred, not the time the event was mentioned in a message. + + + {facts} + + + # These are the most relevant entities + # ENTITY_NAME: entity summary + + {entities} + +""" + +MEM0_CONTEXT_TEMPLATE = """ + Memories for user {user_id}: + + {memories} +""" + +MEM0_GRAPH_CONTEXT_TEMPLATE = """ + Memories for user {user_id}: + + {memories} + + Relations: + + {relations} +""" + +MEMOS_CONTEXT_TEMPLATE = """ + Memories for user {user_id}: + + {memories} +""" + +LME_JUDGE_MODEL_TEMPLATE = """ + Your task is to label an answer to a question as ’CORRECT’ or ’WRONG’. You will be given the following data: + (1) a question (posed by one user to another user), + (2) a ’gold’ (ground truth) answer, + (3) a generated answer + which you will score as CORRECT/WRONG. + + The point of the question is to ask about something one user should know about the other user based on their prior conversations. + The gold answer will usually be a concise and short answer that includes the referenced topic, for example: + Question: Where did I buy my new tennis racket from? + Gold answer: the sports store downtown + The generated answer might be much longer, but you should be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT. + + For time related questions, the gold answer will be a specific date, month, year, etc. The generated answer might be much longer or use relative time references (like "last Tuesday" or "next month"), but you should be generous with your grading - as long as it refers to the same date or time period as the gold answer, it should be counted as CORRECT. Even if the format differs (e.g., "May 7th" vs "7 May"), consider it CORRECT if it's the same date. + + Now it’s time for the real question: + Question: {question} + Gold answer: {golden_answer} + Generated answer: {response} + + First, provide a short (one sentence) explanation of your reasoning, then finish with CORRECT or WRONG. + Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script. + + Just return the label CORRECT or WRONG in a json format with the key as "label". + """ diff --git a/examples/basic_modules/embedder.py b/examples/basic_modules/embedder.py index 2053c4a06..7cc7942da 100644 --- a/examples/basic_modules/embedder.py +++ b/examples/basic_modules/embedder.py @@ -46,3 +46,42 @@ text_hf = "This is a sample text for Hugging Face embedding generation." embedding_hf = embedder_hf.embed([text_hf]) print("Scenario 3 HF embedding shape:", len(embedding_hf[0])) +print("==" * 20) + +# === Scenario 4: Using UniversalAPIEmbedder(OpenAI) === + +config_api = EmbedderConfigFactory.model_validate( + { + "backend": "universal_api", + "config": { + "provider": "openai", + "api_key": "", + "model_name_or_path": "text-embedding-3-large", + "base_url": "https://api.myproxy.com/v1", + }, + } +) +embedder_api = EmbedderFactory.from_config(config_api) +text_api = "This is a sample text for embedding generation using OpenAI API." +embedding_api = embedder_api.embed([text_api]) +print("Scenario 4: OpenAI API embedding vector length:", len(embedding_api[0])) +print("Embedding preview:", embedding_api[0][:10]) + +# === Scenario 5: Using UniversalAPIEmbedder(Azure) === + +config_api = EmbedderConfigFactory.model_validate( + { + "backend": "universal_api", + "config": { + "provider": "azure", + "api_key": "", + "model_name_or_path": "text-embedding-3-large", + "base_url": "https://open.azure.com/openapi/online/v2/", + }, + } +) +embedder_api = EmbedderFactory.from_config(config_api) +text_api = "This is a sample text for embedding generation using Azure API." +embedding_api = embedder_api.embed([text_api]) +print("Scenario 5: Azure API embedding vector length:", len(embedding_api[0])) +print("Embedding preview:", embedding_api[0][:10]) diff --git a/examples/basic_modules/llm.py b/examples/basic_modules/llm.py index d5fb16082..d33fc9544 100644 --- a/examples/basic_modules/llm.py +++ b/examples/basic_modules/llm.py @@ -69,6 +69,11 @@ print("Scenario 3:", response) print("==" * 20) +print("Scenario 3:\n") +for chunk in llm.generate_stream(messages): + print(chunk, end="") +print("==" * 20) + # Scenario 4: Using LLMFactory with Huggingface Models @@ -91,3 +96,89 @@ response = llm.generate(messages) print("Scenario 4:", response) print("==" * 20) + + +# Scenario 5: Using LLMFactory with Qwen (DashScope Compatible API) +# Note: +# This example works for any model that supports the OpenAI-compatible Chat Completion API, +# including but not limited to: +# - Qwen models: qwen-plus, qwen-max-2025-01-25 +# - DeepSeek models: deepseek-chat, deepseek-coder, deepseek-v3 +# - Other compatible providers: MiniMax, Fireworks, Groq, OpenRouter, etc. +# +# Just set the correct `api_key`, `api_base`, and `model_name_or_path`. + +config = LLMConfigFactory.model_validate( + { + "backend": "qwen", + "config": { + "model_name_or_path": "qwen-plus", # or qwen-max-2025-01-25 + "temperature": 0.7, + "max_tokens": 1024, + "top_p": 0.9, + "top_k": 50, + "api_key": "sk-xxx", + "api_base": "https://dashscope.aliyuncs.com/compatible-mode/v1", + }, + } +) +llm = LLMFactory.from_config(config) +messages = [ + {"role": "user", "content": "Hello, who are you"}, +] +response = llm.generate(messages) +print("Scenario 5:", response) +print("==" * 20) + +print("Scenario 5:\n") +for chunk in llm.generate_stream(messages): + print(chunk, end="") +print("==" * 20) + +# Scenario 6: Using LLMFactory with Deepseek-chat + +cfg = LLMConfigFactory.model_validate( + { + "backend": "deepseek", + "config": { + "model_name_or_path": "deepseek-chat", + "api_key": "sk-xxx", + "api_base": "https://api.deepseek.com", + "temperature": 0.6, + "max_tokens": 512, + "remove_think_prefix": False, + }, + } +) +llm = LLMFactory.from_config(cfg) +messages = [{"role": "user", "content": "Hello, who are you"}] +resp = llm.generate(messages) +print("Scenario 6:", resp) + + +# Scenario 7: Using LLMFactory with Deepseek-chat + reasoning + CoT + streaming + +cfg2 = LLMConfigFactory.model_validate( + { + "backend": "deepseek", + "config": { + "model_name_or_path": "deepseek-reasoner", + "api_key": "sk-xxx", + "api_base": "https://api.deepseek.com", + "temperature": 0.2, + "max_tokens": 1024, + "remove_think_prefix": False, + }, + } +) +llm = LLMFactory.from_config(cfg2) +messages = [ + { + "role": "user", + "content": "Explain how to solve this problem step-by-step. Be explicit in your thinking process. Question: If a train travels from city A to city B at 60 mph and returns at 40 mph, what is its average speed for the entire trip? Let's think step by step.", + }, +] +print("Scenario 7:\n") +for chunk in llm.generate_stream(messages): + print(chunk, end="") +print("==" * 20) diff --git a/examples/basic_modules/neo4j_example.py b/examples/basic_modules/neo4j_example.py index 93b317264..082ad8c3e 100644 --- a/examples/basic_modules/neo4j_example.py +++ b/examples/basic_modules/neo4j_example.py @@ -8,12 +8,7 @@ embedder_config = EmbedderConfigFactory.model_validate( - { - "backend": "sentence_transformer", - "config": { - "model_name_or_path": "nomic-ai/nomic-embed-text-v1.5", - }, - } + {"backend": "ollama", "config": {"model_name_or_path": "nomic-embed-text:latest"}} ) embedder = EmbedderFactory.from_config(embedder_config) @@ -22,7 +17,7 @@ def embed_memory_item(memory: str) -> list[float]: return embedder.embed([memory])[0] -def example_1_paper(db_name: str = "paper"): +def example_multi_db(db_name: str = "paper"): # Step 1: Build factory config config = GraphDBConfigFactory( backend="neo4j", @@ -33,6 +28,7 @@ def example_1_paper(db_name: str = "paper"): "db_name": db_name, "auto_create": True, "embedding_dimension": 768, + "use_multi_db": True, }, ) @@ -46,7 +42,6 @@ def example_1_paper(db_name: str = "paper"): metadata=TreeNodeTextualMemoryMetadata( memory_type="LongTermMemory", key="Multi-UAV Long-Term Coverage", - value="Research topic on distributed multi-agent UAV navigation and coverage", hierarchy_level="topic", type="fact", memory_time="2024-01-01", @@ -68,7 +63,7 @@ def example_1_paper(db_name: str = "paper"): ) graph.add_node( - id=topic.id, content=topic.memory, metadata=topic.metadata.model_dump(exclude_none=True) + id=topic.id, memory=topic.memory, metadata=topic.metadata.model_dump(exclude_none=True) ) # Step 4: Define and write concept nodes @@ -78,7 +73,6 @@ def example_1_paper(db_name: str = "paper"): metadata=TreeNodeTextualMemoryMetadata( memory_type="LongTermMemory", key="Reward Function Design", - value="Combines coverage, energy efficiency, and overlap penalty", hierarchy_level="concept", type="fact", memory_time="2024-01-01", @@ -103,7 +97,6 @@ def example_1_paper(db_name: str = "paper"): metadata=TreeNodeTextualMemoryMetadata( memory_type="LongTermMemory", key="Energy Model", - value="Includes communication and motion energy consumption", hierarchy_level="concept", type="fact", memory_time="2024-01-01", @@ -126,7 +119,6 @@ def example_1_paper(db_name: str = "paper"): metadata=TreeNodeTextualMemoryMetadata( memory_type="LongTermMemory", key="Coverage Metrics", - value="CT and FT used for long-term area and fairness evaluation", hierarchy_level="concept", type="fact", memory_time="2024-01-01", @@ -150,7 +142,7 @@ def example_1_paper(db_name: str = "paper"): for concept in concepts: graph.add_node( id=concept.id, - content=concept.memory, + memory=concept.memory, metadata=concept.metadata.model_dump(exclude_none=True), ) graph.add_edge(source_id=concept.id, target_id=topic.id, type="RELATED") @@ -165,7 +157,6 @@ def example_1_paper(db_name: str = "paper"): metadata=TreeNodeTextualMemoryMetadata( memory_type="WorkingMemory", key="Reward Components", - value="Coverage gain, energy usage penalty, overlap penalty", hierarchy_level="fact", type="fact", memory_time="2024-01-01", @@ -190,7 +181,6 @@ def example_1_paper(db_name: str = "paper"): metadata=TreeNodeTextualMemoryMetadata( memory_type="LongTermMemory", key="Energy Cost Components", - value="Includes movement and communication energy", hierarchy_level="fact", type="fact", memory_time="2024-01-01", @@ -215,7 +205,6 @@ def example_1_paper(db_name: str = "paper"): metadata=TreeNodeTextualMemoryMetadata( memory_type="LongTermMemory", key="CT and FT Definition", - value="CT: total coverage duration; FT: fairness index", hierarchy_level="fact", type="fact", memory_time="2024-01-01", @@ -259,113 +248,294 @@ def example_1_paper(db_name: str = "paper"): print(graph.get_node(node_i["id"])) -def example_2_travel(db_name: str = "travel"): - # Step 1: Build factory config - config = GraphDBConfigFactory( +def example_shared_db(db_name: str = "shared-traval-group"): + """ + Example: Single(Shared)-DB multi-tenant (logical isolation) + Multiple users' data in the same Neo4j DB with user_name as a tag. + """ + # users + user_list = ["travel_member_alice", "travel_member_bob"] + + for user_name in user_list: + # Step 1: Build factory config + config = GraphDBConfigFactory( + backend="neo4j", + config={ + "uri": "bolt://localhost:7687", + "user": "neo4j", + "password": "12345678", + "db_name": db_name, + "user_name": user_name, + "use_multi_db": False, + "auto_create": True, + "embedding_dimension": 768, + }, + ) + # Step 2: Instantiate graph store + graph = GraphStoreFactory.from_config(config) + print(f"\n[INFO] Working in shared DB: {db_name}, for user: {user_name}") + graph.clear() + + # Step 3: Create topic node + topic = TextualMemoryItem( + memory=f"Travel notes for {user_name}", + metadata=TreeNodeTextualMemoryMetadata( + memory_type="LongTermMemory", + hierarchy_level="topic", + status="activated", + visibility="public", + embedding=embed_memory_item(f"Travel notes for {user_name}"), + ), + ) + + graph.add_node( + id=topic.id, memory=topic.memory, metadata=topic.metadata.model_dump(exclude_none=True) + ) + + # Step 4: Add a concept for each user + concept = TextualMemoryItem( + memory=f"Itinerary plan for {user_name}", + metadata=TreeNodeTextualMemoryMetadata( + memory_type="LongTermMemory", + hierarchy_level="concept", + status="activated", + visibility="public", + embedding=embed_memory_item(f"Itinerary plan for {user_name}"), + ), + ) + + graph.add_node( + id=concept.id, + memory=concept.memory, + metadata=concept.metadata.model_dump(exclude_none=True), + ) + + # Link concept to topic + graph.add_edge(source_id=concept.id, target_id=topic.id, type="INCLUDE") + + print(f"[INFO] Added nodes for {user_name}") + + # Step 5: Query and print ALL for verification + print("\n=== Export entire DB (for verification, includes ALL users) ===") + graph = GraphStoreFactory.from_config(config) + all_graph_data = graph.export_graph() + print(all_graph_data) + + # Step 6: Search for alice's data only + print("\n=== Search for travel_member_alice ===") + config_alice = GraphDBConfigFactory( backend="neo4j", config={ "uri": "bolt://localhost:7687", "user": "neo4j", "password": "12345678", "db_name": db_name, - "auto_create": True, + "user_name": user_list[0], "embedding_dimension": 768, }, ) - - # Step 2: Instantiate the graph store + graph_alice = GraphStoreFactory.from_config(config_alice) + nodes = graph_alice.search_by_embedding(vector=embed_memory_item("travel itinerary"), top_k=1) + for node in nodes: + print(graph_alice.get_node(node["id"])) + + +def run_user_session( + user_name: str, + db_name: str, + topic_text: str, + concept_texts: list[str], + fact_texts: list[str], + community: bool = False, +): + print(f"\n=== {user_name} starts building their memory graph ===") + + # Manually initialize correct GraphDB class + if community: + config = GraphDBConfigFactory( + backend="neo4j-community", + config={ + "uri": "bolt://localhost:7687", + "user": "neo4j", + "password": "12345678", + "db_name": db_name, + "user_name": user_name, + "use_multi_db": False, + "auto_create": False, # Neo4j Community does not allow auto DB creation + "embedding_dimension": 768, + "vec_config": { + # Pass nested config to initialize external vector DB + # If you use qdrant, please use Server instead of local mode. + "backend": "qdrant", + "config": { + "collection_name": "neo4j_vec_db", + "vector_dimension": 768, + "distance_metric": "cosine", + "host": "localhost", + "port": 6333, + }, + }, + }, + ) + else: + config = GraphDBConfigFactory( + backend="neo4j", + config={ + "uri": "bolt://localhost:7687", + "user": "neo4j", + "password": "12345678", + "db_name": db_name, + "user_name": user_name, + "use_multi_db": False, + "auto_create": True, + "embedding_dimension": 768, + }, + ) graph = GraphStoreFactory.from_config(config) + + # Start with a clean slate for this user graph.clear() - # Step 3: Create topic node + now = datetime.utcnow().isoformat() + + # === Step 1: Create a root topic node (e.g., user's research focus) === topic = TextualMemoryItem( - memory="Travel", + memory=topic_text, metadata=TreeNodeTextualMemoryMetadata( memory_type="LongTermMemory", + key="Research Topic", hierarchy_level="topic", - status="activated", - visibility="public", - embedding=embed_memory_item("Travel"), - ), - ) - - graph.add_node( - id=topic.id, content=topic.memory, metadata=topic.metadata.model_dump(exclude_none=True) - ) - - concept1 = TextualMemoryItem( - memory="Travel in Italy", - metadata=TreeNodeTextualMemoryMetadata( - memory_type="LongTermMemory", - hierarchy_level="concept", - status="activated", - visibility="public", - embedding=embed_memory_item("Travel in Italy"), - ), - ) - - graph.add_node( - id=concept1.id, - content=concept1.memory, - metadata=concept1.metadata.model_dump(exclude_none=True), - ) - graph.add_edge(source_id=topic.id, target_id=concept1.id, type="INCLUDE") - - concept2 = TextualMemoryItem( - memory="Traval plan", - metadata=TreeNodeTextualMemoryMetadata( - memory_type="LongTermMemory", - hierarchy_level="concept", - status="activated", - visibility="public", - embedding=embed_memory_item("Traval plan"), - ), - ) - - graph.add_node( - id=concept2.id, - content=concept2.memory, - metadata=concept2.metadata.model_dump(exclude_none=True), - ) - graph.add_edge(source_id=concept1.id, target_id=concept2.id, type="INCLUDE") - - fact1 = TextualMemoryItem( - memory="10-Day Itinerary for Traveling in Italy", - metadata=TreeNodeTextualMemoryMetadata( - memory_type="WorkingMemory", - key="Reward Components", - value="Coverage gain, energy usage penalty, overlap penalty", - hierarchy_level="fact", type="fact", memory_time="2024-01-01", - source="file", - sources=["paper://multi-uav-coverage/reward-details"], status="activated", - confidence=90.0, - tags=["reward", "overlap", "multi-agent"], - entities=["coverage", "energy", "overlap"], visibility="public", - embedding=embed_memory_item("10-Day Itinerary for Traveling in Italy"), - updated_at=datetime.now().isoformat(), + updated_at=now, + embedding=embed_memory_item(topic_text), ), ) + graph.add_node(topic.id, topic.memory, topic.metadata.model_dump(exclude_none=True)) - graph.add_node( - id=fact1.id, content=fact1.memory, metadata=fact1.metadata.model_dump(exclude_none=True) + # === Step 2: Create two concept nodes linked to the topic === + concept_items = [] + for i, text in enumerate(concept_texts): + concept = TextualMemoryItem( + memory=text, + metadata=TreeNodeTextualMemoryMetadata( + memory_type="LongTermMemory", + key=f"Concept {i + 1}", + hierarchy_level="concept", + type="fact", + memory_time="2024-01-01", + status="activated", + visibility="public", + updated_at=now, + embedding=embed_memory_item(text), + tags=["concept"], + confidence=90 + i, + ), + ) + graph.add_node(concept.id, concept.memory, concept.metadata.model_dump(exclude_none=True)) + graph.add_edge(topic.id, concept.id, type="PARENT") + concept_items.append(concept) + + # === Step 3: Create supporting facts under each concept === + for i, text in enumerate(fact_texts): + fact = TextualMemoryItem( + memory=text, + metadata=TreeNodeTextualMemoryMetadata( + memory_type="WorkingMemory", + key=f"Fact {i + 1}", + hierarchy_level="fact", + type="fact", + memory_time="2024-01-01", + status="activated", + visibility="public", + updated_at=now, + embedding=embed_memory_item(text), + confidence=85.0, + tags=["fact"], + ), + ) + graph.add_node(fact.id, fact.memory, fact.metadata.model_dump(exclude_none=True)) + graph.add_edge(concept_items[i % len(concept_items)].id, fact.id, type="PARENT") + + # === Step 4: Retrieve memory using semantic search === + vector = embed_memory_item("How is memory retrieved?") + search_result = graph.search_by_embedding(vector, top_k=2) + for r in search_result: + node = graph.get_node(r["id"]) + print("🔍 Search result:", node["memory"]) + + # === Step 5: Tag-based neighborhood discovery === + neighbors = graph.get_neighbors_by_tag(["concept"], exclude_ids=[], top_k=2) + print("📎 Tag-related nodes:", [neighbor["memory"] for neighbor in neighbors]) + + # === Step 6: Retrieve children (facts) of first concept === + children = graph.get_children_with_embeddings(concept_items[0].id) + print("📍 Children of concept:", [child["memory"] for child in children]) + + # === Step 7: Export a local subgraph and grouped statistics === + subgraph = graph.get_subgraph(topic.id, depth=2) + print("📌 Subgraph node count:", len(subgraph["neighbors"])) + + stats = graph.get_grouped_counts(["memory_type", "status"]) + print("📊 Grouped counts:", stats) + + # === Step 8: Demonstrate updates and cleanup === + graph.update_node(concept_items[0].id, {"confidence": 99.0}) + graph.remove_oldest_memory("WorkingMemory", keep_latest=1) + graph.delete_edge(topic.id, concept_items[0].id, type="PARENT") + graph.delete_node(concept_items[1].id) + + # === Step 9: Export and re-import the entire graph structure === + exported = graph.export_graph() + graph.import_graph(exported) + print("📦 Graph exported and re-imported, total nodes:", len(exported["nodes"])) + + +def example_complex_shared_db(db_name: str = "shared-traval-group-complex", community=False): + # User 1: Alice explores structured memory for LLMs + run_user_session( + user_name="alice", + db_name=db_name, + topic_text="Alice studies structured memory and long-term memory optimization in LLMs.", + concept_texts=[ + "Short-term memory can be simulated using WorkingMemory blocks.", + "A structured memory graph improves retrieval precision for agents.", + ], + fact_texts=[ + "Embedding search is used to find semantically similar memory items.", + "User memories are stored as node-edge structures that support hierarchical reasoning.", + ], + community=community, ) - graph.add_edge(source_id=concept2.id, target_id=fact1.id, type="INCLUDE") - all_graph_data = graph.export_graph() - print(all_graph_data) + # User 2: Bob focuses on GNN-based reasoning + run_user_session( + user_name="bob", + db_name=db_name, + topic_text="Bob investigates how graph neural networks can support knowledge reasoning.", + concept_texts=[ + "GNNs can learn high-order relations among entities.", + "Attention mechanisms in graphs improve inference precision.", + ], + fact_texts=[ + "GAT outperforms GCN in graph classification tasks.", + "Multi-hop reasoning helps answer complex queries.", + ], + community=community, + ) - nodes = graph.search_by_embedding(vector=embed_memory_item("what does FT reflect?"), top_k=1) - for node_i in nodes: - print(graph.get_node(node_i["id"])) +if __name__ == "__main__": + print("\n=== Example: Multi-DB ===") + example_multi_db(db_name="paper") + print("\n=== Example: Single-DB ===") + example_shared_db(db_name="shared-traval-group") -if __name__ == "__main__": - example_1_paper(db_name="paper") + print("\n=== Example: Single-DB-Complex ===") + example_complex_shared_db(db_name="shared-traval-group-complex-new") -if __name__ == "__main__": - example_2_travel(db_name="traval") + print("\n=== Example: Single-Community-DB-Complex ===") + example_complex_shared_db(db_name="paper", community=True) diff --git a/examples/core_memories/tree_textual_memory.py b/examples/core_memories/tree_textual_memory.py index 9a4035b87..17f68832e 100644 --- a/examples/core_memories/tree_textual_memory.py +++ b/examples/core_memories/tree_textual_memory.py @@ -1,3 +1,5 @@ +import time + from memos import log from memos.configs.embedder import EmbedderConfigFactory from memos.configs.mem_reader import SimpleStructMemReaderConfig @@ -25,7 +27,9 @@ def embed_memory_item(memory: str) -> list[float]: return embedder.embed([memory])[0] -tree_config = TreeTextMemoryConfig.from_json_file("examples/data/config/tree_config.json") +tree_config = TreeTextMemoryConfig.from_json_file( + "examples/data/config/tree_config_shared_database.json" +) my_tree_textual_memory = TreeTextMemory(tree_config) my_tree_textual_memory.delete_all() @@ -182,9 +186,13 @@ def embed_memory_item(memory: str) -> list[float]: memory = reader.get_memory(scene_data, type="chat", info={"user_id": "1234", "session_id": "2222"}) for m_list in memory: - my_tree_textual_memory.add(m_list) + added_ids = my_tree_textual_memory.add(m_list) + for i, id in enumerate(added_ids): + print(f"{i}'th added result is:" + my_tree_textual_memory.get(id).memory) my_tree_textual_memory.memory_manager.wait_reorganizer() +time.sleep(60) + results = my_tree_textual_memory.search( "Talk about the user's childhood story?", top_k=10, @@ -211,7 +219,7 @@ def embed_memory_item(memory: str) -> list[float]: doc_memory = reader.get_memory(doc_paths, "doc", info={"user_id": "1111", "session_id": "2222"}) for m_list in doc_memory: - my_tree_textual_memory.add(m_list) + added_ids = my_tree_textual_memory.add(m_list) my_tree_textual_memory.memory_manager.wait_reorganizer() results = my_tree_textual_memory.search( diff --git a/examples/core_memories/vllm_kv_cache_memory.py b/examples/core_memories/vllm_kv_cache_memory.py new file mode 100644 index 000000000..65ae64911 --- /dev/null +++ b/examples/core_memories/vllm_kv_cache_memory.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +""" +Example demonstrating how to use VLLMKVCacheMemory with vLLM backend. +This example shows how to use the new vLLM-compatible KV cache memory. +""" + +from memos.configs.memory import MemoryConfigFactory +from memos.memories.factory import MemoryFactory + + +def main(): + """Main function demonstrating VLLMKVCacheMemory usage.""" + + print("=== VLLM KV Cache Memory Example ===\n") + + # 1. Create config for VLLMKVCacheMemory (using vLLM backend) + config = MemoryConfigFactory( + backend="vllm_kv_cache", # Use the new vLLM KV cache backend + config={ + "extractor_llm": { + "backend": "vllm", + "config": { + "model_name_or_path": "/mnt/afs/models/hf_models/Qwen2.5-7B", + "api_base": "http://localhost:8088/v1", + "temperature": 0.7, + "max_tokens": 1024, + "model_schema": "memos.configs.llm.VLLMLLMConfig", + }, + }, + }, + ) + + # 2. Instantiate VLLMKVCacheMemory using the factory + print("Initializing VLLM KV Cache Memory...") + vllm_kv_mem = MemoryFactory.from_config(config) + print("✓ VLLM KV Cache Memory initialized successfully.\n") + + # 3. Extract a VLLMKVCacheItem from a prompt + print("===== Extract VLLMKVCacheItem =====") + system_prompt = [ + {"role": "system", "content": "You are a helpful AI assistant."}, + {"role": "user", "content": "What is MemOS?"}, + {"role": "assistant", "content": "MemOS is a memory operating system for LLMs."}, + ] + + try: + cache_item = vllm_kv_mem.extract(system_prompt) + print("✓ KV cache item extracted successfully") + print(f" ID: {cache_item.id}") + print(f" Memory (prompt): {cache_item.memory[:100]}...") + print(f" Metadata: {cache_item.metadata}") + print() + except Exception as e: + print(f"✗ Failed to extract KV cache item: {e}") + return + + # 4. Add the extracted VLLMKVCacheItem + print("===== Add VLLMKVCacheItem =====") + vllm_kv_mem.add([cache_item]) + all_items = vllm_kv_mem.get_all() + print(f"✓ Added cache item. Total items: {len(all_items)}") + print() + + # 5. Get by id + print("===== Get VLLMKVCacheItem by id =====") + retrieved = vllm_kv_mem.get(cache_item.id) + if retrieved: + print(f"✓ Retrieved cache item: {retrieved.id}") + print(f" Memory (prompt): {retrieved.memory[:100]}...") + else: + print("✗ Failed to retrieve cache item") + print() + + # 6. Get cache (returns prompt string for vLLM) + print("===== Get Cache (Prompt String) =====") + prompt_string = vllm_kv_mem.get_cache([cache_item.id]) + if prompt_string: + print(f"✓ Retrieved prompt string: {prompt_string[:100]}...") + print(" This prompt can be used for vLLM generation with preloaded KV cache") + else: + print("✗ Failed to retrieve prompt string") + print() + + # 7. Extract another cache item for demonstration + print("===== Extract Another VLLMKVCacheItem =====") + another_prompt = [ + {"role": "system", "content": "You are a coding assistant."}, + {"role": "user", "content": "Write a Python function to calculate fibonacci numbers."}, + ] + + try: + cache_item2 = vllm_kv_mem.extract(another_prompt) + vllm_kv_mem.add([cache_item2]) + print(f"✓ Added second cache item. Total items: {len(vllm_kv_mem.get_all())}") + print() + except Exception as e: + print(f"✗ Failed to extract second KV cache item: {e}") + print() + + # 8. Preload KV cache on vLLM server + print("===== Preload KV Cache on vLLM Server =====") + try: + vllm_kv_mem.preload_kv_cache([cache_item.id, cache_item2.id]) + print("✓ KV cache preloaded on vLLM server successfully") + print(" The server now has the KV cache ready for fast generation") + except Exception as e: + print(f"✗ Failed to preload KV cache: {e}") + print() + + # 9. Delete one item + print("===== Delete One VLLMKVCacheItem =====") + vllm_kv_mem.delete([cache_item.id]) + remaining_items = vllm_kv_mem.get_all() + print(f"✓ Deleted cache item. Remaining items: {len(remaining_items)}") + print() + + # 10. Dump and load + print("===== Dump and Load VLLMKVCacheMemory =====") + try: + vllm_kv_mem.dump("tmp/vllm_kv_mem") + print("✓ Memory dumped to 'tmp/vllm_kv_mem'") + + # Clear memory and reload + vllm_kv_mem.delete_all() + vllm_kv_mem.load("tmp/vllm_kv_mem") + reloaded_items = vllm_kv_mem.get_all() + print(f"✓ Memory loaded from 'tmp/vllm_kv_mem': {len(reloaded_items)} items") + except Exception as e: + print(f"✗ Failed to dump/load memory: {e}") + print() + + print("=== Example completed successfully ===") + + +if __name__ == "__main__": + main() diff --git a/examples/data/config/mem_scheduler/general_scheduler_config.yaml b/examples/data/config/mem_scheduler/general_scheduler_config.yaml index 036f96a29..5c82db24c 100644 --- a/examples/data/config/mem_scheduler/general_scheduler_config.yaml +++ b/examples/data/config/mem_scheduler/general_scheduler_config.yaml @@ -2,9 +2,8 @@ backend: general_scheduler config: top_k: 10 top_n: 5 - act_mem_update_interval: 300 + act_mem_update_interval: 30 context_window_size: 5 - activation_mem_size: 5 thread_pool_max_workers: 5 consume_interval_seconds: 3 enable_parallel_dispatch: true diff --git a/examples/data/config/mem_scheduler/mem_cube_config.yaml b/examples/data/config/mem_scheduler/mem_cube_config.yaml index a846b6f7c..76428abb0 100644 --- a/examples/data/config/mem_scheduler/mem_cube_config.yaml +++ b/examples/data/config/mem_scheduler/mem_cube_config.yaml @@ -20,7 +20,7 @@ text_mem: graph_db: backend: "neo4j" config: - uri: "bolt://123.57.48.226:7687" + uri: "bolt://localhost:7687" user: "neo4j" password: "12345678" db_name: "user11alice" diff --git a/examples/data/config/mem_scheduler/memos_config_w_scheduler.yaml b/examples/data/config/mem_scheduler/memos_config_w_scheduler.yaml index c77c8d1a7..a8d46ae1f 100644 --- a/examples/data/config/mem_scheduler/memos_config_w_scheduler.yaml +++ b/examples/data/config/mem_scheduler/memos_config_w_scheduler.yaml @@ -34,9 +34,8 @@ mem_scheduler: config: top_k: 10 top_n: 5 - act_mem_update_interval: 300 + act_mem_update_interval: 30 context_window_size: 5 - activation_mem_size: 1000 thread_pool_max_workers: 10 consume_interval_seconds: 3 enable_parallel_dispatch: true diff --git a/examples/data/config/mem_scheduler/memos_config_w_scheduler_and_openai.yaml b/examples/data/config/mem_scheduler/memos_config_w_scheduler_and_openai.yaml new file mode 100644 index 000000000..cdebb9af3 --- /dev/null +++ b/examples/data/config/mem_scheduler/memos_config_w_scheduler_and_openai.yaml @@ -0,0 +1,49 @@ +user_id: "root" +chat_model: + backend: "huggingface" + config: + model_name_or_path: "Qwen/Qwen3-1.7B" + temperature: 0.1 + remove_think_prefix: true + max_tokens: 4096 +mem_reader: + backend: "simple_struct" + config: + llm: + backend: "openai" + config: + model_name_or_path: "gpt-4o-mini" + temperature: 0.8 + max_tokens: 4096 + top_p: 0.9 + top_k: 50 + remove_think_prefix: true + api_key: "sk-xxxxxx" + api_base: "https://api.openai.com/v1" + embedder: + backend: "ollama" + config: + model_name_or_path: "nomic-embed-text:latest" + chunker: + backend: "sentence" + config: + tokenizer_or_token_counter: "gpt2" + chunk_size: 512 + chunk_overlap: 128 + min_sentences_per_chunk: 1 +mem_scheduler: + backend: "general_scheduler" + config: + top_k: 2 + top_n: 5 + act_mem_update_interval: 30 + context_window_size: 5 + thread_pool_max_workers: 10 + consume_interval_seconds: 3 + enable_parallel_dispatch: true +max_turns_window: 20 +top_k: 5 +enable_textual_memory: true +enable_activation_memory: true +enable_parametric_memory: false +enable_mem_scheduler: true diff --git a/examples/data/config/tree_config_community.json b/examples/data/config/tree_config_community.json new file mode 100644 index 000000000..cbd2c8a07 --- /dev/null +++ b/examples/data/config/tree_config_community.json @@ -0,0 +1,50 @@ +{ + "extractor_llm": { + "backend": "ollama", + "config": { + "model_name_or_path": "qwen3:0.6b", + "temperature": 0.0, + "remove_think_prefix": true, + "max_tokens": 8192 + } + }, + "dispatcher_llm": { + "backend": "ollama", + "config": { + "model_name_or_path": "qwen3:0.6b", + "temperature": 0.0, + "remove_think_prefix": true, + "max_tokens": 8192 + } + }, + "embedder": { + "backend": "ollama", + "config": { + "model_name_or_path": "nomic-embed-text:latest" + } + }, + "graph_db": { + "backend": "neo4j-community", + "config": { + "uri": "bolt://localhost:7687", + "user": "neo4j", + "password": "12345678", + "db_name": "neo4j", + "user_name": "alice", + "use_multi_db": false, + "auto_create": false, + "embedding_dimension": 768, + "vec_config": { + "backend": "qdrant", + "config": { + "collection_name": "neo4j_vec_db", + "vector_dimension": 768, + "distance_metric": "cosine", + "host": "localhost", + "port": 6333 + } + } + } + }, + "reorganize": true +} diff --git a/examples/data/config/tree_config_shared_database.json b/examples/data/config/tree_config_shared_database.json new file mode 100644 index 000000000..914f3c725 --- /dev/null +++ b/examples/data/config/tree_config_shared_database.json @@ -0,0 +1,40 @@ +{ + "extractor_llm": { + "backend": "ollama", + "config": { + "model_name_or_path": "qwen3:0.6b", + "temperature": 0.0, + "remove_think_prefix": true, + "max_tokens": 8192 + } + }, + "dispatcher_llm": { + "backend": "ollama", + "config": { + "model_name_or_path": "qwen3:0.6b", + "temperature": 0.0, + "remove_think_prefix": true, + "max_tokens": 8192 + } + }, + "embedder": { + "backend": "ollama", + "config": { + "model_name_or_path": "nomic-embed-text:latest" + } + }, + "graph_db": { + "backend": "neo4j", + "config": { + "uri": "bolt://localhost:7687", + "user": "neo4j", + "password": "12345678", + "db_name": "shared-tree-textual-memory", + "user_name": "alice", + "auto_create": true, + "use_multi_db": false, + "embedding_dimension": 768 + } + }, + "reorganize": true +} diff --git a/examples/mem_mcp/simple_fastmcp_client.py b/examples/mem_mcp/simple_fastmcp_client.py new file mode 100644 index 000000000..1981e3066 --- /dev/null +++ b/examples/mem_mcp/simple_fastmcp_client.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +"""Working FastMCP Client""" + +import asyncio + +from fastmcp import Client + + +async def main(): + """Main function using FastMCP Client""" + + print("Working FastMCP Client") + print("=" * 40) + + async with Client("http://127.0.0.1:8000/mcp") as client: + print("Connected to MOS MCP server!") + + print("Available tools:") + tools = await client.list_tools() + for tool in tools: + print("**" * 20) + print(f" - {tool.name}: {tool.description}") + + print("Available resources:") + resources = await client.list_resources() + for resource in resources: + print(f" - {resource.uri}: {resource.description}") + + print("Testing tool calls...") + + print(" Getting user info...") + result = await client.call_tool("get_user_info", {}) + print(f" Result: {result.content[0].text}") + + print(" Creating user...") + result = await client.call_tool( + "create_user", + {"user_id": "fastmcp_user", "role": "USER", "user_name": "FastMCP Test User"}, + ) + print(f"Result: {result.content[0].text}") + + print(" register cube...") + result = await client.call_tool( + "register_cube", + { + "cube_name_or_path": "cube_default_user", + "user_id": "fastmcp_user", + "cube_id": "fastmcp_user", + }, + ) + print(f" Result: {result}") + + print(" Adding memory...") + result = await client.call_tool( + "add_memory", + { + "memory_content": "This is a test memory from FastMCP client.", + "cube_id": "fastmcp_user", + "user_id": "fastmcp_user", + }, + ) + print(f" Result: {result.content[0].text}") + + print(" Searching memories...") + result = await client.call_tool( + "search_memories", {"query": "test memory", "user_id": "fastmcp_user"} + ) + print(f" Result: {result.content[0].text[:200]}...") + + print(" Testing chat...") + result = await client.call_tool( + "chat", {"query": "Hello! Tell me about yourself.", "user_id": "fastmcp_user"} + ) + print(f" Result: {result.content[0].text[:200]}...") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/mem_mcp/simple_fastmcp_serve.py b/examples/mem_mcp/simple_fastmcp_serve.py new file mode 100644 index 000000000..78c05cd05 --- /dev/null +++ b/examples/mem_mcp/simple_fastmcp_serve.py @@ -0,0 +1,37 @@ +import argparse +import os + +from memos.api.mcp_serve import MOSMCPStdioServer + + +if __name__ == "__main__": + import argparse + + from dotenv import load_dotenv + + load_dotenv() + + # Parse command line arguments + parser = argparse.ArgumentParser(description="MOS MCP Server") + parser.add_argument( + "--transport", + choices=["stdio", "http", "sse"], + default="stdio", + help="Transport method (default: stdio)", + ) + parser.add_argument("--host", default="localhost", help="Host for HTTP/SSE transport") + parser.add_argument("--port", type=int, default=8000, help="Port for HTTP/SSE transport") + + args = parser.parse_args() + + # Set environment variables + os.environ["OPENAI_API_BASE"] = os.getenv("OPENAI_API_BASE") + os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY") + os.environ["MOS_TEXT_MEM_TYPE"] = "tree_text" # "tree_text" need set neo4j + os.environ["NEO4J_URI"] = os.getenv("NEO4J_URI") + os.environ["NEO4J_USER"] = os.getenv("NEO4J_USER") + os.environ["NEO4J_PASSWORD"] = os.getenv("NEO4J_PASSWORD") + + # Create and run MCP server + server = MOSMCPStdioServer() + server.run(transport=args.transport, host=args.host, port=args.port) diff --git a/examples/mem_os/chat_w_generated_cube_explicit_memory.py b/examples/mem_os/chat_w_generated_cube_explicit_memory.py index 61000190e..d19a68486 100644 --- a/examples/mem_os/chat_w_generated_cube_explicit_memory.py +++ b/examples/mem_os/chat_w_generated_cube_explicit_memory.py @@ -84,7 +84,7 @@ "graph_db": { "backend": "neo4j", "config": { - "uri": "bolt://123.57.48.226:7687", + "uri": "bolt://localhost:7687", "user": "neo4j", "password": "12345678", "db_name": "user03alice11", diff --git a/examples/mem_os/easy_memos.py b/examples/mem_os/easy_memos.py new file mode 100644 index 000000000..a2e8014ed --- /dev/null +++ b/examples/mem_os/easy_memos.py @@ -0,0 +1,21 @@ +#!/usr/bin/env python3 +""" +Simple test script for MOS.simple() functionality. +""" + +import os + +from memos.mem_os.main import MOS + + +# Set environment variables for testing +os.environ["OPENAI_API_BASE"] = "http://xxxxxxxxx" +os.environ["OPENAI_API_KEY"] = "sk-xxxxxxxxxx" +os.environ["MOS_TEXT_MEM_TYPE"] = "general_text" # "tree_text" need set neo4j + + +memory = MOS.simple() +print("MOS.simple() works!") +memory.add(memory_content="my favorite color is blue") +print(memory.chat("what is my favorite color?")) +# Your favorite color is blue! diff --git a/examples/mem_os/locomo_shared_database_memos.py b/examples/mem_os/locomo_shared_database_memos.py new file mode 100644 index 000000000..97efe1fe6 --- /dev/null +++ b/examples/mem_os/locomo_shared_database_memos.py @@ -0,0 +1,203 @@ +import json +import os + +from dotenv import load_dotenv + +from memos import log +from memos.configs.mem_cube import GeneralMemCubeConfig +from memos.configs.mem_os import MOSConfig +from memos.mem_cube.general import GeneralMemCube +from memos.mem_os.product import MOSProduct + + +load_dotenv() + + +logger = log.get_logger(__name__) + + +# === Load conversation === +with open("evaluation/data/locomo/locomo10.json", encoding="utf-8") as f: + conversation = json.load(f) + data = conversation[3] + speaker_a = data["conversation"]["speaker_a"] + speaker_b = data["conversation"]["speaker_b"] + conversation_i = data["conversation"] + +db_name = "shared-db-locomo-case" + +openapi_config = { + "model_name_or_path": "gpt-4o-mini", + "temperature": 0.8, + "max_tokens": 1024, + "api_key": "your-api-key-here", + "api_base": "https://api.openai.com/v1", +} + + +# === Create MOS Config === +def get_user_configs(user_name): + mos_config = MOSConfig( + user_id=user_name, + chat_model={"backend": "openai", "config": openapi_config}, + mem_reader={ + "backend": "simple_struct", + "config": { + "llm": {"backend": "openai", "config": openapi_config}, + "embedder": { + "backend": "universal_api", + "config": { + "provider": "openai", + "api_key": openapi_config["api_key"], + "model_name_or_path": "text-embedding-3-large", + "base_url": openapi_config["api_base"], + }, + }, + "chunker": { + "backend": "sentence", + "config": { + "tokenizer_or_token_counter": "gpt2", + "chunk_size": 512, + "chunk_overlap": 128, + "min_sentences_per_chunk": 1, + }, + }, + }, + }, + enable_textual_memory=True, + enable_activation_memory=False, + enable_parametric_memory=False, + top_k=5, + max_turns_window=20, + ) + + return mos_config + + +# === Get Memory Cube Config === +def get_mem_cube_config(user_name): + neo4j_uri = os.getenv("NEO4J_URI", "bolt://localhost:7687") + neo4j_config = { + "uri": neo4j_uri, + "user": "neo4j", + "password": "12345678", + "db_name": db_name, + "user_name": "will be updated", + "use_multi_db": False, + "embedding_dimension": 3072, + "auto_create": True, + } + cube_config = GeneralMemCubeConfig.model_validate( + { + "user_id": user_name, + "cube_id": f"{user_name}_cube", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": {"backend": "openai", "config": openapi_config}, + "dispatcher_llm": {"backend": "openai", "config": openapi_config}, + "graph_db": {"backend": "neo4j", "config": neo4j_config}, + "embedder": { + "backend": "universal_api", + "config": { + "provider": "openai", + "api_key": openapi_config["api_key"], + "model_name_or_path": "text-embedding-3-large", + "base_url": openapi_config["api_base"], + }, + }, + "reorganize": True, + }, + }, + } + ) + + mem_cube = GeneralMemCube(cube_config) + return mem_cube + + +# === Initialize MOSProduct === +root_config = get_user_configs(user_name="system") +mos_product = MOSProduct(default_config=root_config) + + +# === Register both users === +users = {} +for speaker in [speaker_a, speaker_b]: + user_id = speaker.lower() + "_test" + config = get_user_configs(user_id) + mem_cube = get_mem_cube_config(user_id) + result = mos_product.user_register( + user_id=user_id, + user_name=speaker, + interests=f"I'm {speaker}", + default_mem_cube=mem_cube, + ) + users[speaker] = {"user_id": user_id, "default_cube_id": result["default_cube_id"]} + print(f"✅ Registered: {speaker} -> {result}") + +# === Process conversation, add to both roles === +i = 1 +MAX_CONVERSATION_FOR_TEST = 3 +while ( + f"session_{i}_date_time" in conversation_i and f"session_{i}" in conversation_i +) and i < MAX_CONVERSATION_FOR_TEST: + session_i = conversation_i[f"session_{i}"] + session_time = conversation_i[f"session_{i}_date_time"] + + print(f"\n=== Processing Session {i} | Time: {session_time} ===") + + role1_msgs, role2_msgs = [], [] + + for m in session_i: + if m["speaker"] == speaker_a: + role1_msgs.append( + { + "role": "user", + "content": f"{m['speaker']}:{m['text']}", + "chat_time": session_time, + } + ) + role2_msgs.append( + { + "role": "assistant", + "content": f"{m['speaker']}:{m['text']}", + "chat_time": session_time, + } + ) + elif m["speaker"] == speaker_b: + role1_msgs.append( + { + "role": "assistant", + "content": f"{m['speaker']}:{m['text']}", + "chat_time": session_time, + } + ) + role2_msgs.append( + { + "role": "user", + "content": f"{m['speaker']}:{m['text']}", + "chat_time": session_time, + } + ) + + print(f"\n[Session {i}] {speaker_a} will add {len(role1_msgs)} messages.") + print(f"[Session {i}] {speaker_b} will add {len(role2_msgs)} messages.") + + mos_product.add( + user_id=users[speaker_a]["user_id"], + messages=role1_msgs, + mem_cube_id=users[speaker_a]["default_cube_id"], + ) + mos_product.add( + user_id=users[speaker_b]["user_id"], + messages=role2_msgs, + mem_cube_id=users[speaker_b]["default_cube_id"], + ) + + print(f"[Session {i}] Added messages for both roles") + + i += 1 + +print("\n✅ All messages added for both roles.\n") +mos_product.mem_reorganizer_off() diff --git a/examples/mem_os/multi_user_memos_example.py b/examples/mem_os/multi_user_memos_example.py new file mode 100644 index 000000000..196cb380d --- /dev/null +++ b/examples/mem_os/multi_user_memos_example.py @@ -0,0 +1,125 @@ +""" +Example demonstrating how to use MOSProduct for multi-user scenarios. +""" + +from memos.configs.mem_cube import GeneralMemCubeConfig +from memos.configs.mem_os import MOSConfig +from memos.mem_cube.general import GeneralMemCube +from memos.mem_os.product import MOSProduct + + +def get_config(user_name): + openapi_config = { + "model_name_or_path": "gpt-4o-mini", + "temperature": 0.8, + "max_tokens": 1024, + "top_p": 0.9, + "top_k": 50, + "remove_think_prefix": True, + "api_key": "your-api-key-here", + "api_base": "https://api.openai.com/v1", + } + # Create a default configuration + default_config = MOSConfig( + user_id="root", + chat_model={"backend": "openai", "config": openapi_config}, + mem_reader={ + "backend": "naive", + "config": { + "llm": { + "backend": "openai", + "config": openapi_config, + }, + "embedder": { + "backend": "ollama", + "config": { + "model_name_or_path": "nomic-embed-text:latest", + }, + }, + }, + }, + enable_textual_memory=True, + enable_activation_memory=False, + top_k=5, + max_turns_window=20, + ) + default_cube_config = GeneralMemCubeConfig.model_validate( + { + "user_id": user_name, + "cube_id": f"{user_name}_default_cube", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": {"backend": "openai", "config": openapi_config}, + "dispatcher_llm": {"backend": "openai", "config": openapi_config}, + "graph_db": { + "backend": "neo4j", + "config": { + "uri": "bolt://localhost:7687", + "user": "neo4j", + "password": "12345678", + "db_name": user_name, + "auto_create": True, + }, + }, + "embedder": { + "backend": "ollama", + "config": { + "model_name_or_path": "nomic-embed-text:latest", + }, + }, + }, + }, + "act_mem": {}, + "para_mem": {}, + } + ) + default_mem_cube = GeneralMemCube(default_cube_config) + return default_config, default_mem_cube + + +def main(): + default_config, default_mem_cube = get_config(user_name="alice") + # Initialize MOSProduct with default config + mos_product = MOSProduct(default_config=default_config) + + # Register first user with default config + result1 = mos_product.user_register( + user_id="alice", + user_name="alice", + interests="I'm interested in machine learning and AI research.", + default_mem_cube=default_mem_cube, + ) + print(f"User registration result: {result1}") + + # Chat with Alice + print("\n=== Chatting with Alice ===") + for response_chunk in mos_product.chat(query="What are my interests?", user_id="alice"): + print(response_chunk, end="") + + # Add memory for Alice + mos_product.add( + user_id="alice", + memory_content="I attended a machine learning conference last week.", + mem_cube_id=result1["default_cube_id"], + ) + + # Search memories for Alice + search_result = mos_product.search(query="conference", user_id="alice") + print(f"\nSearch result for Alice: {search_result}") + + # Search memories for Alice + search_result = mos_product.get_all(query="conference", user_id="alice", memory_type="text_mem") + print(f"\nSearch result for Alice: {search_result}") + + # List all users + users = mos_product.list_users() + print(f"\nAll registered users: {users}") + + # Get user info + alice_info = mos_product.get_user_info("alice") + print(f"\nAlice's info: {alice_info}") + + +if __name__ == "__main__": + main() diff --git a/examples/mem_os/persistent_memos_example.py b/examples/mem_os/persistent_memos_example.py new file mode 100644 index 000000000..16353be6a --- /dev/null +++ b/examples/mem_os/persistent_memos_example.py @@ -0,0 +1,192 @@ +""" +Example demonstrating persistent user management in MemOS. + +This example shows how to use the PersistentUserManager to maintain +user configurations across service restarts. +""" + +import os +import tempfile + +from memos.configs.mem_os import MOSConfig +from memos.mem_os.product import MOSProduct +from memos.mem_user.persistent_user_manager import PersistentUserManager, UserRole + + +def create_sample_config(user_id: str) -> MOSConfig: + """Create a sample configuration for a user.""" + return MOSConfig( + user_id=user_id, + chat_model={ + "backend": "openai", + "config": { + "model_name_or_path": "gpt-3.5-turbo", + "api_key": "your-api-key-here", + "temperature": 0.7, + }, + }, + mem_reader={ + "backend": "naive", + "config": { + "llm": { + "backend": "openai", + "config": { + "model_name_or_path": "gpt-3.5-turbo", + "api_key": "your-api-key-here", + }, + }, + "embedder": { + "backend": "ollama", + "config": { + "model_name_or_path": "nomic-embed-text:latest", + }, + }, + }, + }, + enable_textual_memory=True, + enable_activation_memory=False, + top_k=5, + max_turns_window=20, + ) + + +def demonstrate_persistence(): + """Demonstrate the persistence functionality.""" + print("=== MemOS Persistent User Management Demo ===\n") + + # Create a temporary database for this demo + temp_dir = tempfile.mkdtemp() + db_path = os.path.join(temp_dir, "demo_memos.db") + + try: + # Step 1: Create a persistent user manager + print("1. Creating PersistentUserManager...") + user_manager = PersistentUserManager(db_path=db_path) + print(f" Database created at: {db_path}") + + # Step 2: Create some sample configurations + print("\n2. Creating sample user configurations...") + user_configs = {} + for i in range(3): + user_id = f"user_{i + 1}" + user_name = f"User {i + 1}" + config = create_sample_config(user_id) + user_configs[user_id] = config + + # Create user with configuration + created_id = user_manager.create_user_with_config( + user_name, config, UserRole.USER, user_id + ) + print(f" Created user: {user_name} (ID: {created_id})") + + # Step 3: Verify configurations are saved + print("\n3. Verifying configurations are saved...") + for user_id in user_configs: + config = user_manager.get_user_config(user_id) + if config: + print(f" ✓ Configuration found for {user_id}") + print(f" - Textual memory enabled: {config.enable_textual_memory}") + print(f" - Top-k: {config.top_k}") + else: + print(f" ✗ Configuration not found for {user_id}") + + # Step 4: Simulate service restart by creating a new manager instance + print("\n4. Simulating service restart...") + print(" Creating new PersistentUserManager instance...") + new_user_manager = PersistentUserManager(db_path=db_path) + + # Step 5: Verify configurations are restored + print("\n5. Verifying configurations are restored after restart...") + for user_id in user_configs: + config = new_user_manager.get_user_config(user_id) + if config: + print(f" ✓ Configuration restored for {user_id}") + else: + print(f" ✗ Configuration not restored for {user_id}") + + # Step 6: Create MOSProduct and demonstrate restoration + print("\n6. Creating MOSProduct with persistent user manager...") + default_config = create_sample_config("default_user") + mos_product = MOSProduct(default_config=default_config) + + # The MOSProduct should automatically restore user instances + print(f" Active user instances: {len(mos_product.user_instances)}") + for user_id in mos_product.user_instances: + print(f" - {user_id}") + + # Step 7: Demonstrate configuration update + print("\n7. Demonstrating configuration update...") + user_id = "user_1" + original_config = user_manager.get_user_config(user_id) + if original_config: + # Update configuration + updated_config = original_config.model_copy(deep=True) + updated_config.top_k = 10 + updated_config.enable_activation_memory = True + + success = user_manager.save_user_config(user_id, updated_config) + if success: + print(f" ✓ Updated configuration for {user_id}") + print(f" - New top-k: {updated_config.top_k}") + print(f" - Activation memory: {updated_config.enable_activation_memory}") + else: + print(f" ✗ Failed to update configuration for {user_id}") + + # Step 8: List all configurations + print("\n8. Listing all user configurations...") + all_configs = user_manager.list_user_configs() + print(f" Total configurations: {len(all_configs)}") + for user_id, config in all_configs.items(): + print( + f" - {user_id}: top_k={config.top_k}, textual_memory={config.enable_textual_memory}" + ) + + print("\n=== Demo completed successfully! ===") + print(f"Database file: {db_path}") + print("You can inspect this file to see the persistent data.") + + except Exception as e: + print(f"Error during demo: {e}") + raise + finally: + # Cleanup + if os.path.exists(db_path): + os.remove(db_path) + if os.path.exists(temp_dir): + os.rmdir(temp_dir) + + +def demonstrate_api_usage(): + """Demonstrate how the API would work with persistence.""" + print("\n=== API Usage Example ===") + print(""" + With the new persistent system, your API calls would work like this: + + 1. Register a user (configuration is automatically saved): + POST /product/users/register + { + "user_id": "john_doe", + "user_name": "John Doe", + "interests": "AI, machine learning, programming" + } + + 2. Get user configuration: + GET /product/users/john_doe/config + + 3. Update user configuration: + PUT /product/users/john_doe/config + { + "user_id": "john_doe", + "enable_activation_memory": true, + "top_k": 10, + ... + } + + 4. After service restart, all user instances are automatically restored + and the user can immediately use the system without re-registration. + """) + + +if __name__ == "__main__": + demonstrate_persistence() + demonstrate_api_usage() diff --git a/examples/mem_os/simple_openapi_memos_neo4j_community.py b/examples/mem_os/simple_openapi_memos_neo4j_community.py new file mode 100644 index 000000000..aad1b8c77 --- /dev/null +++ b/examples/mem_os/simple_openapi_memos_neo4j_community.py @@ -0,0 +1,315 @@ +import os +import time +import uuid + +from datetime import datetime + +from dotenv import load_dotenv + +from memos.configs.mem_cube import GeneralMemCubeConfig +from memos.configs.mem_os import MOSConfig +from memos.mem_cube.general import GeneralMemCube +from memos.mem_os.main import MOS + + +load_dotenv() + +# 1. Create MOS Config and set openai config +print(f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to create MOS configuration...") +start_time = time.time() + +user_name = str(uuid.uuid4()) +print(user_name) + +# 1.1 Set openai config +openapi_config = { + "model_name_or_path": "gpt-4o-mini", + "temperature": 0.8, + "max_tokens": 1024, + "top_p": 0.9, + "top_k": 50, + "remove_think_prefix": True, + "api_key": os.getenv("OPENAI_API_KEY", "sk-xxxxx"), + "api_base": os.getenv("OPENAI_API_BASE", "https://api.openai.com/v1"), +} +embedder_config = { + "backend": "universal_api", + "config": { + "provider": "openai", + "api_key": os.getenv("OPENAI_API_KEY", "sk-xxxxx"), + "model_name_or_path": "text-embedding-3-large", + "base_url": os.getenv("OPENAI_API_BASE", "https://api.openai.com/v1"), + }, +} +EMBEDDING_DIMENSION = 3072 + +# 1.2 Set neo4j config +neo4j_uri = os.getenv("NEO4J_URI", "bolt://localhost:7687") + +# 1.3 Create MOS Config +config = { + "user_id": user_name, + "chat_model": { + "backend": "openai", + "config": openapi_config, + }, + "mem_reader": { + "backend": "simple_struct", + "config": { + "llm": { + "backend": "openai", + "config": openapi_config, + }, + "embedder": embedder_config, + "chunker": { + "backend": "sentence", + "config": { + "tokenizer_or_token_counter": "gpt2", + "chunk_size": 512, + "chunk_overlap": 128, + "min_sentences_per_chunk": 1, + }, + }, + }, + }, + "max_turns_window": 20, + "top_k": 5, + "enable_textual_memory": True, + "enable_activation_memory": False, + "enable_parametric_memory": False, +} + +mos_config = MOSConfig(**config) +# you can set PRO_MODE to True to enable CoT enhancement mos_config.PRO_MODE = True +mos = MOS(mos_config) + +print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] MOS configuration created successfully, time elapsed: {time.time() - start_time:.2f}s\n" +) + +# 2. Initialize memory cube +print(f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to initialize MemCube configuration...") +start_time = time.time() + +config = GeneralMemCubeConfig.model_validate( + { + "user_id": user_name, + "cube_id": f"{user_name}", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": { + "backend": "openai", + "config": openapi_config, + }, + "dispatcher_llm": { + "backend": "openai", + "config": openapi_config, + }, + "embedder": embedder_config, + "graph_db": { + "backend": "neo4j-community", + "config": { + "uri": neo4j_uri, + "user": "neo4j", + "password": "12345678", + "db_name": "neo4j", + "user_name": "alice", + "use_multi_db": False, + "auto_create": False, + "embedding_dimension": EMBEDDING_DIMENSION, + "vec_config": { + "backend": "qdrant", + "config": { + "collection_name": "neo4j_vec_db", + "vector_dimension": EMBEDDING_DIMENSION, + "distance_metric": "cosine", + "host": "localhost", + "port": 6333, + }, + }, + }, + }, + "reorganize": True, + }, + }, + "act_mem": {}, + "para_mem": {}, + }, +) + +print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] MemCube configuration initialization completed, time elapsed: {time.time() - start_time:.2f}s\n" +) + +# 3. Initialize the MemCube with the configuration +print(f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to create MemCube instance...") +start_time = time.time() + +mem_cube = GeneralMemCube(config) +try: + mem_cube.dump(f"/tmp/{user_name}/") + print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] MemCube created and saved successfully, time elapsed: {time.time() - start_time:.2f}s\n" + ) +except Exception as e: + print( + f"❌ [{datetime.now().strftime('%H:%M:%S')}] MemCube save failed: {e}, time elapsed: {time.time() - start_time:.2f}s\n" + ) + +# 4. Register the MemCube +print(f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to register MemCube...") +start_time = time.time() + +mos.register_mem_cube(f"/tmp/{user_name}", mem_cube_id=user_name) + +print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] MemCube registration completed, time elapsed: {time.time() - start_time:.2f}s\n" +) + +# 5. Add, get, search memory +print(f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to add single memory...") +start_time = time.time() + +mos.add(memory_content="I like playing football.") + +print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] Single memory added successfully, time elapsed: {time.time() - start_time:.2f}s" +) + +print(f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to get all memories...") +start_time = time.time() + +get_all_results = mos.get_all() + + +# Filter out embedding fields, keeping only necessary fields +def filter_memory_data(memories_data): + filtered_data = {} + for key, value in memories_data.items(): + if key == "text_mem": + filtered_data[key] = [] + for mem_group in value: + # Check if it's the new data structure (list of TextualMemoryItem objects) + if "memories" in mem_group and isinstance(mem_group["memories"], list): + # New data structure: directly a list of TextualMemoryItem objects + filtered_memories = [] + for memory_item in mem_group["memories"]: + # Create filtered dictionary + filtered_item = { + "id": memory_item.id, + "memory": memory_item.memory, + "metadata": {}, + } + # Filter metadata, excluding embedding + if hasattr(memory_item, "metadata") and memory_item.metadata: + for attr_name in dir(memory_item.metadata): + if not attr_name.startswith("_") and attr_name != "embedding": + attr_value = getattr(memory_item.metadata, attr_name) + if not callable(attr_value): + filtered_item["metadata"][attr_name] = attr_value + filtered_memories.append(filtered_item) + + filtered_group = { + "cube_id": mem_group.get("cube_id", ""), + "memories": filtered_memories, + } + filtered_data[key].append(filtered_group) + else: + # Old data structure: dictionary with nodes and edges + filtered_group = { + "memories": {"nodes": [], "edges": mem_group["memories"].get("edges", [])} + } + for node in mem_group["memories"].get("nodes", []): + filtered_node = { + "id": node.get("id"), + "memory": node.get("memory"), + "metadata": { + k: v + for k, v in node.get("metadata", {}).items() + if k != "embedding" + }, + } + filtered_group["memories"]["nodes"].append(filtered_node) + filtered_data[key].append(filtered_group) + else: + filtered_data[key] = value + return filtered_data + + +filtered_results = filter_memory_data(get_all_results) +print(f"Get all results after add memory: {filtered_results['text_mem'][0]['memories']}") + +print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] Get all memories completed, time elapsed: {time.time() - start_time:.2f}s\n" +) + +# 6. Add messages +print(f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to add conversation messages...") +start_time = time.time() + +messages = [ + {"role": "user", "content": "I like playing football."}, + {"role": "assistant", "content": "yes football is my favorite game."}, +] +mos.add(messages) + +print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] Conversation messages added successfully, time elapsed: {time.time() - start_time:.2f}s" +) + +print( + f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to get all memories (after adding messages)..." +) +start_time = time.time() + +get_all_results = mos.get_all() +filtered_results = filter_memory_data(get_all_results) +print(f"Get all results after add messages: {filtered_results}") + +print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] Get all memories completed, time elapsed: {time.time() - start_time:.2f}s\n" +) + +# 7. Add document +print(f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to add document...") +start_time = time.time() +## 7.1 add pdf for ./tmp/data if use doc mem mos.add(doc_path="./tmp/data/") +start_time = time.time() + +get_all_results = mos.get_all() +filtered_results = filter_memory_data(get_all_results) +print(f"Get all results after add doc: {filtered_results}") + +print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] Get all memories completed, time elapsed: {time.time() - start_time:.2f}s\n" +) + +# 8. Search +print(f"🚀 [{datetime.now().strftime('%H:%M:%S')}] Starting to search memories...") +start_time = time.time() + +search_results = mos.search(query="my favorite football game", user_id=user_name) +filtered_search_results = filter_memory_data(search_results) +print(f"Search results: {filtered_search_results}") + +print( + f"✅ [{datetime.now().strftime('%H:%M:%S')}] Memory search completed, time elapsed: {time.time() - start_time:.2f}s\n" +) + +# 9. Chat +print(f"🎯 [{datetime.now().strftime('%H:%M:%S')}] Starting chat mode...") +while True: + user_input = input("👤 [You] ").strip() + if user_input.lower() in ["quit", "exit"]: + break + + print() + chat_start_time = time.time() + response = mos.chat(user_input) + chat_duration = time.time() - chat_start_time + + print(f"🤖 [Assistant] {response}") + print(f"⏱️ [Response time: {chat_duration:.2f}s]\n") + +print("📢 [System] MemChat has stopped.") diff --git a/examples/mem_os/simple_vllm_memos.py b/examples/mem_os/simple_vllm_memos.py new file mode 100644 index 000000000..ffa7a3a24 --- /dev/null +++ b/examples/mem_os/simple_vllm_memos.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +""" +Simple example demonstrating how to use VLLMLLM with an existing vLLM server. +Requires a vLLM server to be running. +""" + +from typing import TYPE_CHECKING + +from memos.configs.llm import VLLMLLMConfig +from memos.llms.vllm import VLLMLLM + + +if TYPE_CHECKING: + from memos.types import MessageDict + + +def main(): + """Main function demonstrating VLLMLLM usage.""" + + # Configuration for connecting to existing vLLM server + config = VLLMLLMConfig( + model_name_or_path="/mnt/afs/models/hf_models/Qwen2.5-7B", # MUST MATCH the --model arg of vLLM server + api_key="", # Not needed for local server + api_base="http://localhost:8088/v1", # vLLM server address with /v1 + temperature=0.7, + max_tokens=512, + top_p=0.9, + model_schema="memos.configs.llm.VLLMLLMConfig", + ) + + # Initialize VLLM LLM + print("Initializing VLLM LLM...") + llm = VLLMLLM(config) + + # Test messages for KV cache building + print("\nBuilding KV cache for system messages...") + system_messages: list[MessageDict] = [ + {"role": "system", "content": "You are a helpful AI assistant."}, + {"role": "user", "content": "Hello! Can you tell me about vLLM?"}, + ] + try: + prompt = llm.build_vllm_kv_cache(system_messages) + print(f"✓ KV cache built successfully for prompt: '{prompt[:100]}...'") + except Exception as e: + print(f"✗ Failed to build KV cache: {e}") + + # Test with different messages for generation + print("\nGenerating response...") + user_messages: list[MessageDict] = [ + {"role": "system", "content": "You are a helpful AI assistant. Please Introduce yourself "}, + {"role": "user", "content": "What are the benefits of using vLLM?"}, + ] + try: + response = llm.generate(user_messages) + print(f"Response: {response}") + except Exception as e: + print(f"Error generating response: {e}") + + +if __name__ == "__main__": + main() diff --git a/examples/mem_scheduler/eval_for_scheduler.py b/examples/mem_scheduler/eval_for_scheduler.py deleted file mode 100644 index adf3c4fd3..000000000 --- a/examples/mem_scheduler/eval_for_scheduler.py +++ /dev/null @@ -1,425 +0,0 @@ -import concurrent.futures -import json -import os -import shutil -import sys - -from collections import defaultdict -from datetime import datetime, timezone -from pathlib import Path -from time import time -from uuid import uuid4 - -import pandas as pd - -from dotenv import load_dotenv -from pydantic import BaseModel, Field - -from memos.configs.mem_cube import GeneralMemCubeConfig -from memos.configs.mem_os import MOSConfig -from memos.configs.mem_scheduler import SchedulerConfigFactory -from memos.configs.memory import MemoryConfigFactory -from memos.log import get_logger -from memos.mem_cube.general import GeneralMemCube -from memos.mem_os.main import MOS -from memos.mem_scheduler.scheduler_factory import SchedulerFactory -from memos.mem_scheduler.utils import parse_yaml -from memos.memories.factory import MemoryFactory - - -logger = get_logger(__name__) - -FILE_PATH = Path(__file__).absolute() -BASE_DIR = FILE_PATH.parent.parent.parent -sys.path.insert(0, str(BASE_DIR)) # Enable execution from any working directory - - -custom_instructions = """ -Generate personal memories that follow these guidelines: - -1. Each memory should be self-contained with complete context, including: - - The person's name, do not use "user" while creating memories - - Personal details (career aspirations, hobbies, life circumstances) - - Emotional states and reactions - - Ongoing journeys or future plans - - Specific dates when events occurred - -2. Include meaningful personal narratives focusing on: - - Identity and self-acceptance journeys - - Family planning and parenting - - Creative outlets and hobbies - - Mental health and self-care activities - - Career aspirations and education goals - - Important life events and milestones - -3. Make each memory rich with specific details rather than general statements - - Include timeframes (exact dates when possible) - - Name specific activities (e.g., "charity race for mental health" rather than just "exercise") - - Include emotional context and personal growth elements - -4. Extract memories only from user messages, not incorporating assistant responses - -5. Format each memory as a paragraph with a clear narrative structure that captures the person's experience, challenges, and aspirations -""" - -ANSWER_PROMPT_MEMOS = """ - You are a knowledgeable and helpful AI assistant. - You have access to conversation memories that help you provide more personalized responses. - Use the memories to understand the user's context, preferences, and past interactions. - If memories are provided, reference them naturally when relevant, but don't explicitly mention having memories. - - - ## Memories: - - {context} - - Question: {question} - Answer: - """ - - -def get_client(frame: str, user_id: str | None = None, version: str = "default"): - config = MemoryConfigFactory( - backend="general_text", - config={ - "extractor_llm": { - "backend": "openai", - "config": { - "model_name_or_path": os.getenv("MODEL"), - "temperature": 0, - "max_tokens": 8192, - "api_key": os.getenv("OPENAI_API_KEY"), - "api_base": os.getenv("OPENAI_BASE_URL"), - }, - }, - "vector_db": { - "backend": "qdrant", - "config": { - "path": f"results/locomo/memos-{version}/storages/{user_id}/qdrant", - "collection_name": "test_textual_memory", - "distance_metric": "cosine", - "vector_dimension": 768, # nomic-embed-text model's embedding dimension is 768 - }, - }, - "embedder": { - "backend": "ollama", - "config": { - "model_name_or_path": os.getenv("EMBEDDING_MODEL"), - }, - }, - }, - ) - m = MemoryFactory.from_config(config) - return m - - -def get_mem_cube(user_id: str | None = None, model_name: str | None = None): - config = MemoryConfigFactory( - backend="general_text", - config={ - "extractor_llm": { - "backend": "openai", - "config": { - "model_name_or_path": os.getenv("MODEL"), - "temperature": 0, - "max_tokens": 8192, - "api_key": os.getenv("OPENAI_API_KEY"), - "api_base": os.getenv("OPENAI_BASE_URL"), - }, - }, - "vector_db": { - "backend": "qdrant", - "config": { - "path": f"{BASE_DIR}/outputs/evaluation/locomo/{model_name}/memos/storages/{user_id}/qdrant", - "collection_name": "test_textual_memory", - "distance_metric": "cosine", - "vector_dimension": 768, # nomic-embed-text model's embedding dimension is 768 - }, - }, - "embedder": { - "backend": "ollama", - "config": { - "model_name_or_path": os.getenv("EMBEDDING_MODEL"), - }, - }, - }, - ) - m = MemoryFactory.from_config(config) - return m - - -def ingest_session(client, session, metadata): - session_date = metadata["session_date"] - date_format = "%I:%M %p on %d %B, %Y UTC" - date_string = datetime.strptime(session_date, date_format).replace(tzinfo=timezone.utc) - iso_date = date_string.isoformat() - conv_idx = metadata["conv_idx"] - conv_id = "locomo_exp_user_" + str(conv_idx) - - for chat in session: - blip_caption = chat.get("blip_captions") - img_description = ( - f"(description of attached image: {blip_caption})" if blip_caption is not None else "" - ) - data = chat.get("speaker") + ": " + chat.get("text") + img_description - logger.info({"context": data, "conv_id": conv_id, "created_at": iso_date}) - msg = [{"role": "user", "content": data}] - - try: - memories = client.extract(msg) - except Exception as ex: - logger.error(f"Error extracting message {msg}: {ex}") - memories = [] - - client.add(memories) - - -def search_query(client, query, metadata): - start = time() - search_results = client.search(query, top_k=20) - context = "" - for item in search_results: - item = item.to_dict() - context += f"{item['memory']}\n" - print(query, context) - duration_ms = (time() - start) * 1000 - return context, duration_ms - - -def process_qa(qa, search_result, llm_client): - start = time() - query = qa.get("question") - gold_answer = qa.get("answer") - qa_category = qa.get("category") - - prompt = ANSWER_PROMPT_MEMOS.format( - context=search_result.get("context"), - question=query, - ) - response = llm_client.chat.completions.create( - model=os.getenv("MODEL"), - messages=[ - {"role": "system", "content": prompt}, - ], - temperature=0, - ) - answer = response.choices[0].message.content or "" - - response_duration_ms = (time() - start) * 1000 - - print(f"Processed question: {query}") - print(f"Answer: {answer}") - return { - "question": query, - "answer": answer, - "category": qa_category, - "golden_answer": gold_answer, - "search_context": search_result.get("context", ""), - "response_duration_ms": response_duration_ms, - "search_duration_ms": search_result.get("duration_ms", 0), - } - - -def calculate_f1_score(gold_tokens, response_tokens): - try: - gold_set = set(gold_tokens) - response_set = set(response_tokens) - - if len(gold_set) == 0 or len(response_set) == 0: - return 0.0 - - precision = len(gold_set.intersection(response_set)) / len(response_set) - recall = len(gold_set.intersection(response_set)) / len(gold_set) - - if precision + recall > 0: - return 2 * precision * recall / (precision + recall) - return 0.0 - except Exception as e: - print(f"Failed to calculate F1 score: {e}") - return 0.0 - - -class LLMGrade(BaseModel): - llm_judgment: str = Field(description="CORRECT or WRONG") - llm_reasoning: str = Field(description="Explain why the answer is correct or incorrect.") - - -def locomo_grader(llm_client, question: str, gold_answer: str, response: str) -> bool: - system_prompt = """ - You are an expert grader that determines if answers to questions match a gold standard answer - """ - - accuracy_prompt = f""" - Your task is to label an answer to a question as ’CORRECT’ or ’WRONG’. You williolw23 be given the following data: - (1) a question (posed by one user to another user), - (2) a ’gold’ (ground truth) answer, - (3) a generated answer - which you will score as CORRECT/WRONG. - - The point of the question is to ask about something one user should know about the other user based on their prior conversations. - The gold answer will usually be a concise and short answer that includes the referenced topic, for example: - Question: Do you remember what I got the last time I went to Hawaii? - Gold answer: A shell necklace - The generated answer might be much longer, but you should be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT. - - For time related questions, the gold answer will be a specific date, month, year, etc. The generated answer might be much longer or use relative time references (like "last Tuesday" or "next month"), but you should be generous with your grading - as long as it refers to the same date or time period as the gold answer, it should be counted as CORRECT. Even if the format differs (e.g., "May 7th" vs "7 May"), consider it CORRECT if it's the same date. - - Now it’s time for the real question: - Question: {question} - Gold answer: {gold_answer} - Generated answer: {response} - - First, provide a short (one sentence) explanation of your reasoning, then finish with CORRECT or WRONG. - Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script. - - Just return the label CORRECT or WRONG in a json format with the key as "label". - """ - - response = llm_client.beta.chat.completions.parse( - model=os.getenv("MODEL"), - messages=[ - {"role": "system", "content": system_prompt}, - {"role": "user", "content": accuracy_prompt}, - ], - response_format=LLMGrade, - temperature=0, - ) - result = response.choices[0].message.parsed - - return result.llm_judgment.strip().lower() == "correct" - - -# the entry function -def process_user(conv_idx, locomo_df, model, model_name): - try: - # ============= 1. generate memories ================= - logger.info("============= 1. generate memories =================") - conversation = locomo_df["conversation"].iloc[conv_idx] - max_session_count = 35 - - conv_id = "locomo_exp_user_" + str(conv_idx) - client = get_client("memos") - - for session_idx in range(max_session_count): - session_key = f"session_{session_idx}" - session = conversation.get(session_key) - if session is None: - continue - print(f"User {conv_idx}, {session_key}") - metadata = { - "session_date": conversation.get(f"session_{session_idx}_date_time") + " UTC", - "speaker_a": conversation.get("speaker_a"), - "speaker_b": conversation.get("speaker_b"), - "speaker_a_user_id": f"{conversation.get('speaker_a')}_{conv_idx}", - "speaker_b_user_id": f"{conversation.get('speaker_b')}_{conv_idx}", - "conv_idx": conv_idx, - "session_key": session_key, - } - ingest_session(client, session, metadata) - - conv_id = "locomo_exp_user_" + str(conv_idx) - client.dump(f"{BASE_DIR}/outputs/evaluation/locomo/{model_name}/memos/storages/{conv_id}") - - logger.info(f"Completed processing user {conv_idx}") - - # ============= 2. search memories ================= - logger.info("============= 2. search memories =================") - search_results = defaultdict(list) - qa_set = locomo_df["qa"].iloc[conv_idx] - - metadata = { - "speaker_a": conversation.get("speaker_a"), - "speaker_b": conversation.get("speaker_b"), - "speaker_a_user_id": f"{conversation.get('speaker_a')}_{conv_idx}", - "speaker_b_user_id": f"{conversation.get('speaker_b')}_{conv_idx}", - "conv_idx": conv_idx, - "conv_id": conv_id, - } - qa_filtered_set = [] - for qa in qa_set: - query = qa.get("question") - if qa.get("category") == 5: - continue - qa_filtered_set.append(qa) - context, duration_ms = search_query(client, query, metadata) - search_results[conv_id].append({"context": context, "duration_ms": duration_ms}) - logger.info({"context": context[:20] + "...", "duration_ms": duration_ms}) - - search_path = Path( - f"{BASE_DIR}/outputs/evaluation/locomo/{model_name}/locomo_search_results.json" - ) - search_path.parent.mkdir(exist_ok=True, parents=True) - with search_path.open("w", encoding="utf-8") as fw: - json.dump(dict(search_results), fw, indent=2) - logger.info(f"Save search results {conv_idx}") - - except Exception as e: - return f"Error processing user {conv_idx}: {e!s}" - - -def main(): - # Load environment variables - load_dotenv() - - # Load JSON data - locomo_df = pd.read_json(f"{BASE_DIR}/evaluation/data/locomo/locomo10.json") - - # Process each user in parallel - num_users = 10 - - max_workers = min(num_users, os.cpu_count() * 2) - - # 1. Create Mos Config - config = parse_yaml(f"{BASE_DIR}/examples/data/config/mem_scheduler/memos_config.yaml") - - mos_config = MOSConfig(**config) - mos = MOS(mos_config) - - # 2. Initialization - user_id = f"user_{uuid4!s}" - mos.create_user(user_id) - - config = GeneralMemCubeConfig.from_yaml_file( - f"{BASE_DIR}/examples/data/config/mem_scheduler/mem_cube_config.yaml" - ) - mem_cube_id = "mem_cube_5" - mem_cube_name_or_path = f"{BASE_DIR}/outputs/mem_scheduler/{user_id}/{mem_cube_id}" - if Path(mem_cube_name_or_path).exists(): - shutil.rmtree(mem_cube_name_or_path) - print(f"{mem_cube_name_or_path} is not empty, and has been removed.") - mem_cube = GeneralMemCube(config) - mem_cube.dump(mem_cube_name_or_path) - mos.register_mem_cube( - mem_cube_name_or_path=mem_cube_name_or_path, mem_cube_id=mem_cube_id, user_id=user_id - ) - - # 3. set mem_scheduler - example_scheduler_config_path = ( - f"{BASE_DIR}/examples/data/config/mem_scheduler/general_scheduler_config.yaml" - ) - scheduler_config = SchedulerConfigFactory.from_yaml_file( - yaml_path=example_scheduler_config_path - ) - mem_scheduler = SchedulerFactory.from_config(scheduler_config) - mem_scheduler.initialize_modules(chat_llm=mos.chat_llm) - mos.mem_scheduler = mem_scheduler - - mos.mem_scheduler.start() - - with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor: - futures = { - executor.submit(process_user, i, locomo_df, mos, "memos"): i for i in range(num_users) - } - for future in concurrent.futures.as_completed(futures): - user_id = futures[future] - try: - result = future.result() - print(result) - except Exception as e: - print(f"Error processing user {user_id}: {e!s}") - - -if __name__ == "__main__": - os.environ["MODEL"] = "gpt-4o-mini" - # TODO: This code is not finished yet. - main() diff --git a/examples/mem_scheduler/rabbitmq_example.py b/examples/mem_scheduler/rabbitmq_example.py new file mode 100644 index 000000000..39b982346 --- /dev/null +++ b/examples/mem_scheduler/rabbitmq_example.py @@ -0,0 +1,58 @@ +import threading +import time + +from memos.configs.mem_scheduler import AuthConfig +from memos.mem_scheduler.modules.rabbitmq_service import RabbitMQSchedulerModule + + +def publish_message(rabbitmq_module, message): + """Function to publish a message.""" + rabbitmq_module.rabbitmq_publish_message(message) + print(f"Published message: {message}") + + +def main(): + # Initialize RabbitMQ module + rabbitmq_module = RabbitMQSchedulerModule() + rabbitmq_module.rabbit_queue_name = "test" + + # Initialize from configuration dictionary + if not AuthConfig.default_config_exists(): + print("Please set configs for rabbitmq.") + return + else: + rabbitmq_module.initialize_rabbitmq(config=AuthConfig.from_local_yaml().rabbitmq) + + try: + # Start consumer + rabbitmq_module.rabbitmq_start_consuming() + + # === Publish some test messages === + # List to hold thread references + threads = [] + + # Publish some test messages using multiple threads + for i in range(3): + message = {"type": "test", "data": f"Message {i}", "timestamp": time.time()} + thread = threading.Thread(target=publish_message, args=(rabbitmq_module, message)) + thread.start() + threads.append(thread) + + # Join threads to ensure all messages are published before proceeding + for thread in threads: + thread.join() + + except KeyboardInterrupt: + print("\nProgram interrupted by user") + + finally: + # Give some time for cleanup + time.sleep(5) + + # Close connections + rabbitmq_module.rabbitmq_close() + print("RabbitMQ connection closed") + + +if __name__ == "__main__": + main() diff --git a/examples/mem_scheduler/redis_example.py b/examples/mem_scheduler/redis_example.py index 0e23dbd4d..528893d30 100644 --- a/examples/mem_scheduler/redis_example.py +++ b/examples/mem_scheduler/redis_example.py @@ -7,7 +7,6 @@ from uuid import uuid4 from memos.configs.mem_scheduler import SchedulerConfigFactory -from memos.log import get_logger from memos.mem_cube.general import GeneralMemCube from memos.mem_scheduler.modules.schemas import QUERY_LABEL, ScheduleMessageItem from memos.mem_scheduler.scheduler_factory import SchedulerFactory @@ -21,8 +20,6 @@ BASE_DIR = FILE_PATH.parent.parent.parent sys.path.insert(0, str(BASE_DIR)) # Enable execution from any working directory -logger = get_logger(__name__) - async def service_run(): # Init @@ -43,35 +40,34 @@ async def service_run(): {"question": "What food should I avoid due to allergy?", "category": "Allergy"}, ] init_mem_cube = f"{BASE_DIR}/examples/data/mem_cube_2" - logger.debug("Loading MemChatCube...") + print("Loading MemChatCube...") mem_cube = GeneralMemCube.init_from_dir(init_mem_cube) user_id = str(uuid4) mem_scheduler.initialize_redis() - mem_scheduler.start_listening() + mem_scheduler.redis_start_listening() for item in questions: query = item["question"] message_item = ScheduleMessageItem( user_id=user_id, - cube_id=f"{BASE_DIR}/examples/data/mem_cube_2", + mem_cube_id="mem_cube_2", label=QUERY_LABEL, - cube=mem_cube, + mem_cube=mem_cube, content=query, timestamp=datetime.now(), ) - - res = await mem_scheduler.add_message_stream(message=message_item.to_dict()) - logger.debug( + res = await mem_scheduler.redis_add_message_stream(message=message_item.to_dict()) + print( f"Added: {res}", ) await asyncio.sleep(0.5) - mem_scheduler.stop_listening() + mem_scheduler.redis_stop_listening() - mem_scheduler.close() + mem_scheduler.redis_close() if __name__ == "__main__": diff --git a/examples/mem_scheduler/schedule_chat_and_web.py b/examples/mem_scheduler/schedule_chat_and_web.py new file mode 100644 index 000000000..666f15e02 --- /dev/null +++ b/examples/mem_scheduler/schedule_chat_and_web.py @@ -0,0 +1,179 @@ +import shutil +import sys + +from pathlib import Path +from queue import Queue +from typing import TYPE_CHECKING + +from memos.configs.mem_cube import GeneralMemCubeConfig +from memos.configs.mem_os import MOSConfig +from memos.configs.mem_scheduler import AuthConfig +from memos.log import get_logger +from memos.mem_cube.general import GeneralMemCube +from memos.mem_scheduler.general_scheduler import GeneralScheduler +from memos.mem_scheduler.mos_for_test_scheduler import MOSForTestScheduler + + +if TYPE_CHECKING: + from memos.mem_scheduler.modules.schemas import ( + ScheduleLogForWebItem, + ) + + +FILE_PATH = Path(__file__).absolute() +BASE_DIR = FILE_PATH.parent.parent.parent +sys.path.insert(0, str(BASE_DIR)) # Enable execution from any working directory + +logger = get_logger(__name__) + + +def init_task(): + conversations = [ + { + "role": "user", + "content": "I have two dogs - Max (golden retriever) and Bella (pug). We live in Seattle.", + }, + {"role": "assistant", "content": "Great! Any special care for them?"}, + { + "role": "user", + "content": "Max needs joint supplements. Actually, we're moving to Chicago next month.", + }, + { + "role": "user", + "content": "Correction: Bella is 6, not 5. And she's allergic to chicken.", + }, + { + "role": "user", + "content": "My partner's cat Whiskers visits weekends. Bella chases her sometimes.", + }, + ] + + questions = [ + # 1. Basic factual recall (simple) + { + "question": "What breed is Max?", + "category": "Pet", + "expected": "golden retriever", + "difficulty": "easy", + }, + # 2. Temporal context (medium) + { + "question": "Where will I live next month?", + "category": "Location", + "expected": "Chicago", + "difficulty": "medium", + }, + # 3. Information correction (hard) + { + "question": "How old is Bella really?", + "category": "Pet", + "expected": "6", + "difficulty": "hard", + "hint": "User corrected the age later", + }, + # 4. Relationship inference (harder) + { + "question": "Why might Whiskers be nervous around my pets?", + "category": "Behavior", + "expected": "Bella chases her sometimes", + "difficulty": "harder", + }, + # 5. Combined medical info (hardest) + { + "question": "Which pets have health considerations?", + "category": "Health", + "expected": "Max needs joint supplements, Bella is allergic to chicken", + "difficulty": "hardest", + "requires": ["combining multiple facts", "ignoring outdated info"], + }, + ] + return conversations, questions + + +def show_web_logs(mem_scheduler: GeneralScheduler): + """Display all web log entries from the scheduler's log queue. + + Args: + mem_scheduler: The scheduler instance containing web logs to display + """ + if mem_scheduler._web_log_message_queue.empty(): + print("Web log queue is currently empty.") + return + + print("\n" + "=" * 50 + " WEB LOGS " + "=" * 50) + + # Create a temporary queue to preserve the original queue contents + temp_queue = Queue() + log_count = 0 + + while not mem_scheduler._web_log_message_queue.empty(): + log_item: ScheduleLogForWebItem = mem_scheduler._web_log_message_queue.get() + temp_queue.put(log_item) + log_count += 1 + + # Print log entry details + print(f"\nLog Entry #{log_count}:") + print(f'- "{log_item.label}" log: {log_item}') + + print("-" * 50) + + # Restore items back to the original queue + while not temp_queue.empty(): + mem_scheduler._web_log_message_queue.put(temp_queue.get()) + + print(f"\nTotal {log_count} web log entries displayed.") + print("=" * 110 + "\n") + + +if __name__ == "__main__": + # set up data + conversations, questions = init_task() + + # set configs + mos_config = MOSConfig.from_yaml_file( + f"{BASE_DIR}/examples/data/config/mem_scheduler/memos_config_w_scheduler_and_openai.yaml" + ) + + mem_cube_config = GeneralMemCubeConfig.from_yaml_file( + f"{BASE_DIR}/examples/data/config/mem_scheduler/mem_cube_config.yaml" + ) + + # default local graphdb uri + if AuthConfig.default_config_exists(): + auth_config = AuthConfig.from_local_yaml() + + mos_config.mem_reader.config.llm.config.api_key = auth_config.openai.api_key + mos_config.mem_reader.config.llm.config.api_base = auth_config.openai.base_url + + mem_cube_config.text_mem.config.graph_db.config.uri = auth_config.graph_db.uri + + # Initialization + mos = MOSForTestScheduler(mos_config) + + user_id = "user_1" + mos.create_user(user_id) + + mem_cube_id = "mem_cube_5" + mem_cube_name_or_path = f"{BASE_DIR}/outputs/mem_scheduler/{user_id}/{mem_cube_id}" + + if Path(mem_cube_name_or_path).exists(): + shutil.rmtree(mem_cube_name_or_path) + print(f"{mem_cube_name_or_path} is not empty, and has been removed.") + + mem_cube = GeneralMemCube(mem_cube_config) + mem_cube.dump(mem_cube_name_or_path) + mos.register_mem_cube( + mem_cube_name_or_path=mem_cube_name_or_path, mem_cube_id=mem_cube_id, user_id=user_id + ) + + mos.add(conversations, user_id=user_id, mem_cube_id=mem_cube_id) + + for item in questions: + query = item["question"] + + response = mos.chat(query=query, user_id=user_id) + print(f"Query:\n {query}\n\nAnswer:\n {response}") + + show_web_logs(mos.mem_scheduler) + + mos.mem_scheduler.stop() diff --git a/examples/mem_scheduler/schedule_tree_textual_memory.py b/examples/mem_scheduler/schedule_tree_textual_memory.py deleted file mode 100644 index f2886b1df..000000000 --- a/examples/mem_scheduler/schedule_tree_textual_memory.py +++ /dev/null @@ -1,87 +0,0 @@ -import json -import sys - -from pathlib import Path - -from memos.configs.mem_chat import MemChatConfigFactory -from memos.configs.mem_reader import NaiveMemReaderConfig -from memos.configs.mem_scheduler import SchedulerConfigFactory -from memos.configs.memory import TreeTextMemoryConfig -from memos.llms.factory import LLMFactory -from memos.mem_cube.general import GeneralMemCube -from memos.mem_reader.naive import NaiveMemReader -from memos.mem_scheduler.general_scheduler import GeneralScheduler -from memos.mem_scheduler.scheduler_factory import SchedulerFactory -from memos.memories.textual.tree import TreeTextMemory - - -FILE_PATH = Path(__file__).absolute() -BASE_DIR = FILE_PATH.parent.parent.parent -sys.path.insert(0, str(BASE_DIR)) # Enable execution from any working directory - - -def run_mem_scheduler(mem_scheduler): - turns = [ - {"question": "What is quantum entanglement?"}, - {"question": "How is it different from classical physics?"}, - {"question": "So, what is its relationship with quantum computing?"}, - ] - for turn in turns: - print(f"Processing turn: {turn['question']}") - print( - f"Working memory: {[m.memory for m in mem_scheduler.mem_cube.text_mem.get_working_memory()]}" - ) - session_result = mem_scheduler.process_session_turn(turn) - print( - f"Working memory after process:{[m.memory for m in mem_scheduler.mem_cube.text_mem.get_working_memory()]}" - ) - print(session_result) - - -if __name__ == "__main__": - print("Initializing MemChatConfig...") - config = MemChatConfigFactory.from_yaml_file( - f"{BASE_DIR}/examples/data/config/mem_scheduler/mem_chat_config.yaml" - ) - chat_llm = LLMFactory.from_config(config.config.chat_llm) - - # initialize mem cube - init_mem_cube = f"{BASE_DIR}/examples/data/mem_cube_2" - print("Loading MemChatCube...") - mem_cube = GeneralMemCube.init_from_dir(init_mem_cube) - - # initialize mem scheduler - example_scheduler_config_path = ( - f"{BASE_DIR}/examples/data/config/mem_scheduler/general_scheduler_config.yaml" - ) - scheduler_config = SchedulerConfigFactory.from_yaml_file( - yaml_path=example_scheduler_config_path - ) - mem_scheduler: GeneralScheduler = SchedulerFactory.from_config(scheduler_config) - mem_scheduler.initialize_modules(chat_llm=chat_llm) - mem_scheduler.mem_cube = mem_cube - - tree_config = TreeTextMemoryConfig.from_json_file( - f"{BASE_DIR}/examples/data/config/tree_config.json" - ) - tree_config.graph_db.config.uri = "bolt://123.57.48.226:7687" - text_mem = TreeTextMemory(tree_config) - mem_scheduler.mem_cube.text_mem = text_mem - - # Create a memory reader instance - reader_config = NaiveMemReaderConfig.from_json_file( - f"{BASE_DIR}/examples/data/config/naive_reader_config.json" - ) - - reader = NaiveMemReader(reader_config) - scene_data_file = Path(f"{BASE_DIR}/examples/data/mem_scheduler/scene_data.json") - scene_data = json.load(scene_data_file.open("r", encoding="utf-8")) - # Acquiring memories - memory = reader.get_memory( - scene_data, type="chat", info={"user_id": "1234", "session_id": "2222"} - ) - - print("==== Add memories ====") - for m_list in memory: - text_mem.add(m_list) - run_mem_scheduler(mem_scheduler) diff --git a/examples/mem_scheduler/schedule_w_memos.py b/examples/mem_scheduler/schedule_w_memos.py index 84dac7ad5..9032ded13 100644 --- a/examples/mem_scheduler/schedule_w_memos.py +++ b/examples/mem_scheduler/schedule_w_memos.py @@ -3,19 +3,16 @@ from datetime import datetime from pathlib import Path -from queue import Queue from memos.configs.mem_cube import GeneralMemCubeConfig from memos.configs.mem_os import MOSConfig -from memos.configs.mem_scheduler import SchedulerConfigFactory +from memos.configs.mem_scheduler import AuthConfig, SchedulerConfigFactory from memos.log import get_logger from memos.mem_cube.general import GeneralMemCube from memos.mem_os.main import MOS -from memos.mem_scheduler.general_scheduler import GeneralScheduler from memos.mem_scheduler.modules.schemas import ( ANSWER_LABEL, QUERY_LABEL, - ScheduleLogForWebItem, ScheduleMessageItem, ) from memos.mem_scheduler.scheduler_factory import SchedulerFactory @@ -72,41 +69,6 @@ def init_task(): return conversations, questions -def show_web_logs(mem_scheduler: GeneralScheduler): - """Display all web log entries from the scheduler's log queue. - - Args: - mem_scheduler: The scheduler instance containing web logs to display - """ - if mem_scheduler._web_log_message_queue.empty(): - print("Web log queue is currently empty.") - return - - print("\n" + "=" * 50 + " WEB LOGS " + "=" * 50) - - # Create a temporary queue to preserve the original queue contents - temp_queue = Queue() - log_count = 0 - - while not mem_scheduler._web_log_message_queue.empty(): - log_item: ScheduleLogForWebItem = mem_scheduler._web_log_message_queue.get() - temp_queue.put(log_item) - log_count += 1 - - # Print log entry details - print(f"\nLog Entry #{log_count}:") - print(f"- log: {log_item}") - - print("-" * 50) - - # Restore items back to the original queue - while not temp_queue.empty(): - mem_scheduler._web_log_message_queue.put(temp_queue.get()) - - print(f"\nTotal {log_count} web log entries displayed.") - print("=" * 110 + "\n") - - def run_with_automatic_scheduler_init(): print("==== run_with_automatic_scheduler_init ====") conversations, questions = init_task() @@ -129,6 +91,12 @@ def run_with_automatic_scheduler_init(): if Path(mem_cube_name_or_path).exists(): shutil.rmtree(mem_cube_name_or_path) print(f"{mem_cube_name_or_path} is not empty, and has been removed.") + + # default local graphdb uri + if AuthConfig.default_config_exists(): + auth_config = AuthConfig.from_local_yaml() + config.text_mem.config.graph_db.config.uri = auth_config.graph_db.uri + mem_cube = GeneralMemCube(config) mem_cube.dump(mem_cube_name_or_path) mos.register_mem_cube( @@ -141,7 +109,6 @@ def run_with_automatic_scheduler_init(): response = mos.chat(query, user_id=user_id) print(f"Query:\n {query}\n\nAnswer:\n {response}") - show_web_logs(mos.mem_scheduler) mos.mem_scheduler.stop() @@ -167,6 +134,12 @@ def run_with_manual_scheduler_init(): if Path(mem_cube_name_or_path).exists(): shutil.rmtree(mem_cube_name_or_path) print(f"{mem_cube_name_or_path} is not empty, and has been removed.") + + # default local graphdb uri + if AuthConfig.default_config_exists(): + auth_config = AuthConfig.from_local_yaml() + config.text_mem.config.graph_db.config.uri = auth_config.graph_db.uri + mem_cube = GeneralMemCube(config) mem_cube.dump(mem_cube_name_or_path) mos.register_mem_cube( @@ -211,7 +184,6 @@ def run_with_manual_scheduler_init(): mos.mem_scheduler.submit_messages(messages=message_item) print(f"Query:\n {query}\n\nAnswer:\n {response}") - show_web_logs(mos.mem_scheduler) mos.mem_scheduler.stop() diff --git a/examples/mem_scheduler/try_schedule_modules.py b/examples/mem_scheduler/try_schedule_modules.py new file mode 100644 index 000000000..e735ebfea --- /dev/null +++ b/examples/mem_scheduler/try_schedule_modules.py @@ -0,0 +1,198 @@ +import shutil +import sys + +from pathlib import Path +from queue import Queue +from typing import TYPE_CHECKING + +from tqdm import tqdm + +from memos.configs.mem_cube import GeneralMemCubeConfig +from memos.configs.mem_os import MOSConfig +from memos.configs.mem_scheduler import AuthConfig +from memos.log import get_logger +from memos.mem_cube.general import GeneralMemCube +from memos.mem_scheduler.general_scheduler import GeneralScheduler +from memos.mem_scheduler.modules.schemas import NOT_APPLICABLE_TYPE +from memos.mem_scheduler.mos_for_test_scheduler import MOSForTestScheduler + + +if TYPE_CHECKING: + from memos.mem_scheduler.modules.schemas import ( + ScheduleLogForWebItem, + ) + + +FILE_PATH = Path(__file__).absolute() +BASE_DIR = FILE_PATH.parent.parent.parent +sys.path.insert(0, str(BASE_DIR)) # Enable execution from any working directory + +logger = get_logger(__name__) + + +def init_task(): + conversations = [ + { + "role": "user", + "content": "I have two dogs - Max (golden retriever) and Bella (pug). We live in Seattle.", + }, + {"role": "assistant", "content": "Great! Any special care for them?"}, + { + "role": "user", + "content": "Max needs joint supplements. Actually, we're moving to Chicago next month.", + }, + { + "role": "user", + "content": "Correction: Bella is 6, not 5. And she's allergic to chicken.", + }, + { + "role": "user", + "content": "My partner's cat Whiskers visits weekends. Bella chases her sometimes.", + }, + ] + + questions = [ + # 1. Basic factual recall (simple) + { + "question": "What breed is Max?", + "category": "Pet", + "expected": "golden retriever", + "difficulty": "easy", + }, + # 2. Temporal context (medium) + { + "question": "Where will I live next month?", + "category": "Location", + "expected": "Chicago", + "difficulty": "medium", + }, + # 3. Information correction (hard) + { + "question": "How old is Bella really?", + "category": "Pet", + "expected": "6", + "difficulty": "hard", + "hint": "User corrected the age later", + }, + # 4. Relationship inference (harder) + { + "question": "Why might Whiskers be nervous around my pets?", + "category": "Behavior", + "expected": "Bella chases her sometimes", + "difficulty": "harder", + }, + # 5. Combined medical info (hardest) + { + "question": "Which pets have health considerations?", + "category": "Health", + "expected": "Max needs joint supplements, Bella is allergic to chicken", + "difficulty": "hardest", + "requires": ["combining multiple facts", "ignoring outdated info"], + }, + ] + return conversations, questions + + +def show_web_logs(mem_scheduler: GeneralScheduler): + """Display all web log entries from the scheduler's log queue. + + Args: + mem_scheduler: The scheduler instance containing web logs to display + """ + if mem_scheduler._web_log_message_queue.empty(): + print("Web log queue is currently empty.") + return + + print("\n" + "=" * 50 + " WEB LOGS " + "=" * 50) + + # Create a temporary queue to preserve the original queue contents + temp_queue = Queue() + log_count = 0 + + while not mem_scheduler._web_log_message_queue.empty(): + log_item: ScheduleLogForWebItem = mem_scheduler._web_log_message_queue.get() + temp_queue.put(log_item) + log_count += 1 + + # Print log entry details + print(f"\nLog Entry #{log_count}:") + print(f'- "{log_item.label}" log: {log_item}') + + print("-" * 50) + + # Restore items back to the original queue + while not temp_queue.empty(): + mem_scheduler._web_log_message_queue.put(temp_queue.get()) + + print(f"\nTotal {log_count} web log entries displayed.") + print("=" * 110 + "\n") + + +if __name__ == "__main__": + # set up data + conversations, questions = init_task() + + # set configs + mos_config = MOSConfig.from_yaml_file( + f"{BASE_DIR}/examples/data/config/mem_scheduler/memos_config_w_scheduler_and_openai.yaml" + ) + + mem_cube_config = GeneralMemCubeConfig.from_yaml_file( + f"{BASE_DIR}/examples/data/config/mem_scheduler/mem_cube_config.yaml" + ) + + # default local graphdb uri + if AuthConfig.default_config_exists(): + auth_config = AuthConfig.from_local_yaml() + + mos_config.mem_reader.config.llm.config.api_key = auth_config.openai.api_key + mos_config.mem_reader.config.llm.config.api_base = auth_config.openai.base_url + + mem_cube_config.text_mem.config.graph_db.config.uri = auth_config.graph_db.uri + + # Initialization + mos = MOSForTestScheduler(mos_config) + + user_id = "user_1" + mos.create_user(user_id) + + mem_cube_id = "mem_cube_5" + mem_cube_name_or_path = f"{BASE_DIR}/outputs/mem_scheduler/{user_id}/{mem_cube_id}" + + if Path(mem_cube_name_or_path).exists(): + shutil.rmtree(mem_cube_name_or_path) + print(f"{mem_cube_name_or_path} is not empty, and has been removed.") + + mem_cube = GeneralMemCube(mem_cube_config) + mem_cube.dump(mem_cube_name_or_path) + mos.register_mem_cube( + mem_cube_name_or_path=mem_cube_name_or_path, mem_cube_id=mem_cube_id, user_id=user_id + ) + + mos.add(conversations, user_id=user_id, mem_cube_id=mem_cube_id) + + for item in tqdm(questions, desc="processing queries"): + query = item["question"] + + # test process_session_turn + mos.mem_scheduler.process_session_turn( + queries=[query], + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + top_k=10, + query_history=None, + ) + + # test activation memory update + mos.mem_scheduler.update_activation_memory_periodically( + interval_seconds=0, + label=NOT_APPLICABLE_TYPE, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + ) + + show_web_logs(mos.mem_scheduler) + + mos.mem_scheduler.stop() diff --git a/poetry.lock b/poetry.lock index 0df9b1ddc..845f3f65c 100644 --- a/poetry.lock +++ b/poetry.lock @@ -2,49 +2,16 @@ [[package]] name = "absl-py" -version = "2.3.0" +version = "2.3.1" description = "Abseil Python Common Libraries, see https://github.com/abseil/abseil-py." optional = false python-versions = ">=3.8" groups = ["eval"] files = [ - {file = "absl_py-2.3.0-py3-none-any.whl", hash = "sha256:9824a48b654a306168f63e0d97714665f8490b8d89ec7bf2efc24bf67cf579b3"}, - {file = "absl_py-2.3.0.tar.gz", hash = "sha256:d96fda5c884f1b22178852f30ffa85766d50b99e00775ea626c23304f582fc4f"}, + {file = "absl_py-2.3.1-py3-none-any.whl", hash = "sha256:eeecf07f0c2a93ace0772c92e596ace6d3d3996c042b2128459aaae2a76de11d"}, + {file = "absl_py-2.3.1.tar.gz", hash = "sha256:a97820526f7fbfd2ec1bce83f3f25e3a14840dac0d8e02a0b71cd75db3f77fc9"}, ] -[[package]] -name = "accelerate" -version = "1.7.0" -description = "Accelerate" -optional = false -python-versions = ">=3.9.0" -groups = ["main"] -files = [ - {file = "accelerate-1.7.0-py3-none-any.whl", hash = "sha256:cf57165cca28769c6cf2650812371c81b18e05743dfa3c748524b1bb4f2b272f"}, - {file = "accelerate-1.7.0.tar.gz", hash = "sha256:e8a2a5503d6237b9eee73cc8d36cf543f9c2d8dd2c6713450b322f5e6d53a610"}, -] - -[package.dependencies] -huggingface-hub = ">=0.21.0" -numpy = ">=1.17,<3.0.0" -packaging = ">=20.0" -psutil = "*" -pyyaml = "*" -safetensors = ">=0.4.3" -torch = ">=2.0.0" - -[package.extras] -deepspeed = ["deepspeed"] -dev = ["bitsandbytes", "black (>=23.1,<24.0)", "datasets", "diffusers", "evaluate", "hf-doc-builder (>=0.3.0)", "parameterized", "pytest (>=7.2.0,<=8.0.0)", "pytest-order", "pytest-subtests", "pytest-xdist", "rich", "ruff (>=0.11.2,<0.12.0)", "scikit-learn", "scipy", "timm", "torchdata (>=0.8.0)", "torchpippy (>=0.2.0)", "tqdm", "transformers"] -quality = ["black (>=23.1,<24.0)", "hf-doc-builder (>=0.3.0)", "ruff (>=0.11.2,<0.12.0)"] -rich = ["rich"] -sagemaker = ["sagemaker"] -test-dev = ["bitsandbytes", "datasets", "diffusers", "evaluate", "scikit-learn", "scipy", "timm", "torchdata (>=0.8.0)", "torchpippy (>=0.2.0)", "tqdm", "transformers"] -test-fp8 = ["torchao"] -test-prod = ["parameterized", "pytest (>=7.2.0,<=8.0.0)", "pytest-order", "pytest-subtests", "pytest-xdist"] -test-trackers = ["comet-ml", "dvclive", "matplotlib", "mlflow", "tensorboard", "wandb"] -testing = ["bitsandbytes", "datasets", "diffusers", "evaluate", "parameterized", "pytest (>=7.2.0,<=8.0.0)", "pytest-order", "pytest-subtests", "pytest-xdist", "scikit-learn", "scipy", "timm", "torchdata (>=0.8.0)", "torchpippy (>=0.2.0)", "tqdm", "transformers"] - [[package]] name = "annotated-types" version = "0.7.0" @@ -113,25 +80,60 @@ description = "Timeout context manager for asyncio programs" optional = false python-versions = ">=3.7" groups = ["main", "eval"] -markers = "python_version == \"3.10\"" files = [ {file = "async-timeout-4.0.3.tar.gz", hash = "sha256:4640d96be84d82d02ed59ea2b7105a0f7b33abe8703703cd0ab0bf87c427522f"}, {file = "async_timeout-4.0.3-py3-none-any.whl", hash = "sha256:7405140ff1230c310e51dc27b3145b9092d659ce68ff733fb0cefe3ee42be028"}, ] +markers = {main = "(extra == \"mem-scheduler\" or extra == \"all\") and python_version == \"3.10\"", eval = "python_version == \"3.10\""} [[package]] name = "async-timeout" version = "5.0.1" description = "Timeout context manager for asyncio programs" -optional = false +optional = true python-versions = ">=3.8" groups = ["main"] -markers = "python_version == \"3.11\" and python_full_version < \"3.11.3\"" +markers = "(extra == \"mem-scheduler\" or extra == \"all\") and python_full_version < \"3.11.3\" and python_version == \"3.11\"" files = [ {file = "async_timeout-5.0.1-py3-none-any.whl", hash = "sha256:39e3809566ff85354557ec2398b55e096c8364bacac9405a7a1fa429e77fe76c"}, {file = "async_timeout-5.0.1.tar.gz", hash = "sha256:d9321a7a3d5a6a5e187e824d2fa0793ce379a202935782d555d6e9d2735677d3"}, ] +[[package]] +name = "attrs" +version = "25.3.0" +description = "Classes Without Boilerplate" +optional = false +python-versions = ">=3.8" +groups = ["main"] +files = [ + {file = "attrs-25.3.0-py3-none-any.whl", hash = "sha256:427318ce031701fea540783410126f03899a97ffc6f61596ad581ac2e40e3bc3"}, + {file = "attrs-25.3.0.tar.gz", hash = "sha256:75d7cefc7fb576747b2c81b4442d4d4a1ce0900973527c011d1030fd3bf4af1b"}, +] + +[package.extras] +benchmark = ["cloudpickle ; platform_python_implementation == \"CPython\"", "hypothesis", "mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pympler", "pytest (>=4.3.0)", "pytest-codspeed", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-xdist[psutil]"] +cov = ["cloudpickle ; platform_python_implementation == \"CPython\"", "coverage[toml] (>=5.3)", "hypothesis", "mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-xdist[psutil]"] +dev = ["cloudpickle ; platform_python_implementation == \"CPython\"", "hypothesis", "mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pre-commit-uv", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-xdist[psutil]"] +docs = ["cogapp", "furo", "myst-parser", "sphinx", "sphinx-notfound-page", "sphinxcontrib-towncrier", "towncrier"] +tests = ["cloudpickle ; platform_python_implementation == \"CPython\"", "hypothesis", "mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-xdist[psutil]"] +tests-mypy = ["mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\""] + +[[package]] +name = "authlib" +version = "1.6.0" +description = "The ultimate Python library in building OAuth and OpenID Connect servers and clients." +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "authlib-1.6.0-py2.py3-none-any.whl", hash = "sha256:91685589498f79e8655e8a8947431ad6288831d643f11c55c2143ffcc738048d"}, + {file = "authlib-1.6.0.tar.gz", hash = "sha256:4367d32031b7af175ad3a323d571dc7257b7099d55978087ceae4a0d88cd3210"}, +] + +[package.dependencies] +cryptography = "*" + [[package]] name = "backoff" version = "2.2.1" @@ -148,9 +150,10 @@ files = [ name = "beautifulsoup4" version = "4.13.4" description = "Screen-scraping library" -optional = false +optional = true python-versions = ">=3.7.0" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "beautifulsoup4-4.13.4-py3-none-any.whl", hash = "sha256:9bbbb14bfde9d79f38b8cd5f8c7c85f4b8f2523190ebed90e950a8dea4cb1c4b"}, {file = "beautifulsoup4-4.13.4.tar.gz", hash = "sha256:dbb3c4e1ceae6aefebdaf2423247260cd062430a410e38c66f2baa50a8437195"}, @@ -191,14 +194,14 @@ transformers = ">=3.0.0" [[package]] name = "certifi" -version = "2025.4.26" +version = "2025.7.14" description = "Python package for providing Mozilla's CA Bundle." optional = false -python-versions = ">=3.6" +python-versions = ">=3.7" groups = ["main", "eval"] files = [ - {file = "certifi-2025.4.26-py3-none-any.whl", hash = "sha256:30350364dfe371162649852c63336a15c70c6510c2ad5015b21c2345311805f3"}, - {file = "certifi-2025.4.26.tar.gz", hash = "sha256:0a816057ea3cdefcef70270d2c515e4506bbc954f417fa5ade2021213bb8f0c6"}, + {file = "certifi-2025.7.14-py3-none-any.whl", hash = "sha256:6b31f564a415d79ee77df69d757bb49a5bb53bd9f756cbbe24394ffd6fc1f4b2"}, + {file = "certifi-2025.7.14.tar.gz", hash = "sha256:8ea99dbdfaaf2ba2f9bac77b9249ef62ec5218e7c2b2e903378ed5fccf765995"}, ] [[package]] @@ -398,50 +401,68 @@ files = [ [[package]] name = "chonkie" -version = "1.0.7" +version = "1.1.1" description = "🦛 CHONK your texts with Chonkie ✨ - The no-nonsense chunking library" -optional = false +optional = true python-versions = ">=3.9" groups = ["main"] -files = [ - {file = "chonkie-1.0.7-py3-none-any.whl", hash = "sha256:2cce530c765a8a525d6c34bdede81a0f174e8bfd6e0b7f25bb968c7b410c43d4"}, - {file = "chonkie-1.0.7.tar.gz", hash = "sha256:dddc29116aef3aa472632a98234ce37c697169adf045d2f9211f39aa512162dc"}, +markers = "extra == \"mem-reader\" or extra == \"all\"" +files = [ + {file = "chonkie-1.1.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:c56cff89f38ff5cc06b2e8b4c9a802b85b77ba8ecdda6896f5dba6b0c54d4303"}, + {file = "chonkie-1.1.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8bb0d88b4254a9bac7b494349ee3c94f94d3ee2f8cd4970d23e0c0ef3e6392a4"}, + {file = "chonkie-1.1.1-cp310-cp310-win_amd64.whl", hash = "sha256:515f200597f86a4e877d8f57cb21e50b5ab67df123f796cf87396c9ad3dd5f73"}, + {file = "chonkie-1.1.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ebb19d8888f7c4cac6c32bd3fd050cfec3067e21fb4bb02ea95795d19a8954cb"}, + {file = "chonkie-1.1.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3f0f85f1682809564377fdbdbf36d1d170f5ec904ad81297c7bc1ad96617033c"}, + {file = "chonkie-1.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:0644f2664b563550c520f33f54586cfe6fcc0892de79078e41c2e1e1488e8d81"}, + {file = "chonkie-1.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c9e3249d55173c08fb9692992ce5021704a83c91c01615f1386748b0cb24a25e"}, + {file = "chonkie-1.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:acfc4f6827e2bba98c31805fcd71feba654957140c42970a3bd995eff89625b7"}, + {file = "chonkie-1.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:afad359a02f3db46d3fcad2639d062ead5a3ad2bae45cb9f511ae0efdec79656"}, + {file = "chonkie-1.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ecf8f99a9f5981d399b22d69c050a8dab4e376d221f6bb298bbc126571315049"}, + {file = "chonkie-1.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:653f53702f286df1348657702f2660010cabb2dee90c8783978d5adfa818fec4"}, + {file = "chonkie-1.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:5905b31adf73006b832f4bd20f8f31a180b31114d9c75e9b8ff2ca74d12a2172"}, + {file = "chonkie-1.1.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:11130d189252dabace35967d2b8d16d325448fec2e0b8a926fcd333c0093737b"}, + {file = "chonkie-1.1.1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:35038142af6141c26bc777c195a440c56693d9def7ffc4e647cec8e0493904a6"}, + {file = "chonkie-1.1.1-cp39-cp39-win_amd64.whl", hash = "sha256:e2e00362bbc697dc345dcdf3b77853bd3848a1377e79074aad64ce8dac319d42"}, + {file = "chonkie-1.1.1.tar.gz", hash = "sha256:391f90bd137fba5b82ed34411082a2abe46b6d12c00d75d73655f59f6c1ae504"}, ] [package.dependencies] -tokenizers = ">=0.16.0" tqdm = ">=4.64.0" [package.extras] -all = ["accelerate (>=1.6.0)", "chromadb (>=1.0.0)", "cohere (>=5.13.0)", "google-genai (>=1.0.0)", "huggingface-hub (>=0.24.0)", "jsonschema (>=4.23.0)", "magika (>=0.6.0,<0.7.0)", "model2vec (>=0.3.0)", "numpy (>=2.0.0,<3.0)", "openai (>=1.0.0)", "pydantic (>=2.0.0)", "qdrant-client (>=1.0.0)", "rich (>=13.0.0)", "sentence-transformers (>=3.0.0)", "tiktoken (>=0.5.0)", "torch (>=2.0.0,<3.0)", "transformers (>=4.0.0)", "tree-sitter (>=0.20.0)", "tree-sitter-language-pack (>=0.7.0)", "turbopuffer[fast] (>=0.2.0)"] +all = ["accelerate (>=1.6.0)", "chromadb (>=1.0.0)", "cohere (>=5.13.0)", "google-genai (>=1.0.0)", "huggingface-hub (>=0.24.0)", "jsonschema (>=4.23.0)", "magika (>=0.6.0,<0.7.0)", "model2vec (>=0.3.0)", "numpy (>=2.0.0,<3.0)", "openai (>=1.0.0)", "pydantic (>=2.0.0)", "qdrant-client (>=1.0.0)", "rich (>=13.0.0)", "sentence-transformers (>=3.0.0)", "tiktoken (>=0.5.0)", "tokenizers (>=0.16.0)", "torch (>=2.0.0,<3.0)", "transformers (>=4.0.0)", "tree-sitter (>=0.20.0)", "tree-sitter-language-pack (>=0.7.0)", "turbopuffer[fast] (>=0.2.0)"] chroma = ["chromadb (>=1.0.0)"] code = ["magika (>=0.6.0,<0.7.0)", "tree-sitter (>=0.20.0)", "tree-sitter-language-pack (>=0.7.0)"] cohere = ["cohere (>=5.13.0)", "numpy (>=2.0.0,<3.0)"] -dev = ["coverage", "datasets (>=1.14.0)", "mypy (>=1.11.0)", "pytest (>=6.2.0)", "pytest-asyncio (>=0.26.0)", "pytest-cov (>=4.0.0)", "pytest-xdist (>=2.5.0)", "ruff (>=0.0.265)", "transformers (>=4.0.0)"] +datasets = ["datasets (>=4.0.0)"] +dev = ["coverage", "cython (>=3.0.0)", "datasets (>=1.14.0)", "mypy (>=1.11.0)", "pytest (>=6.2.0)", "pytest-asyncio (>=0.26.0)", "pytest-cov (>=4.0.0)", "pytest-xdist (>=2.5.0)", "ruff (>=0.0.265)", "transformers (>=4.0.0)"] gemini = ["google-genai (>=1.0.0)", "pydantic (>=2.0.0)"] genie = ["google-genai (>=1.0.0)", "pydantic (>=2.0.0)"] hub = ["huggingface-hub (>=0.24.0)", "jsonschema (>=4.23.0)"] jina = ["numpy (>=2.0.0,<3.0)"] model2vec = ["model2vec (>=0.3.0)", "numpy (>=2.0.0,<3.0)"] neural = ["torch (>=2.0.0,<3.0)", "transformers (>=4.0.0)"] -openai = ["numpy (>=2.0.0,<3.0)", "openai (>=1.0.0)", "pydantic (>=2.0.0)", "tiktoken (>=0.5.0)"] +openai = ["numpy (>=2.0.0,<3.0)", "openai (>=1.0.0)", "pydantic (>=2.0.0)"] +pgvector = ["vecs (>=0.4.0)"] qdrant = ["qdrant-client (>=1.0.0)"] semantic = ["model2vec (>=0.3.0)", "numpy (>=2.0.0,<3.0)"] st = ["accelerate (>=1.6.0)", "numpy (>=2.0.0,<3.0)", "sentence-transformers (>=3.0.0)"] +tiktoken = ["tiktoken (>=0.5.0)"] +tokenizers = ["tokenizers (>=0.16.0)"] tpuf = ["turbopuffer[fast] (>=0.2.0)"] viz = ["rich (>=13.0.0)"] voyageai = ["numpy (>=2.0.0,<3.0)", "voyageai (>=0.3.2)"] [[package]] name = "click" -version = "8.2.0" +version = "8.2.1" description = "Composable command line interface toolkit" optional = false python-versions = ">=3.10" groups = ["main", "eval"] files = [ - {file = "click-8.2.0-py3-none-any.whl", hash = "sha256:6b303f0b2aa85f1cb4e5303078fadcbcd4e476f114fab9b5007005711839325c"}, - {file = "click-8.2.0.tar.gz", hash = "sha256:f5452aeddd9988eefa20f90f05ab66f17fce1ee2a36907fd30b05bbb5953814d"}, + {file = "click-8.2.1-py3-none-any.whl", hash = "sha256:61a3265b914e850b85317d0b3109c7f8cd35a670f963866005d6ef1d5175a12b"}, + {file = "click-8.2.1.tar.gz", hash = "sha256:27c491cc05d968d271d5a1db13e3b5a184636d9d930f148c50b038f0d0646202"}, ] [package.dependencies] @@ -451,9 +472,10 @@ colorama = {version = "*", markers = "platform_system == \"Windows\""} name = "cobble" version = "0.1.4" description = "Create data objects" -optional = false +optional = true python-versions = ">=3.5" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "cobble-0.1.4-py3-none-any.whl", hash = "sha256:36c91b1655e599fd428e2b95fdd5f0da1ca2e9f1abb0bc871dec21a0e78a2b44"}, {file = "cobble-0.1.4.tar.gz", hash = "sha256:de38be1539992c8a06e569630717c485a5f91be2192c461ea2b220607dfa78aa"}, @@ -476,9 +498,10 @@ markers = {main = "platform_system == \"Windows\" or sys_platform == \"win32\"", name = "coloredlogs" version = "15.0.1" description = "Colored terminal output for Python's logging module" -optional = false +optional = true python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "coloredlogs-15.0.1-py2.py3-none-any.whl", hash = "sha256:612ee75c546f53e92e70049c9dbfcc18c935a2b9a53b66085ce9ef6a6e5c0934"}, {file = "coloredlogs-15.0.1.tar.gz", hash = "sha256:7c991aa71a4577af2f82600d8f8f3a89f936baeaf9b50a9c197da014e5bf16b0"}, @@ -569,62 +592,62 @@ test-no-images = ["pytest", "pytest-cov", "pytest-rerunfailures", "pytest-xdist" [[package]] name = "cryptography" -version = "44.0.3" +version = "45.0.5" description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers." optional = false python-versions = "!=3.9.0,!=3.9.1,>=3.7" groups = ["main"] files = [ - {file = "cryptography-44.0.3-cp37-abi3-macosx_10_9_universal2.whl", hash = "sha256:962bc30480a08d133e631e8dfd4783ab71cc9e33d5d7c1e192f0b7c06397bb88"}, - {file = "cryptography-44.0.3-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4ffc61e8f3bf5b60346d89cd3d37231019c17a081208dfbbd6e1605ba03fa137"}, - {file = "cryptography-44.0.3-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:58968d331425a6f9eedcee087f77fd3c927c88f55368f43ff7e0a19891f2642c"}, - {file = "cryptography-44.0.3-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e28d62e59a4dbd1d22e747f57d4f00c459af22181f0b2f787ea83f5a876d7c76"}, - {file = "cryptography-44.0.3-cp37-abi3-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:af653022a0c25ef2e3ffb2c673a50e5a0d02fecc41608f4954176f1933b12359"}, - {file = "cryptography-44.0.3-cp37-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:157f1f3b8d941c2bd8f3ffee0af9b049c9665c39d3da9db2dc338feca5e98a43"}, - {file = "cryptography-44.0.3-cp37-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:c6cd67722619e4d55fdb42ead64ed8843d64638e9c07f4011163e46bc512cf01"}, - {file = "cryptography-44.0.3-cp37-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:b424563394c369a804ecbee9b06dfb34997f19d00b3518e39f83a5642618397d"}, - {file = "cryptography-44.0.3-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c91fc8e8fd78af553f98bc7f2a1d8db977334e4eea302a4bfd75b9461c2d8904"}, - {file = "cryptography-44.0.3-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:25cd194c39fa5a0aa4169125ee27d1172097857b27109a45fadc59653ec06f44"}, - {file = "cryptography-44.0.3-cp37-abi3-win32.whl", hash = "sha256:3be3f649d91cb182c3a6bd336de8b61a0a71965bd13d1a04a0e15b39c3d5809d"}, - {file = "cryptography-44.0.3-cp37-abi3-win_amd64.whl", hash = "sha256:3883076d5c4cc56dbef0b898a74eb6992fdac29a7b9013870b34efe4ddb39a0d"}, - {file = "cryptography-44.0.3-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:5639c2b16764c6f76eedf722dbad9a0914960d3489c0cc38694ddf9464f1bb2f"}, - {file = "cryptography-44.0.3-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f3ffef566ac88f75967d7abd852ed5f182da252d23fac11b4766da3957766759"}, - {file = "cryptography-44.0.3-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:192ed30fac1728f7587c6f4613c29c584abdc565d7417c13904708db10206645"}, - {file = "cryptography-44.0.3-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:7d5fe7195c27c32a64955740b949070f21cba664604291c298518d2e255931d2"}, - {file = "cryptography-44.0.3-cp39-abi3-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3f07943aa4d7dad689e3bb1638ddc4944cc5e0921e3c227486daae0e31a05e54"}, - {file = "cryptography-44.0.3-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:cb90f60e03d563ca2445099edf605c16ed1d5b15182d21831f58460c48bffb93"}, - {file = "cryptography-44.0.3-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:ab0b005721cc0039e885ac3503825661bd9810b15d4f374e473f8c89b7d5460c"}, - {file = "cryptography-44.0.3-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:3bb0847e6363c037df8f6ede57d88eaf3410ca2267fb12275370a76f85786a6f"}, - {file = "cryptography-44.0.3-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:b0cc66c74c797e1db750aaa842ad5b8b78e14805a9b5d1348dc603612d3e3ff5"}, - {file = "cryptography-44.0.3-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6866df152b581f9429020320e5eb9794c8780e90f7ccb021940d7f50ee00ae0b"}, - {file = "cryptography-44.0.3-cp39-abi3-win32.whl", hash = "sha256:c138abae3a12a94c75c10499f1cbae81294a6f983b3af066390adee73f433028"}, - {file = "cryptography-44.0.3-cp39-abi3-win_amd64.whl", hash = "sha256:5d186f32e52e66994dce4f766884bcb9c68b8da62d61d9d215bfe5fb56d21334"}, - {file = "cryptography-44.0.3-pp310-pypy310_pp73-macosx_10_9_x86_64.whl", hash = "sha256:cad399780053fb383dc067475135e41c9fe7d901a97dd5d9c5dfb5611afc0d7d"}, - {file = "cryptography-44.0.3-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:21a83f6f35b9cc656d71b5de8d519f566df01e660ac2578805ab245ffd8523f8"}, - {file = "cryptography-44.0.3-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:fc3c9babc1e1faefd62704bb46a69f359a9819eb0292e40df3fb6e3574715cd4"}, - {file = "cryptography-44.0.3-pp310-pypy310_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:e909df4053064a97f1e6565153ff8bb389af12c5c8d29c343308760890560aff"}, - {file = "cryptography-44.0.3-pp310-pypy310_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:dad80b45c22e05b259e33ddd458e9e2ba099c86ccf4e88db7bbab4b747b18d06"}, - {file = "cryptography-44.0.3-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:479d92908277bed6e1a1c69b277734a7771c2b78633c224445b5c60a9f4bc1d9"}, - {file = "cryptography-44.0.3-pp311-pypy311_pp73-macosx_10_9_x86_64.whl", hash = "sha256:896530bc9107b226f265effa7ef3f21270f18a2026bc09fed1ebd7b66ddf6375"}, - {file = "cryptography-44.0.3-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:9b4d4a5dbee05a2c390bf212e78b99434efec37b17a4bff42f50285c5c8c9647"}, - {file = "cryptography-44.0.3-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:02f55fb4f8b79c1221b0961488eaae21015b69b210e18c386b69de182ebb1259"}, - {file = "cryptography-44.0.3-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:dd3db61b8fe5be220eee484a17233287d0be6932d056cf5738225b9c05ef4fff"}, - {file = "cryptography-44.0.3-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:978631ec51a6bbc0b7e58f23b68a8ce9e5f09721940933e9c217068388789fe5"}, - {file = "cryptography-44.0.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:5d20cc348cca3a8aa7312f42ab953a56e15323800ca3ab0706b8cd452a3a056c"}, - {file = "cryptography-44.0.3.tar.gz", hash = "sha256:fe19d8bc5536a91a24a8133328880a41831b6c5df54599a8417b62fe015d3053"}, + {file = "cryptography-45.0.5-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:101ee65078f6dd3e5a028d4f19c07ffa4dd22cce6a20eaa160f8b5219911e7d8"}, + {file = "cryptography-45.0.5-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3a264aae5f7fbb089dbc01e0242d3b67dffe3e6292e1f5182122bdf58e65215d"}, + {file = "cryptography-45.0.5-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e74d30ec9c7cb2f404af331d5b4099a9b322a8a6b25c4632755c8757345baac5"}, + {file = "cryptography-45.0.5-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:3af26738f2db354aafe492fb3869e955b12b2ef2e16908c8b9cb928128d42c57"}, + {file = "cryptography-45.0.5-cp311-abi3-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e6c00130ed423201c5bc5544c23359141660b07999ad82e34e7bb8f882bb78e0"}, + {file = "cryptography-45.0.5-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:dd420e577921c8c2d31289536c386aaa30140b473835e97f83bc71ea9d2baf2d"}, + {file = "cryptography-45.0.5-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:d05a38884db2ba215218745f0781775806bde4f32e07b135348355fe8e4991d9"}, + {file = "cryptography-45.0.5-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:ad0caded895a00261a5b4aa9af828baede54638754b51955a0ac75576b831b27"}, + {file = "cryptography-45.0.5-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:9024beb59aca9d31d36fcdc1604dd9bbeed0a55bface9f1908df19178e2f116e"}, + {file = "cryptography-45.0.5-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:91098f02ca81579c85f66df8a588c78f331ca19089763d733e34ad359f474174"}, + {file = "cryptography-45.0.5-cp311-abi3-win32.whl", hash = "sha256:926c3ea71a6043921050eaa639137e13dbe7b4ab25800932a8498364fc1abec9"}, + {file = "cryptography-45.0.5-cp311-abi3-win_amd64.whl", hash = "sha256:b85980d1e345fe769cfc57c57db2b59cff5464ee0c045d52c0df087e926fbe63"}, + {file = "cryptography-45.0.5-cp37-abi3-macosx_10_9_universal2.whl", hash = "sha256:f3562c2f23c612f2e4a6964a61d942f891d29ee320edb62ff48ffb99f3de9ae8"}, + {file = "cryptography-45.0.5-cp37-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3fcfbefc4a7f332dece7272a88e410f611e79458fab97b5efe14e54fe476f4fd"}, + {file = "cryptography-45.0.5-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:460f8c39ba66af7db0545a8c6f2eabcbc5a5528fc1cf6c3fa9a1e44cec33385e"}, + {file = "cryptography-45.0.5-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:9b4cf6318915dccfe218e69bbec417fdd7c7185aa7aab139a2c0beb7468c89f0"}, + {file = "cryptography-45.0.5-cp37-abi3-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:2089cc8f70a6e454601525e5bf2779e665d7865af002a5dec8d14e561002e135"}, + {file = "cryptography-45.0.5-cp37-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:0027d566d65a38497bc37e0dd7c2f8ceda73597d2ac9ba93810204f56f52ebc7"}, + {file = "cryptography-45.0.5-cp37-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:be97d3a19c16a9be00edf79dca949c8fa7eff621763666a145f9f9535a5d7f42"}, + {file = "cryptography-45.0.5-cp37-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:7760c1c2e1a7084153a0f68fab76e754083b126a47d0117c9ed15e69e2103492"}, + {file = "cryptography-45.0.5-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:6ff8728d8d890b3dda5765276d1bc6fb099252915a2cd3aff960c4c195745dd0"}, + {file = "cryptography-45.0.5-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:7259038202a47fdecee7e62e0fd0b0738b6daa335354396c6ddebdbe1206af2a"}, + {file = "cryptography-45.0.5-cp37-abi3-win32.whl", hash = "sha256:1e1da5accc0c750056c556a93c3e9cb828970206c68867712ca5805e46dc806f"}, + {file = "cryptography-45.0.5-cp37-abi3-win_amd64.whl", hash = "sha256:90cb0a7bb35959f37e23303b7eed0a32280510030daba3f7fdfbb65defde6a97"}, + {file = "cryptography-45.0.5-pp310-pypy310_pp73-macosx_10_9_x86_64.whl", hash = "sha256:206210d03c1193f4e1ff681d22885181d47efa1ab3018766a7b32a7b3d6e6afd"}, + {file = "cryptography-45.0.5-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:c648025b6840fe62e57107e0a25f604db740e728bd67da4f6f060f03017d5097"}, + {file = "cryptography-45.0.5-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:b8fa8b0a35a9982a3c60ec79905ba5bb090fc0b9addcfd3dc2dd04267e45f25e"}, + {file = "cryptography-45.0.5-pp310-pypy310_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:14d96584701a887763384f3c47f0ca7c1cce322aa1c31172680eb596b890ec30"}, + {file = "cryptography-45.0.5-pp310-pypy310_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:57c816dfbd1659a367831baca4b775b2a5b43c003daf52e9d57e1d30bc2e1b0e"}, + {file = "cryptography-45.0.5-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:b9e38e0a83cd51e07f5a48ff9691cae95a79bea28fe4ded168a8e5c6c77e819d"}, + {file = "cryptography-45.0.5-pp311-pypy311_pp73-macosx_10_9_x86_64.whl", hash = "sha256:8c4a6ff8a30e9e3d38ac0539e9a9e02540ab3f827a3394f8852432f6b0ea152e"}, + {file = "cryptography-45.0.5-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:bd4c45986472694e5121084c6ebbd112aa919a25e783b87eb95953c9573906d6"}, + {file = "cryptography-45.0.5-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:982518cd64c54fcada9d7e5cf28eabd3ee76bd03ab18e08a48cad7e8b6f31b18"}, + {file = "cryptography-45.0.5-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:12e55281d993a793b0e883066f590c1ae1e802e3acb67f8b442e721e475e6463"}, + {file = "cryptography-45.0.5-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:5aa1e32983d4443e310f726ee4b071ab7569f58eedfdd65e9675484a4eb67bd1"}, + {file = "cryptography-45.0.5-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:e357286c1b76403dd384d938f93c46b2b058ed4dfcdce64a770f0537ed3feb6f"}, + {file = "cryptography-45.0.5.tar.gz", hash = "sha256:72e76caa004ab63accdf26023fccd1d087f6d90ec6048ff33ad0445abf7f605a"}, ] [package.dependencies] -cffi = {version = ">=1.12", markers = "platform_python_implementation != \"PyPy\""} +cffi = {version = ">=1.14", markers = "platform_python_implementation != \"PyPy\""} [package.extras] -docs = ["sphinx (>=5.3.0)", "sphinx-rtd-theme (>=3.0.0) ; python_version >= \"3.8\""] +docs = ["sphinx (>=5.3.0)", "sphinx-inline-tabs ; python_full_version >= \"3.8.0\"", "sphinx-rtd-theme (>=3.0.0) ; python_full_version >= \"3.8.0\""] docstest = ["pyenchant (>=3)", "readme-renderer (>=30.0)", "sphinxcontrib-spelling (>=7.3.1)"] -nox = ["nox (>=2024.4.15)", "nox[uv] (>=2024.3.2) ; python_version >= \"3.8\""] -pep8test = ["check-sdist ; python_version >= \"3.8\"", "click (>=8.0.1)", "mypy (>=1.4)", "ruff (>=0.3.6)"] +nox = ["nox (>=2024.4.15)", "nox[uv] (>=2024.3.2) ; python_full_version >= \"3.8.0\""] +pep8test = ["check-sdist ; python_full_version >= \"3.8.0\"", "click (>=8.0.1)", "mypy (>=1.4)", "ruff (>=0.3.6)"] sdist = ["build (>=1.0.0)"] ssh = ["bcrypt (>=3.1.5)"] -test = ["certifi (>=2024)", "cryptography-vectors (==44.0.3)", "pretend (>=0.7)", "pytest (>=7.4.0)", "pytest-benchmark (>=4.0)", "pytest-cov (>=2.10.1)", "pytest-xdist (>=3.5.0)"] +test = ["certifi (>=2024)", "cryptography-vectors (==45.0.5)", "pretend (>=0.7)", "pytest (>=7.4.0)", "pytest-benchmark (>=4.0)", "pytest-cov (>=2.10.1)", "pytest-xdist (>=3.5.0)"] test-randomorder = ["pytest-randomly"] [[package]] @@ -643,16 +666,53 @@ files = [ docs = ["ipython", "matplotlib", "numpydoc", "sphinx"] tests = ["pytest", "pytest-cov", "pytest-xdist"] +[[package]] +name = "cyclopts" +version = "3.22.2" +description = "Intuitive, easy CLIs based on type hints." +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "cyclopts-3.22.2-py3-none-any.whl", hash = "sha256:6681b0815fa2de2bccc364468fd25b15aa9617cb505c0b16ca62e2b18a57619e"}, + {file = "cyclopts-3.22.2.tar.gz", hash = "sha256:d3495231af6ae86479579777d212ddf77b113200f828badeaf401162ed87227d"}, +] + +[package.dependencies] +attrs = ">=23.1.0" +docstring-parser = {version = ">=0.15", markers = "python_version < \"4.0\""} +rich = ">=13.6.0" +rich-rst = ">=1.3.1,<2.0.0" +typing-extensions = {version = ">=4.8.0", markers = "python_version < \"3.11\""} + +[package.extras] +toml = ["tomli (>=2.0.0) ; python_version < \"3.11\""] +trio = ["trio (>=0.10.0)"] +yaml = ["pyyaml (>=6.0.1)"] + +[[package]] +name = "defusedxml" +version = "0.7.1" +description = "XML bomb protection for Python stdlib modules" +optional = true +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" +groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" +files = [ + {file = "defusedxml-0.7.1-py2.py3-none-any.whl", hash = "sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61"}, + {file = "defusedxml-0.7.1.tar.gz", hash = "sha256:1bb3032db185915b62d7c6209c5a8792be6a32ab2fedacc84e01b52c51aa3e69"}, +] + [[package]] name = "distlib" -version = "0.3.9" +version = "0.4.0" description = "Distribution utilities" optional = false python-versions = "*" groups = ["dev"] files = [ - {file = "distlib-0.3.9-py2.py3-none-any.whl", hash = "sha256:47f8c22fd27c27e25a65601af709b38e4f0a45ea4fc2e710f65755fa8caaaf87"}, - {file = "distlib-0.3.9.tar.gz", hash = "sha256:a60f20dea646b8a33f3e7772f74dc0b2d0772d2837ee1342a00645c81edf9403"}, + {file = "distlib-0.4.0-py2.py3-none-any.whl", hash = "sha256:9659f7d87e46584a30b5780e43ac7a2143098441670ff0a49d5f9034c54a6c16"}, + {file = "distlib-0.4.0.tar.gz", hash = "sha256:feec40075be03a04501a973d81f633735b4b69f98b05450592310c0f401a4e0d"}, ] [[package]] @@ -688,6 +748,30 @@ idna = ["idna (>=3.7)"] trio = ["trio (>=0.23)"] wmi = ["wmi (>=1.5.1)"] +[[package]] +name = "docstring-parser" +version = "0.16" +description = "Parse Python docstrings in reST, Google and Numpydoc format" +optional = false +python-versions = ">=3.6,<4.0" +groups = ["main"] +files = [ + {file = "docstring_parser-0.16-py3-none-any.whl", hash = "sha256:bf0a1387354d3691d102edef7ec124f219ef639982d096e26e3b60aeffa90637"}, + {file = "docstring_parser-0.16.tar.gz", hash = "sha256:538beabd0af1e2db0146b6bd3caa526c35a34d61af9fd2887f3a8a27a739aa6e"}, +] + +[[package]] +name = "docutils" +version = "0.21.2" +description = "Docutils -- Python Documentation Utilities" +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "docutils-0.21.2-py3-none-any.whl", hash = "sha256:dafca5b9e384f0e419294eb4d2ff9fa826435bf15f15b7bd45723e8ad76811b2"}, + {file = "docutils-0.21.2.tar.gz", hash = "sha256:3a6b18732edf182daa3cd12775bbb338cf5691468f91eeeb109deff6ebfa986f"}, +] + [[package]] name = "dotenv" version = "0.9.9" @@ -740,9 +824,10 @@ idna = ">=2.0.0" name = "et-xmlfile" version = "2.0.0" description = "An implementation of lxml.xmlfile for the standard library" -optional = false +optional = true python-versions = ">=3.8" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "et_xmlfile-2.0.0-py3-none-any.whl", hash = "sha256:7a91720bc756843502c3b7504c77b8fe44217c85c537d85037f0f536151b2caa"}, {file = "et_xmlfile-2.0.0.tar.gz", hash = "sha256:dab3f4764309081ce75662649be815c4c9081e88f0837825f90fd28317d4da54"}, @@ -750,30 +835,33 @@ files = [ [[package]] name = "exceptiongroup" -version = "1.2.2" +version = "1.3.0" description = "Backport of PEP 654 (exception groups)" optional = false python-versions = ">=3.7" groups = ["main", "eval", "test"] -markers = "python_version == \"3.10\"" files = [ - {file = "exceptiongroup-1.2.2-py3-none-any.whl", hash = "sha256:3111b9d131c238bec2f8f516e123e14ba243563fb135d3fe885990585aa7795b"}, - {file = "exceptiongroup-1.2.2.tar.gz", hash = "sha256:47c2edf7c6738fafb49fd34290706d1a1a2f4d1c6df275526b62cbb4aa5393cc"}, + {file = "exceptiongroup-1.3.0-py3-none-any.whl", hash = "sha256:4d111e6e0c13d0644cad6ddaa7ed0261a0b36971f6d23e7ec9b4b9097da78a10"}, + {file = "exceptiongroup-1.3.0.tar.gz", hash = "sha256:b241f5885f560bc56a59ee63ca4c6a8bfa46ae4ad651af316d4e81817bb9fd88"}, ] +markers = {eval = "python_version == \"3.10\"", test = "python_version == \"3.10\""} + +[package.dependencies] +typing-extensions = {version = ">=4.6.0", markers = "python_version < \"3.13\""} [package.extras] test = ["pytest (>=6)"] [[package]] name = "fastapi" -version = "0.115.13" +version = "0.115.14" description = "FastAPI framework, high performance, easy to learn, fast to code, ready for production" optional = false python-versions = ">=3.8" groups = ["main"] files = [ - {file = "fastapi-0.115.13-py3-none-any.whl", hash = "sha256:0a0cab59afa7bab22f5eb347f8c9864b681558c278395e94035a741fc10cd865"}, - {file = "fastapi-0.115.13.tar.gz", hash = "sha256:55d1d25c2e1e0a0a50aceb1c8705cd932def273c102bff0b1c1da88b3c6eb307"}, + {file = "fastapi-0.115.14-py3-none-any.whl", hash = "sha256:6c0c8bf9420bd58f565e585036d971872472b4f7d3f6c73b698e10cffdefb3ca"}, + {file = "fastapi-0.115.14.tar.gz", hash = "sha256:b1de15cdc1c499a4da47914db35d0e4ef8f1ce62b624e94e0e5824421df99739"}, ] [package.dependencies] @@ -799,24 +887,77 @@ standard = ["email-validator (>=2.0.0)", "fastapi-cli[standard] (>=0.0.5)", "htt [[package]] name = "fastapi-cli" -version = "0.0.7" +version = "0.0.8" description = "Run and manage FastAPI apps from the command line with FastAPI CLI. 🚀" optional = false python-versions = ">=3.8" groups = ["main"] files = [ - {file = "fastapi_cli-0.0.7-py3-none-any.whl", hash = "sha256:d549368ff584b2804336c61f192d86ddea080c11255f375959627911944804f4"}, - {file = "fastapi_cli-0.0.7.tar.gz", hash = "sha256:02b3b65956f526412515907a0793c9094abd4bfb5457b389f645b0ea6ba3605e"}, + {file = "fastapi_cli-0.0.8-py3-none-any.whl", hash = "sha256:0ea95d882c85b9219a75a65ab27e8da17dac02873e456850fa0a726e96e985eb"}, + {file = "fastapi_cli-0.0.8.tar.gz", hash = "sha256:2360f2989b1ab4a3d7fc8b3a0b20e8288680d8af2e31de7c38309934d7f8a0ee"}, ] [package.dependencies] -rich-toolkit = ">=0.11.1" +fastapi-cloud-cli = {version = ">=0.1.1", optional = true, markers = "extra == \"standard\""} +rich-toolkit = ">=0.14.8" +typer = ">=0.15.1" +uvicorn = {version = ">=0.15.0", extras = ["standard"]} + +[package.extras] +standard = ["fastapi-cloud-cli (>=0.1.1)", "uvicorn[standard] (>=0.15.0)"] +standard-no-fastapi-cloud-cli = ["uvicorn[standard] (>=0.15.0)"] + +[[package]] +name = "fastapi-cloud-cli" +version = "0.1.4" +description = "Deploy and manage FastAPI Cloud apps from the command line 🚀" +optional = false +python-versions = ">=3.8" +groups = ["main"] +files = [ + {file = "fastapi_cloud_cli-0.1.4-py3-none-any.whl", hash = "sha256:1db1ba757aa46a16a5e5dacf7cddc137ca0a3c42f65dba2b1cc6a8f24c41be42"}, + {file = "fastapi_cloud_cli-0.1.4.tar.gz", hash = "sha256:a0ab7633d71d864b4041896b3fe2f462de61546db7c52eb13e963f4d40af0eba"}, +] + +[package.dependencies] +httpx = ">=0.27.0" +pydantic = {version = ">=1.6.1", extras = ["email"]} +rich-toolkit = ">=0.14.5" +rignore = ">=0.5.1" +sentry-sdk = ">=2.20.0" typer = ">=0.12.3" uvicorn = {version = ">=0.15.0", extras = ["standard"]} [package.extras] standard = ["uvicorn[standard] (>=0.15.0)"] +[[package]] +name = "fastmcp" +version = "2.10.5" +description = "The fast, Pythonic way to build MCP servers and clients." +optional = false +python-versions = ">=3.10" +groups = ["main"] +files = [ + {file = "fastmcp-2.10.5-py3-none-any.whl", hash = "sha256:ab218f6a66b61f6f83c413d37aa18f5c30882c44c8925f39ecd02dd855826540"}, + {file = "fastmcp-2.10.5.tar.gz", hash = "sha256:f829e0b11c4d136db1d81e20e8acb19cf5108f64059482d1853f3c940326cf04"}, +] + +[package.dependencies] +authlib = ">=1.5.2" +cyclopts = ">=3.0.0" +exceptiongroup = ">=1.2.2" +httpx = ">=0.28.1" +mcp = ">=1.10.0" +openapi-pydantic = ">=0.5.1" +pydantic = {version = ">=2.11.7", extras = ["email"]} +pyperclip = ">=1.9.0" +python-dotenv = ">=1.1.0" +rich = ">=13.9.4" + +[package.extras] +websockets = ["websockets (>=15.0.1)"] + [[package]] name = "filelock" version = "3.18.0" @@ -838,9 +979,10 @@ typing = ["typing-extensions (>=4.12.2) ; python_version < \"3.11\""] name = "flatbuffers" version = "25.2.10" description = "The FlatBuffers serialization format for Python" -optional = false +optional = true python-versions = "*" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "flatbuffers-25.2.10-py2.py3-none-any.whl", hash = "sha256:ebba5f4d5ea615af3f7fd70fc310636fbb2bbd1f566ac0a23d98dd412de50051"}, {file = "flatbuffers-25.2.10.tar.gz", hash = "sha256:97e451377a41262f8d9bd4295cc836133415cc03d8cb966410a4af92eb00d26e"}, @@ -848,58 +990,58 @@ files = [ [[package]] name = "fonttools" -version = "4.58.4" +version = "4.59.0" description = "Tools to manipulate font files" optional = false python-versions = ">=3.9" groups = ["eval"] files = [ - {file = "fonttools-4.58.4-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:834542f13fee7625ad753b2db035edb674b07522fcbdd0ed9e9a9e2a1034467f"}, - {file = "fonttools-4.58.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:2e6c61ce330142525296170cd65666e46121fc0d44383cbbcfa39cf8f58383df"}, - {file = "fonttools-4.58.4-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e9c75f8faa29579c0fbf29b56ae6a3660c6c025f3b671803cb6a9caa7e4e3a98"}, - {file = "fonttools-4.58.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:88dedcedbd5549e35b2ea3db3de02579c27e62e51af56779c021e7b33caadd0e"}, - {file = "fonttools-4.58.4-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:ae80a895adab43586f4da1521d58fd4f4377cef322ee0cc205abcefa3a5effc3"}, - {file = "fonttools-4.58.4-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:0d3acc7f0d151da116e87a182aefb569cf0a3c8e0fd4c9cd0a7c1e7d3e7adb26"}, - {file = "fonttools-4.58.4-cp310-cp310-win32.whl", hash = "sha256:1244f69686008e7e8d2581d9f37eef330a73fee3843f1107993eb82c9d306577"}, - {file = "fonttools-4.58.4-cp310-cp310-win_amd64.whl", hash = "sha256:2a66c0af8a01eb2b78645af60f3b787de5fe5eb1fd8348163715b80bdbfbde1f"}, - {file = "fonttools-4.58.4-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:a3841991c9ee2dc0562eb7f23d333d34ce81e8e27c903846f0487da21e0028eb"}, - {file = "fonttools-4.58.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:3c98f91b6a9604e7ffb5ece6ea346fa617f967c2c0944228801246ed56084664"}, - {file = "fonttools-4.58.4-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ab9f891eb687ddf6a4e5f82901e00f992e18012ca97ab7acd15f13632acd14c1"}, - {file = "fonttools-4.58.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:891c5771e8f0094b7c0dc90eda8fc75e72930b32581418f2c285a9feedfd9a68"}, - {file = "fonttools-4.58.4-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:43ba4d9646045c375d22e3473b7d82b18b31ee2ac715cd94220ffab7bc2d5c1d"}, - {file = "fonttools-4.58.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:33d19f16e6d2ffd6669bda574a6589941f6c99a8d5cfb9f464038244c71555de"}, - {file = "fonttools-4.58.4-cp311-cp311-win32.whl", hash = "sha256:b59e5109b907da19dc9df1287454821a34a75f2632a491dd406e46ff432c2a24"}, - {file = "fonttools-4.58.4-cp311-cp311-win_amd64.whl", hash = "sha256:3d471a5b567a0d1648f2e148c9a8bcf00d9ac76eb89e976d9976582044cc2509"}, - {file = "fonttools-4.58.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:462211c0f37a278494e74267a994f6be9a2023d0557aaa9ecbcbfce0f403b5a6"}, - {file = "fonttools-4.58.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0c7a12fb6f769165547f00fcaa8d0df9517603ae7e04b625e5acb8639809b82d"}, - {file = "fonttools-4.58.4-cp312-cp312-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:2d42c63020a922154add0a326388a60a55504629edc3274bc273cd3806b4659f"}, - {file = "fonttools-4.58.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8f2b4e6fd45edc6805f5f2c355590b092ffc7e10a945bd6a569fc66c1d2ae7aa"}, - {file = "fonttools-4.58.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f155b927f6efb1213a79334e4cb9904d1e18973376ffc17a0d7cd43d31981f1e"}, - {file = "fonttools-4.58.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e38f687d5de97c7fb7da3e58169fb5ba349e464e141f83c3c2e2beb91d317816"}, - {file = "fonttools-4.58.4-cp312-cp312-win32.whl", hash = "sha256:636c073b4da9db053aa683db99580cac0f7c213a953b678f69acbca3443c12cc"}, - {file = "fonttools-4.58.4-cp312-cp312-win_amd64.whl", hash = "sha256:82e8470535743409b30913ba2822e20077acf9ea70acec40b10fcf5671dceb58"}, - {file = "fonttools-4.58.4-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:5f4a64846495c543796fa59b90b7a7a9dff6839bd852741ab35a71994d685c6d"}, - {file = "fonttools-4.58.4-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e80661793a5d4d7ad132a2aa1eae2e160fbdbb50831a0edf37c7c63b2ed36574"}, - {file = "fonttools-4.58.4-cp313-cp313-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:fe5807fc64e4ba5130f1974c045a6e8d795f3b7fb6debfa511d1773290dbb76b"}, - {file = "fonttools-4.58.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b610b9bef841cb8f4b50472494158b1e347d15cad56eac414c722eda695a6cfd"}, - {file = "fonttools-4.58.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2daa7f0e213c38f05f054eb5e1730bd0424aebddbeac094489ea1585807dd187"}, - {file = "fonttools-4.58.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:66cccb6c0b944496b7f26450e9a66e997739c513ffaac728d24930df2fd9d35b"}, - {file = "fonttools-4.58.4-cp313-cp313-win32.whl", hash = "sha256:94d2aebb5ca59a5107825520fde596e344652c1f18170ef01dacbe48fa60c889"}, - {file = "fonttools-4.58.4-cp313-cp313-win_amd64.whl", hash = "sha256:b554bd6e80bba582fd326ddab296e563c20c64dca816d5e30489760e0c41529f"}, - {file = "fonttools-4.58.4-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:ca773fe7812e4e1197ee4e63b9691e89650ab55f679e12ac86052d2fe0d152cd"}, - {file = "fonttools-4.58.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:e31289101221910f44245472e02b1a2f7d671c6d06a45c07b354ecb25829ad92"}, - {file = "fonttools-4.58.4-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:90c9e3c01475bb9602cb617f69f02c4ba7ab7784d93f0b0d685e84286f4c1a10"}, - {file = "fonttools-4.58.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e00a826f2bc745a010341ac102082fe5e3fb9f0861b90ed9ff32277598813711"}, - {file = "fonttools-4.58.4-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:bc75e72e9d2a4ad0935c59713bd38679d51c6fefab1eadde80e3ed4c2a11ea84"}, - {file = "fonttools-4.58.4-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:f57a795e540059ce3de68508acfaaf177899b39c36ef0a2833b2308db98c71f1"}, - {file = "fonttools-4.58.4-cp39-cp39-win32.whl", hash = "sha256:a7d04f64c88b48ede655abcf76f2b2952f04933567884d99be7c89e0a4495131"}, - {file = "fonttools-4.58.4-cp39-cp39-win_amd64.whl", hash = "sha256:5a8bc5dfd425c89b1c38380bc138787b0a830f761b82b37139aa080915503b69"}, - {file = "fonttools-4.58.4-py3-none-any.whl", hash = "sha256:a10ce13a13f26cbb9f37512a4346bb437ad7e002ff6fa966a7ce7ff5ac3528bd"}, - {file = "fonttools-4.58.4.tar.gz", hash = "sha256:928a8009b9884ed3aae17724b960987575155ca23c6f0b8146e400cc9e0d44ba"}, + {file = "fonttools-4.59.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:524133c1be38445c5c0575eacea42dbd44374b310b1ffc4b60ff01d881fabb96"}, + {file = "fonttools-4.59.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:21e606b2d38fed938dde871c5736822dd6bda7a4631b92e509a1f5cd1b90c5df"}, + {file = "fonttools-4.59.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e93df708c69a193fc7987192f94df250f83f3851fda49413f02ba5dded639482"}, + {file = "fonttools-4.59.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:62224a9bb85b4b66d1b46d45cbe43d71cbf8f527d332b177e3b96191ffbc1e64"}, + {file = "fonttools-4.59.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:b8974b2a266b54c96709bd5e239979cddfd2dbceed331aa567ea1d7c4a2202db"}, + {file = "fonttools-4.59.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:209b75943d158f610b78320eacb5539aa9e920bee2c775445b2846c65d20e19d"}, + {file = "fonttools-4.59.0-cp310-cp310-win32.whl", hash = "sha256:4c908a7036f0f3677f8afa577bcd973e3e20ddd2f7c42a33208d18bee95cdb6f"}, + {file = "fonttools-4.59.0-cp310-cp310-win_amd64.whl", hash = "sha256:8b4309a2775e4feee7356e63b163969a215d663399cce1b3d3b65e7ec2d9680e"}, + {file = "fonttools-4.59.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:841b2186adce48903c0fef235421ae21549020eca942c1da773ac380b056ab3c"}, + {file = "fonttools-4.59.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:9bcc1e77fbd1609198966ded6b2a9897bd6c6bcbd2287a2fc7d75f1a254179c5"}, + {file = "fonttools-4.59.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:37c377f7cb2ab2eca8a0b319c68146d34a339792f9420fca6cd49cf28d370705"}, + {file = "fonttools-4.59.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fa39475eaccb98f9199eccfda4298abaf35ae0caec676ffc25b3a5e224044464"}, + {file = "fonttools-4.59.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d3972b13148c1d1fbc092b27678a33b3080d1ac0ca305742b0119b75f9e87e38"}, + {file = "fonttools-4.59.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:a408c3c51358c89b29cfa5317cf11518b7ce5de1717abb55c5ae2d2921027de6"}, + {file = "fonttools-4.59.0-cp311-cp311-win32.whl", hash = "sha256:6770d7da00f358183d8fd5c4615436189e4f683bdb6affb02cad3d221d7bb757"}, + {file = "fonttools-4.59.0-cp311-cp311-win_amd64.whl", hash = "sha256:84fc186980231a287b28560d3123bd255d3c6b6659828c642b4cf961e2b923d0"}, + {file = "fonttools-4.59.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:f9b3a78f69dcbd803cf2fb3f972779875b244c1115481dfbdd567b2c22b31f6b"}, + {file = "fonttools-4.59.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:57bb7e26928573ee7c6504f54c05860d867fd35e675769f3ce01b52af38d48e2"}, + {file = "fonttools-4.59.0-cp312-cp312-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:4536f2695fe5c1ffb528d84a35a7d3967e5558d2af58b4775e7ab1449d65767b"}, + {file = "fonttools-4.59.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:885bde7d26e5b40e15c47bd5def48b38cbd50830a65f98122a8fb90962af7cd1"}, + {file = "fonttools-4.59.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6801aeddb6acb2c42eafa45bc1cb98ba236871ae6f33f31e984670b749a8e58e"}, + {file = "fonttools-4.59.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:31003b6a10f70742a63126b80863ab48175fb8272a18ca0846c0482968f0588e"}, + {file = "fonttools-4.59.0-cp312-cp312-win32.whl", hash = "sha256:fbce6dae41b692a5973d0f2158f782b9ad05babc2c2019a970a1094a23909b1b"}, + {file = "fonttools-4.59.0-cp312-cp312-win_amd64.whl", hash = "sha256:332bfe685d1ac58ca8d62b8d6c71c2e52a6c64bc218dc8f7825c9ea51385aa01"}, + {file = "fonttools-4.59.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:78813b49d749e1bb4db1c57f2d4d7e6db22c253cb0a86ad819f5dc197710d4b2"}, + {file = "fonttools-4.59.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:401b1941ce37e78b8fd119b419b617277c65ae9417742a63282257434fd68ea2"}, + {file = "fonttools-4.59.0-cp313-cp313-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:efd7e6660674e234e29937bc1481dceb7e0336bfae75b856b4fb272b5093c5d4"}, + {file = "fonttools-4.59.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:51ab1ff33c19e336c02dee1e9fd1abd974a4ca3d8f7eef2a104d0816a241ce97"}, + {file = "fonttools-4.59.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a9bf8adc9e1f3012edc8f09b08336272aec0c55bc677422273e21280db748f7c"}, + {file = "fonttools-4.59.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:37e01c6ec0c98599778c2e688350d624fa4770fbd6144551bd5e032f1199171c"}, + {file = "fonttools-4.59.0-cp313-cp313-win32.whl", hash = "sha256:70d6b3ceaa9cc5a6ac52884f3b3d9544e8e231e95b23f138bdb78e6d4dc0eae3"}, + {file = "fonttools-4.59.0-cp313-cp313-win_amd64.whl", hash = "sha256:26731739daa23b872643f0e4072d5939960237d540c35c14e6a06d47d71ca8fe"}, + {file = "fonttools-4.59.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:8d77f92438daeaddc05682f0f3dac90c5b9829bcac75b57e8ce09cb67786073c"}, + {file = "fonttools-4.59.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:60f6665579e909b618282f3c14fa0b80570fbf1ee0e67678b9a9d43aa5d67a37"}, + {file = "fonttools-4.59.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:169b99a2553a227f7b5fea8d9ecd673aa258617f466b2abc6091fe4512a0dcd0"}, + {file = "fonttools-4.59.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:052444a5d0151878e87e3e512a1aa1a0ab35ee4c28afde0a778e23b0ace4a7de"}, + {file = "fonttools-4.59.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:d40dcf533ca481355aa7b682e9e079f766f35715defa4929aeb5597f9604272e"}, + {file = "fonttools-4.59.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:b818db35879d2edf7f46c7e729c700a0bce03b61b9412f5a7118406687cb151d"}, + {file = "fonttools-4.59.0-cp39-cp39-win32.whl", hash = "sha256:2e7cf8044ce2598bb87e44ba1d2c6e45d7a8decf56055b92906dc53f67c76d64"}, + {file = "fonttools-4.59.0-cp39-cp39-win_amd64.whl", hash = "sha256:902425f5afe28572d65d2bf9c33edd5265c612ff82c69e6f83ea13eafc0dcbea"}, + {file = "fonttools-4.59.0-py3-none-any.whl", hash = "sha256:241313683afd3baacb32a6bd124d0bce7404bc5280e12e291bae1b9bba28711d"}, + {file = "fonttools-4.59.0.tar.gz", hash = "sha256:be392ec3529e2f57faa28709d60723a763904f71a2b63aabe14fee6648fe3b14"}, ] [package.extras] -all = ["brotli (>=1.0.1) ; platform_python_implementation == \"CPython\"", "brotlicffi (>=0.8.0) ; platform_python_implementation != \"CPython\"", "fs (>=2.2.0,<3)", "lxml (>=4.0)", "lz4 (>=1.7.4.2)", "matplotlib", "munkres ; platform_python_implementation == \"PyPy\"", "pycairo", "scipy ; platform_python_implementation != \"PyPy\"", "skia-pathops (>=0.5.0)", "sympy", "uharfbuzz (>=0.23.0)", "unicodedata2 (>=15.1.0) ; python_version <= \"3.12\"", "xattr ; sys_platform == \"darwin\"", "zopfli (>=0.1.4)"] +all = ["brotli (>=1.0.1) ; platform_python_implementation == \"CPython\"", "brotlicffi (>=0.8.0) ; platform_python_implementation != \"CPython\"", "lxml (>=4.0)", "lz4 (>=1.7.4.2)", "matplotlib", "munkres ; platform_python_implementation == \"PyPy\"", "pycairo", "scipy ; platform_python_implementation != \"PyPy\"", "skia-pathops (>=0.5.0)", "sympy", "uharfbuzz (>=0.23.0)", "unicodedata2 (>=15.1.0) ; python_version <= \"3.12\"", "xattr ; sys_platform == \"darwin\"", "zopfli (>=0.1.4)"] graphite = ["lz4 (>=1.7.4.2)"] interpolatable = ["munkres ; platform_python_implementation == \"PyPy\"", "pycairo", "scipy ; platform_python_implementation != \"PyPy\""] lxml = ["lxml (>=4.0)"] @@ -908,20 +1050,19 @@ plot = ["matplotlib"] repacker = ["uharfbuzz (>=0.23.0)"] symfont = ["sympy"] type1 = ["xattr ; sys_platform == \"darwin\""] -ufo = ["fs (>=2.2.0,<3)"] unicode = ["unicodedata2 (>=15.1.0) ; python_version <= \"3.12\""] woff = ["brotli (>=1.0.1) ; platform_python_implementation == \"CPython\"", "brotlicffi (>=0.8.0) ; platform_python_implementation != \"CPython\"", "zopfli (>=0.1.4)"] [[package]] name = "fsspec" -version = "2025.3.2" +version = "2025.7.0" description = "File-system specification" optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "fsspec-2025.3.2-py3-none-any.whl", hash = "sha256:2daf8dc3d1dfa65b6aa37748d112773a7a08416f6c70d96b264c96476ecaf711"}, - {file = "fsspec-2025.3.2.tar.gz", hash = "sha256:e52c77ef398680bbd6a98c0e628fbc469491282981209907bbc8aea76a04fdc6"}, + {file = "fsspec-2025.7.0-py3-none-any.whl", hash = "sha256:8b012e39f63c7d5f10474de957f3ab793b47b45ae7d39f2fb735f8bbe25c0e21"}, + {file = "fsspec-2025.7.0.tar.gz", hash = "sha256:786120687ffa54b8283d942929540d8bc5ccfa820deb555a2b5d0ed2b737bf58"}, ] [package.extras] @@ -929,7 +1070,7 @@ abfs = ["adlfs"] adl = ["adlfs"] arrow = ["pyarrow (>=1)"] dask = ["dask", "distributed"] -dev = ["pre-commit", "ruff"] +dev = ["pre-commit", "ruff (>=0.5)"] doc = ["numpydoc", "sphinx", "sphinx-design", "sphinx-rtd-theme", "yarl"] dropbox = ["dropbox", "dropboxdrivefs", "requests"] full = ["adlfs", "aiohttp (!=4.0.0a0,!=4.0.0a1)", "dask", "distributed", "dropbox", "dropboxdrivefs", "fusepy", "gcsfs", "libarchive-c", "ocifs", "panel", "paramiko", "pyarrow (>=1)", "pygit2", "requests", "s3fs", "smbprotocol", "tqdm"] @@ -949,7 +1090,7 @@ smb = ["smbprotocol"] ssh = ["paramiko"] test = ["aiohttp (!=4.0.0a0,!=4.0.0a1)", "numpy", "pytest", "pytest-asyncio (!=0.22.0)", "pytest-benchmark", "pytest-cov", "pytest-mock", "pytest-recording", "pytest-rerunfailures", "requests"] test-downstream = ["aiobotocore (>=2.5.4,<3.0.0)", "dask[dataframe,test]", "moto[server] (>4,<5)", "pytest-timeout", "xarray"] -test-full = ["adlfs", "aiohttp (!=4.0.0a0,!=4.0.0a1)", "cloudpickle", "dask", "distributed", "dropbox", "dropboxdrivefs", "fastparquet", "fusepy", "gcsfs", "jinja2", "kerchunk", "libarchive-c", "lz4", "notebook", "numpy", "ocifs", "pandas", "panel", "paramiko", "pyarrow", "pyarrow (>=1)", "pyftpdlib", "pygit2", "pytest", "pytest-asyncio (!=0.22.0)", "pytest-benchmark", "pytest-cov", "pytest-mock", "pytest-recording", "pytest-rerunfailures", "python-snappy", "requests", "smbprotocol", "tqdm", "urllib3", "zarr", "zstandard"] +test-full = ["adlfs", "aiohttp (!=4.0.0a0,!=4.0.0a1)", "cloudpickle", "dask", "distributed", "dropbox", "dropboxdrivefs", "fastparquet", "fusepy", "gcsfs", "jinja2", "kerchunk", "libarchive-c", "lz4", "notebook", "numpy", "ocifs", "pandas", "panel", "paramiko", "pyarrow", "pyarrow (>=1)", "pyftpdlib", "pygit2", "pytest", "pytest-asyncio (!=0.22.0)", "pytest-benchmark", "pytest-cov", "pytest-mock", "pytest-recording", "pytest-rerunfailures", "python-snappy", "requests", "smbprotocol", "tqdm", "urllib3", "zarr", "zstandard ; python_version < \"3.14\""] tqdm = ["tqdm"] [[package]] @@ -1023,67 +1164,68 @@ test = ["objgraph", "psutil"] [[package]] name = "grpcio" -version = "1.71.0" +version = "1.73.1" description = "HTTP/2-based RPC framework" optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "grpcio-1.71.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:c200cb6f2393468142eb50ab19613229dcc7829b5ccee8b658a36005f6669fdd"}, - {file = "grpcio-1.71.0-cp310-cp310-macosx_12_0_universal2.whl", hash = "sha256:b2266862c5ad664a380fbbcdbdb8289d71464c42a8c29053820ee78ba0119e5d"}, - {file = "grpcio-1.71.0-cp310-cp310-manylinux_2_17_aarch64.whl", hash = "sha256:0ab8b2864396663a5b0b0d6d79495657ae85fa37dcb6498a2669d067c65c11ea"}, - {file = "grpcio-1.71.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c30f393f9d5ff00a71bb56de4aa75b8fe91b161aeb61d39528db6b768d7eac69"}, - {file = "grpcio-1.71.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f250ff44843d9a0615e350c77f890082102a0318d66a99540f54769c8766ab73"}, - {file = "grpcio-1.71.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:e6d8de076528f7c43a2f576bc311799f89d795aa6c9b637377cc2b1616473804"}, - {file = "grpcio-1.71.0-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:9b91879d6da1605811ebc60d21ab6a7e4bae6c35f6b63a061d61eb818c8168f6"}, - {file = "grpcio-1.71.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:f71574afdf944e6652203cd1badcda195b2a27d9c83e6d88dc1ce3cfb73b31a5"}, - {file = "grpcio-1.71.0-cp310-cp310-win32.whl", hash = "sha256:8997d6785e93308f277884ee6899ba63baafa0dfb4729748200fcc537858a509"}, - {file = "grpcio-1.71.0-cp310-cp310-win_amd64.whl", hash = "sha256:7d6ac9481d9d0d129224f6d5934d5832c4b1cddb96b59e7eba8416868909786a"}, - {file = "grpcio-1.71.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:d6aa986318c36508dc1d5001a3ff169a15b99b9f96ef5e98e13522c506b37eef"}, - {file = "grpcio-1.71.0-cp311-cp311-macosx_10_14_universal2.whl", hash = "sha256:d2c170247315f2d7e5798a22358e982ad6eeb68fa20cf7a820bb74c11f0736e7"}, - {file = "grpcio-1.71.0-cp311-cp311-manylinux_2_17_aarch64.whl", hash = "sha256:e6f83a583ed0a5b08c5bc7a3fe860bb3c2eac1f03f1f63e0bc2091325605d2b7"}, - {file = "grpcio-1.71.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4be74ddeeb92cc87190e0e376dbc8fc7736dbb6d3d454f2fa1f5be1dee26b9d7"}, - {file = "grpcio-1.71.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4dd0dfbe4d5eb1fcfec9490ca13f82b089a309dc3678e2edabc144051270a66e"}, - {file = "grpcio-1.71.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:a2242d6950dc892afdf9e951ed7ff89473aaf744b7d5727ad56bdaace363722b"}, - {file = "grpcio-1.71.0-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:0fa05ee31a20456b13ae49ad2e5d585265f71dd19fbd9ef983c28f926d45d0a7"}, - {file = "grpcio-1.71.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:3d081e859fb1ebe176de33fc3adb26c7d46b8812f906042705346b314bde32c3"}, - {file = "grpcio-1.71.0-cp311-cp311-win32.whl", hash = "sha256:d6de81c9c00c8a23047136b11794b3584cdc1460ed7cbc10eada50614baa1444"}, - {file = "grpcio-1.71.0-cp311-cp311-win_amd64.whl", hash = "sha256:24e867651fc67717b6f896d5f0cac0ec863a8b5fb7d6441c2ab428f52c651c6b"}, - {file = "grpcio-1.71.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:0ff35c8d807c1c7531d3002be03221ff9ae15712b53ab46e2a0b4bb271f38537"}, - {file = "grpcio-1.71.0-cp312-cp312-macosx_10_14_universal2.whl", hash = "sha256:b78a99cd1ece4be92ab7c07765a0b038194ded2e0a26fd654591ee136088d8d7"}, - {file = "grpcio-1.71.0-cp312-cp312-manylinux_2_17_aarch64.whl", hash = "sha256:dc1a1231ed23caac1de9f943d031f1bc38d0f69d2a3b243ea0d664fc1fbd7fec"}, - {file = "grpcio-1.71.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e6beeea5566092c5e3c4896c6d1d307fb46b1d4bdf3e70c8340b190a69198594"}, - {file = "grpcio-1.71.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d5170929109450a2c031cfe87d6716f2fae39695ad5335d9106ae88cc32dc84c"}, - {file = "grpcio-1.71.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:5b08d03ace7aca7b2fadd4baf291139b4a5f058805a8327bfe9aece7253b6d67"}, - {file = "grpcio-1.71.0-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:f903017db76bf9cc2b2d8bdd37bf04b505bbccad6be8a81e1542206875d0e9db"}, - {file = "grpcio-1.71.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:469f42a0b410883185eab4689060a20488a1a0a00f8bbb3cbc1061197b4c5a79"}, - {file = "grpcio-1.71.0-cp312-cp312-win32.whl", hash = "sha256:ad9f30838550695b5eb302add33f21f7301b882937460dd24f24b3cc5a95067a"}, - {file = "grpcio-1.71.0-cp312-cp312-win_amd64.whl", hash = "sha256:652350609332de6dac4ece254e5d7e1ff834e203d6afb769601f286886f6f3a8"}, - {file = "grpcio-1.71.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:cebc1b34ba40a312ab480ccdb396ff3c529377a2fce72c45a741f7215bfe8379"}, - {file = "grpcio-1.71.0-cp313-cp313-macosx_10_14_universal2.whl", hash = "sha256:85da336e3649a3d2171e82f696b5cad2c6231fdd5bad52616476235681bee5b3"}, - {file = "grpcio-1.71.0-cp313-cp313-manylinux_2_17_aarch64.whl", hash = "sha256:f9a412f55bb6e8f3bb000e020dbc1e709627dcb3a56f6431fa7076b4c1aab0db"}, - {file = "grpcio-1.71.0-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:47be9584729534660416f6d2a3108aaeac1122f6b5bdbf9fd823e11fe6fbaa29"}, - {file = "grpcio-1.71.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7c9c80ac6091c916db81131d50926a93ab162a7e97e4428ffc186b6e80d6dda4"}, - {file = "grpcio-1.71.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:789d5e2a3a15419374b7b45cd680b1e83bbc1e52b9086e49308e2c0b5bbae6e3"}, - {file = "grpcio-1.71.0-cp313-cp313-musllinux_1_1_i686.whl", hash = "sha256:1be857615e26a86d7363e8a163fade914595c81fec962b3d514a4b1e8760467b"}, - {file = "grpcio-1.71.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:a76d39b5fafd79ed604c4be0a869ec3581a172a707e2a8d7a4858cb05a5a7637"}, - {file = "grpcio-1.71.0-cp313-cp313-win32.whl", hash = "sha256:74258dce215cb1995083daa17b379a1a5a87d275387b7ffe137f1d5131e2cfbb"}, - {file = "grpcio-1.71.0-cp313-cp313-win_amd64.whl", hash = "sha256:22c3bc8d488c039a199f7a003a38cb7635db6656fa96437a8accde8322ce2366"}, - {file = "grpcio-1.71.0-cp39-cp39-linux_armv7l.whl", hash = "sha256:c6a0a28450c16809f94e0b5bfe52cabff63e7e4b97b44123ebf77f448534d07d"}, - {file = "grpcio-1.71.0-cp39-cp39-macosx_10_14_universal2.whl", hash = "sha256:a371e6b6a5379d3692cc4ea1cb92754d2a47bdddeee755d3203d1f84ae08e03e"}, - {file = "grpcio-1.71.0-cp39-cp39-manylinux_2_17_aarch64.whl", hash = "sha256:39983a9245d37394fd59de71e88c4b295eb510a3555e0a847d9965088cdbd033"}, - {file = "grpcio-1.71.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:9182e0063112e55e74ee7584769ec5a0b4f18252c35787f48738627e23a62b97"}, - {file = "grpcio-1.71.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:693bc706c031aeb848849b9d1c6b63ae6bcc64057984bb91a542332b75aa4c3d"}, - {file = "grpcio-1.71.0-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:20e8f653abd5ec606be69540f57289274c9ca503ed38388481e98fa396ed0b41"}, - {file = "grpcio-1.71.0-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:8700a2a57771cc43ea295296330daaddc0d93c088f0a35cc969292b6db959bf3"}, - {file = "grpcio-1.71.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:d35a95f05a8a2cbe8e02be137740138b3b2ea5f80bd004444e4f9a1ffc511e32"}, - {file = "grpcio-1.71.0-cp39-cp39-win32.whl", hash = "sha256:f9c30c464cb2ddfbc2ddf9400287701270fdc0f14be5f08a1e3939f1e749b455"}, - {file = "grpcio-1.71.0-cp39-cp39-win_amd64.whl", hash = "sha256:63e41b91032f298b3e973b3fa4093cbbc620c875e2da7b93e249d4728b54559a"}, - {file = "grpcio-1.71.0.tar.gz", hash = "sha256:2b85f7820475ad3edec209d3d89a7909ada16caab05d3f2e08a7e8ae3200a55c"}, -] + {file = "grpcio-1.73.1-cp310-cp310-linux_armv7l.whl", hash = "sha256:2d70f4ddd0a823436c2624640570ed6097e40935c9194482475fe8e3d9754d55"}, + {file = "grpcio-1.73.1-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:3841a8a5a66830261ab6a3c2a3dc539ed84e4ab019165f77b3eeb9f0ba621f26"}, + {file = "grpcio-1.73.1-cp310-cp310-manylinux_2_17_aarch64.whl", hash = "sha256:628c30f8e77e0258ab788750ec92059fc3d6628590fb4b7cea8c102503623ed7"}, + {file = "grpcio-1.73.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:67a0468256c9db6d5ecb1fde4bf409d016f42cef649323f0a08a72f352d1358b"}, + {file = "grpcio-1.73.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:68b84d65bbdebd5926eb5c53b0b9ec3b3f83408a30e4c20c373c5337b4219ec5"}, + {file = "grpcio-1.73.1-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:c54796ca22b8349cc594d18b01099e39f2b7ffb586ad83217655781a350ce4da"}, + {file = "grpcio-1.73.1-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:75fc8e543962ece2f7ecd32ada2d44c0c8570ae73ec92869f9af8b944863116d"}, + {file = "grpcio-1.73.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:6a6037891cd2b1dd1406b388660522e1565ed340b1fea2955b0234bdd941a862"}, + {file = "grpcio-1.73.1-cp310-cp310-win32.whl", hash = "sha256:cce7265b9617168c2d08ae570fcc2af4eaf72e84f8c710ca657cc546115263af"}, + {file = "grpcio-1.73.1-cp310-cp310-win_amd64.whl", hash = "sha256:6a2b372e65fad38842050943f42ce8fee00c6f2e8ea4f7754ba7478d26a356ee"}, + {file = "grpcio-1.73.1-cp311-cp311-linux_armv7l.whl", hash = "sha256:ba2cea9f7ae4bc21f42015f0ec98f69ae4179848ad744b210e7685112fa507a1"}, + {file = "grpcio-1.73.1-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:d74c3f4f37b79e746271aa6cdb3a1d7e4432aea38735542b23adcabaaee0c097"}, + {file = "grpcio-1.73.1-cp311-cp311-manylinux_2_17_aarch64.whl", hash = "sha256:5b9b1805a7d61c9e90541cbe8dfe0a593dfc8c5c3a43fe623701b6a01b01d710"}, + {file = "grpcio-1.73.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b3215f69a0670a8cfa2ab53236d9e8026bfb7ead5d4baabe7d7dc11d30fda967"}, + {file = "grpcio-1.73.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bc5eccfd9577a5dc7d5612b2ba90cca4ad14c6d949216c68585fdec9848befb1"}, + {file = "grpcio-1.73.1-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:dc7d7fd520614fce2e6455ba89791458020a39716951c7c07694f9dbae28e9c0"}, + {file = "grpcio-1.73.1-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:105492124828911f85127e4825d1c1234b032cb9d238567876b5515d01151379"}, + {file = "grpcio-1.73.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:610e19b04f452ba6f402ac9aa94eb3d21fbc94553368008af634812c4a85a99e"}, + {file = "grpcio-1.73.1-cp311-cp311-win32.whl", hash = "sha256:d60588ab6ba0ac753761ee0e5b30a29398306401bfbceffe7d68ebb21193f9d4"}, + {file = "grpcio-1.73.1-cp311-cp311-win_amd64.whl", hash = "sha256:6957025a4608bb0a5ff42abd75bfbb2ed99eda29d5992ef31d691ab54b753643"}, + {file = "grpcio-1.73.1-cp312-cp312-linux_armv7l.whl", hash = "sha256:921b25618b084e75d424a9f8e6403bfeb7abef074bb6c3174701e0f2542debcf"}, + {file = "grpcio-1.73.1-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:277b426a0ed341e8447fbf6c1d6b68c952adddf585ea4685aa563de0f03df887"}, + {file = "grpcio-1.73.1-cp312-cp312-manylinux_2_17_aarch64.whl", hash = "sha256:96c112333309493c10e118d92f04594f9055774757f5d101b39f8150f8c25582"}, + {file = "grpcio-1.73.1-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f48e862aed925ae987eb7084409a80985de75243389dc9d9c271dd711e589918"}, + {file = "grpcio-1.73.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:83a6c2cce218e28f5040429835fa34a29319071079e3169f9543c3fbeff166d2"}, + {file = "grpcio-1.73.1-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:65b0458a10b100d815a8426b1442bd17001fdb77ea13665b2f7dc9e8587fdc6b"}, + {file = "grpcio-1.73.1-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:0a9f3ea8dce9eae9d7cb36827200133a72b37a63896e0e61a9d5ec7d61a59ab1"}, + {file = "grpcio-1.73.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:de18769aea47f18e782bf6819a37c1c528914bfd5683b8782b9da356506190c8"}, + {file = "grpcio-1.73.1-cp312-cp312-win32.whl", hash = "sha256:24e06a5319e33041e322d32c62b1e728f18ab8c9dbc91729a3d9f9e3ed336642"}, + {file = "grpcio-1.73.1-cp312-cp312-win_amd64.whl", hash = "sha256:303c8135d8ab176f8038c14cc10d698ae1db9c480f2b2823f7a987aa2a4c5646"}, + {file = "grpcio-1.73.1-cp313-cp313-linux_armv7l.whl", hash = "sha256:b310824ab5092cf74750ebd8a8a8981c1810cb2b363210e70d06ef37ad80d4f9"}, + {file = "grpcio-1.73.1-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:8f5a6df3fba31a3485096ac85b2e34b9666ffb0590df0cd044f58694e6a1f6b5"}, + {file = "grpcio-1.73.1-cp313-cp313-manylinux_2_17_aarch64.whl", hash = "sha256:052e28fe9c41357da42250a91926a3e2f74c046575c070b69659467ca5aa976b"}, + {file = "grpcio-1.73.1-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1c0bf15f629b1497436596b1cbddddfa3234273490229ca29561209778ebe182"}, + {file = "grpcio-1.73.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0ab860d5bfa788c5a021fba264802e2593688cd965d1374d31d2b1a34cacd854"}, + {file = "grpcio-1.73.1-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:ad1d958c31cc91ab050bd8a91355480b8e0683e21176522bacea225ce51163f2"}, + {file = "grpcio-1.73.1-cp313-cp313-musllinux_1_1_i686.whl", hash = "sha256:f43ffb3bd415c57224c7427bfb9e6c46a0b6e998754bfa0d00f408e1873dcbb5"}, + {file = "grpcio-1.73.1-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:686231cdd03a8a8055f798b2b54b19428cdf18fa1549bee92249b43607c42668"}, + {file = "grpcio-1.73.1-cp313-cp313-win32.whl", hash = "sha256:89018866a096e2ce21e05eabed1567479713ebe57b1db7cbb0f1e3b896793ba4"}, + {file = "grpcio-1.73.1-cp313-cp313-win_amd64.whl", hash = "sha256:4a68f8c9966b94dff693670a5cf2b54888a48a5011c5d9ce2295a1a1465ee84f"}, + {file = "grpcio-1.73.1-cp39-cp39-linux_armv7l.whl", hash = "sha256:b4adc97d2d7f5c660a5498bda978ebb866066ad10097265a5da0511323ae9f50"}, + {file = "grpcio-1.73.1-cp39-cp39-macosx_11_0_universal2.whl", hash = "sha256:c45a28a0cfb6ddcc7dc50a29de44ecac53d115c3388b2782404218db51cb2df3"}, + {file = "grpcio-1.73.1-cp39-cp39-manylinux_2_17_aarch64.whl", hash = "sha256:10af9f2ab98a39f5b6c1896c6fc2036744b5b41d12739d48bed4c3e15b6cf900"}, + {file = "grpcio-1.73.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:45cf17dcce5ebdb7b4fe9e86cb338fa99d7d1bb71defc78228e1ddf8d0de8cbb"}, + {file = "grpcio-1.73.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1c502c2e950fc7e8bf05c047e8a14522ef7babac59abbfde6dbf46b7a0d9c71e"}, + {file = "grpcio-1.73.1-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:6abfc0f9153dc4924536f40336f88bd4fe7bd7494f028675e2e04291b8c2c62a"}, + {file = "grpcio-1.73.1-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:ed451a0e39c8e51eb1612b78686839efd1a920666d1666c1adfdb4fd51680c0f"}, + {file = "grpcio-1.73.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:07f08705a5505c9b5b0cbcbabafb96462b5a15b7236bbf6bbcc6b0b91e1cbd7e"}, + {file = "grpcio-1.73.1-cp39-cp39-win32.whl", hash = "sha256:ad5c958cc3d98bb9d71714dc69f1c13aaf2f4b53e29d4cc3f1501ef2e4d129b2"}, + {file = "grpcio-1.73.1-cp39-cp39-win_amd64.whl", hash = "sha256:42f0660bce31b745eb9d23f094a332d31f210dcadd0fc8e5be7e4c62a87ce86b"}, + {file = "grpcio-1.73.1.tar.gz", hash = "sha256:7fce2cd1c0c1116cf3850564ebfc3264fba75d3c74a7414373f1238ea365ef87"}, +] +markers = {main = "extra == \"all\""} [package.extras] -protobuf = ["grpcio-tools (>=1.71.0)"] +protobuf = ["grpcio-tools (>=1.73.1)"] [[package]] name = "h11" @@ -1108,11 +1250,34 @@ files = [ {file = "h2-4.2.0-py3-none-any.whl", hash = "sha256:479a53ad425bb29af087f3458a61d30780bc818e4ebcf01f0b536ba916462ed0"}, {file = "h2-4.2.0.tar.gz", hash = "sha256:c8a52129695e88b1a0578d8d2cc6842bbd79128ac685463b887ee278126ad01f"}, ] +markers = {main = "extra == \"all\""} [package.dependencies] hpack = ">=4.1,<5" hyperframe = ">=6.1,<7" +[[package]] +name = "hf-xet" +version = "1.1.5" +description = "Fast transfer of large files with the Hugging Face Hub." +optional = false +python-versions = ">=3.8" +groups = ["main", "eval"] +markers = "platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"arm64\" or platform_machine == \"aarch64\"" +files = [ + {file = "hf_xet-1.1.5-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:f52c2fa3635b8c37c7764d8796dfa72706cc4eded19d638331161e82b0792e23"}, + {file = "hf_xet-1.1.5-cp37-abi3-macosx_11_0_arm64.whl", hash = "sha256:9fa6e3ee5d61912c4a113e0708eaaef987047616465ac7aa30f7121a48fc1af8"}, + {file = "hf_xet-1.1.5-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fc874b5c843e642f45fd85cda1ce599e123308ad2901ead23d3510a47ff506d1"}, + {file = "hf_xet-1.1.5-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:dbba1660e5d810bd0ea77c511a99e9242d920790d0e63c0e4673ed36c4022d18"}, + {file = "hf_xet-1.1.5-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:ab34c4c3104133c495785d5d8bba3b1efc99de52c02e759cf711a91fd39d3a14"}, + {file = "hf_xet-1.1.5-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:83088ecea236d5113de478acb2339f92c95b4fb0462acaa30621fac02f5a534a"}, + {file = "hf_xet-1.1.5-cp37-abi3-win_amd64.whl", hash = "sha256:73e167d9807d166596b4b2f0b585c6d5bd84a26dea32843665a8b58f6edba245"}, + {file = "hf_xet-1.1.5.tar.gz", hash = "sha256:69ebbcfd9ec44fdc2af73441619eeb06b94ee34511bbcf57cd423820090f5694"}, +] + +[package.extras] +tests = ["pytest"] + [[package]] name = "hpack" version = "4.1.0" @@ -1124,6 +1289,7 @@ files = [ {file = "hpack-4.1.0-py3-none-any.whl", hash = "sha256:157ac792668d995c657d93111f46b4535ed114f0c9c8d672271bbec7eae1b496"}, {file = "hpack-4.1.0.tar.gz", hash = "sha256:ec5eca154f7056aa06f196a557655c5b009b382873ac8d1e66e79e87535f1dca"}, ] +markers = {main = "extra == \"all\""} [[package]] name = "httpcore" @@ -1229,21 +1395,34 @@ http2 = ["h2 (>=3,<5)"] socks = ["socksio (==1.*)"] zstd = ["zstandard (>=0.18.0)"] +[[package]] +name = "httpx-sse" +version = "0.4.1" +description = "Consume Server-Sent Event (SSE) messages with HTTPX." +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "httpx_sse-0.4.1-py3-none-any.whl", hash = "sha256:cba42174344c3a5b06f255ce65b350880f962d99ead85e776f23c6618a377a37"}, + {file = "httpx_sse-0.4.1.tar.gz", hash = "sha256:8f44d34414bc7b21bf3602713005c5df4917884f76072479b21f68befa4ea26e"}, +] + [[package]] name = "huggingface-hub" -version = "0.31.2" +version = "0.33.4" description = "Client library to download and publish models, datasets and other repos on the huggingface.co hub" optional = false python-versions = ">=3.8.0" groups = ["main", "eval"] files = [ - {file = "huggingface_hub-0.31.2-py3-none-any.whl", hash = "sha256:8138cd52aa2326b4429bb00a4a1ba8538346b7b8a808cdce30acb6f1f1bdaeec"}, - {file = "huggingface_hub-0.31.2.tar.gz", hash = "sha256:7053561376ed7f6ffdaecf09cc54d70dc784ac6315fa4bb9b93e19662b029675"}, + {file = "huggingface_hub-0.33.4-py3-none-any.whl", hash = "sha256:09f9f4e7ca62547c70f8b82767eefadd2667f4e116acba2e3e62a5a81815a7bb"}, + {file = "huggingface_hub-0.33.4.tar.gz", hash = "sha256:6af13478deae120e765bfd92adad0ae1aec1ad8c439b46f23058ad5956cbca0a"}, ] [package.dependencies] filelock = "*" fsspec = ">=2023.5.0" +hf-xet = {version = ">=1.1.2,<2.0.0", markers = "platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"arm64\" or platform_machine == \"aarch64\""} packaging = ">=20.9" pyyaml = ">=5.1" requests = "*" @@ -1251,17 +1430,19 @@ tqdm = ">=4.42.1" typing-extensions = ">=3.7.4.3" [package.extras] -all = ["InquirerPy (==0.3.4)", "Jinja2", "Pillow", "aiohttp", "fastapi", "gradio (>=4.0.0)", "jedi", "libcst (==1.4.0)", "mypy (==1.5.1)", "numpy", "pytest (>=8.1.1,<8.2.2)", "pytest-asyncio", "pytest-cov", "pytest-env", "pytest-mock", "pytest-rerunfailures", "pytest-vcr", "pytest-xdist", "ruff (>=0.9.0)", "soundfile", "types-PyYAML", "types-requests", "types-simplejson", "types-toml", "types-tqdm", "types-urllib3", "typing-extensions (>=4.8.0)", "urllib3 (<2.0)"] +all = ["InquirerPy (==0.3.4)", "Jinja2", "Pillow", "aiohttp", "authlib (>=1.3.2)", "fastapi", "gradio (>=4.0.0)", "httpx", "itsdangerous", "jedi", "libcst (==1.4.0)", "mypy (==1.15.0) ; python_version >= \"3.9\"", "mypy (>=1.14.1,<1.15.0) ; python_version == \"3.8\"", "numpy", "pytest (>=8.1.1,<8.2.2)", "pytest-asyncio", "pytest-cov", "pytest-env", "pytest-mock", "pytest-rerunfailures", "pytest-vcr", "pytest-xdist", "ruff (>=0.9.0)", "soundfile", "types-PyYAML", "types-requests", "types-simplejson", "types-toml", "types-tqdm", "types-urllib3", "typing-extensions (>=4.8.0)", "urllib3 (<2.0)"] cli = ["InquirerPy (==0.3.4)"] -dev = ["InquirerPy (==0.3.4)", "Jinja2", "Pillow", "aiohttp", "fastapi", "gradio (>=4.0.0)", "jedi", "libcst (==1.4.0)", "mypy (==1.5.1)", "numpy", "pytest (>=8.1.1,<8.2.2)", "pytest-asyncio", "pytest-cov", "pytest-env", "pytest-mock", "pytest-rerunfailures", "pytest-vcr", "pytest-xdist", "ruff (>=0.9.0)", "soundfile", "types-PyYAML", "types-requests", "types-simplejson", "types-toml", "types-tqdm", "types-urllib3", "typing-extensions (>=4.8.0)", "urllib3 (<2.0)"] +dev = ["InquirerPy (==0.3.4)", "Jinja2", "Pillow", "aiohttp", "authlib (>=1.3.2)", "fastapi", "gradio (>=4.0.0)", "httpx", "itsdangerous", "jedi", "libcst (==1.4.0)", "mypy (==1.15.0) ; python_version >= \"3.9\"", "mypy (>=1.14.1,<1.15.0) ; python_version == \"3.8\"", "numpy", "pytest (>=8.1.1,<8.2.2)", "pytest-asyncio", "pytest-cov", "pytest-env", "pytest-mock", "pytest-rerunfailures", "pytest-vcr", "pytest-xdist", "ruff (>=0.9.0)", "soundfile", "types-PyYAML", "types-requests", "types-simplejson", "types-toml", "types-tqdm", "types-urllib3", "typing-extensions (>=4.8.0)", "urllib3 (<2.0)"] fastai = ["fastai (>=2.4)", "fastcore (>=1.3.27)", "toml"] hf-transfer = ["hf-transfer (>=0.1.4)"] -hf-xet = ["hf-xet (>=1.1.1,<2.0.0)"] +hf-xet = ["hf-xet (>=1.1.2,<2.0.0)"] inference = ["aiohttp"] -quality = ["libcst (==1.4.0)", "mypy (==1.5.1)", "ruff (>=0.9.0)"] +mcp = ["aiohttp", "mcp (>=1.8.0)", "typer"] +oauth = ["authlib (>=1.3.2)", "fastapi", "httpx", "itsdangerous"] +quality = ["libcst (==1.4.0)", "mypy (==1.15.0) ; python_version >= \"3.9\"", "mypy (>=1.14.1,<1.15.0) ; python_version == \"3.8\"", "ruff (>=0.9.0)"] tensorflow = ["graphviz", "pydot", "tensorflow"] tensorflow-testing = ["keras (<3.0)", "tensorflow"] -testing = ["InquirerPy (==0.3.4)", "Jinja2", "Pillow", "aiohttp", "fastapi", "gradio (>=4.0.0)", "jedi", "numpy", "pytest (>=8.1.1,<8.2.2)", "pytest-asyncio", "pytest-cov", "pytest-env", "pytest-mock", "pytest-rerunfailures", "pytest-vcr", "pytest-xdist", "soundfile", "urllib3 (<2.0)"] +testing = ["InquirerPy (==0.3.4)", "Jinja2", "Pillow", "aiohttp", "authlib (>=1.3.2)", "fastapi", "gradio (>=4.0.0)", "httpx", "itsdangerous", "jedi", "numpy", "pytest (>=8.1.1,<8.2.2)", "pytest-asyncio", "pytest-cov", "pytest-env", "pytest-mock", "pytest-rerunfailures", "pytest-vcr", "pytest-xdist", "soundfile", "urllib3 (<2.0)"] torch = ["safetensors[torch]", "torch"] typing = ["types-PyYAML", "types-requests", "types-simplejson", "types-toml", "types-tqdm", "types-urllib3", "typing-extensions (>=4.8.0)"] @@ -1269,9 +1450,10 @@ typing = ["types-PyYAML", "types-requests", "types-simplejson", "types-toml", "t name = "humanfriendly" version = "10.0" description = "Human friendly output for text interfaces using Python" -optional = false +optional = true python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "humanfriendly-10.0-py2.py3-none-any.whl", hash = "sha256:1697e1a8a8f550fd43c2865cd84542fc175a61dcb779b6fee18cf6b6ccba1477"}, {file = "humanfriendly-10.0.tar.gz", hash = "sha256:6b0b831ce8f15f7300721aa49829fc4e83921a9a301cc7f606be6686a2288ddc"}, @@ -1291,17 +1473,18 @@ files = [ {file = "hyperframe-6.1.0-py3-none-any.whl", hash = "sha256:b03380493a519fce58ea5af42e4a42317bf9bd425596f7a0835ffce80f1a42e5"}, {file = "hyperframe-6.1.0.tar.gz", hash = "sha256:f630908a00854a7adeabd6382b43923a4c4cd4b821fcb527e6ab9e15382a3b08"}, ] +markers = {main = "extra == \"all\""} [[package]] name = "identify" -version = "2.6.10" +version = "2.6.12" description = "File identification library for Python" optional = false python-versions = ">=3.9" groups = ["dev"] files = [ - {file = "identify-2.6.10-py2.py3-none-any.whl", hash = "sha256:5f34248f54136beed1a7ba6a6b5c4b6cf21ff495aac7c359e1ef831ae3b8ab25"}, - {file = "identify-2.6.10.tar.gz", hash = "sha256:45e92fd704f3da71cc3880036633f48b4b7265fd4de2b57627cb157216eb7eb8"}, + {file = "identify-2.6.12-py2.py3-none-any.whl", hash = "sha256:ad9672d5a72e0d2ff7c5c8809b62dfa60458626352fb0eb7b55e69bdc45334a2"}, + {file = "identify-2.6.12.tar.gz", hash = "sha256:d8de45749f1efb108badef65ee8386f0f7bb19a7f26185f74de6367bffbaf0e6"}, ] [package.extras] @@ -1366,88 +1549,89 @@ i18n = ["Babel (>=2.7)"] [[package]] name = "jiter" -version = "0.9.0" +version = "0.10.0" description = "Fast iterable JSON parser." optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "jiter-0.9.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:816ec9b60fdfd1fec87da1d7ed46c66c44ffec37ab2ef7de5b147b2fce3fd5ad"}, - {file = "jiter-0.9.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9b1d3086f8a3ee0194ecf2008cf81286a5c3e540d977fa038ff23576c023c0ea"}, - {file = "jiter-0.9.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1339f839b91ae30b37c409bf16ccd3dc453e8b8c3ed4bd1d6a567193651a4a51"}, - {file = "jiter-0.9.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ffba79584b3b670fefae66ceb3a28822365d25b7bf811e030609a3d5b876f538"}, - {file = "jiter-0.9.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5cfc7d0a8e899089d11f065e289cb5b2daf3d82fbe028f49b20d7b809193958d"}, - {file = "jiter-0.9.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e00a1a2bbfaaf237e13c3d1592356eab3e9015d7efd59359ac8b51eb56390a12"}, - {file = "jiter-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d1d9870561eb26b11448854dce0ff27a9a27cb616b632468cafc938de25e9e51"}, - {file = "jiter-0.9.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:9872aeff3f21e437651df378cb75aeb7043e5297261222b6441a620218b58708"}, - {file = "jiter-0.9.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:1fd19112d1049bdd47f17bfbb44a2c0001061312dcf0e72765bfa8abd4aa30e5"}, - {file = "jiter-0.9.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:6ef5da104664e526836070e4a23b5f68dec1cc673b60bf1edb1bfbe8a55d0678"}, - {file = "jiter-0.9.0-cp310-cp310-win32.whl", hash = "sha256:cb12e6d65ebbefe5518de819f3eda53b73187b7089040b2d17f5b39001ff31c4"}, - {file = "jiter-0.9.0-cp310-cp310-win_amd64.whl", hash = "sha256:c43ca669493626d8672be3b645dbb406ef25af3f4b6384cfd306da7eb2e70322"}, - {file = "jiter-0.9.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:6c4d99c71508912a7e556d631768dcdef43648a93660670986916b297f1c54af"}, - {file = "jiter-0.9.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8f60fb8ce7df529812bf6c625635a19d27f30806885139e367af93f6e734ef58"}, - {file = "jiter-0.9.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:51c4e1a4f8ea84d98b7b98912aa4290ac3d1eabfde8e3c34541fae30e9d1f08b"}, - {file = "jiter-0.9.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5f4c677c424dc76684fea3e7285a7a2a7493424bea89ac441045e6a1fb1d7b3b"}, - {file = "jiter-0.9.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2221176dfec87f3470b21e6abca056e6b04ce9bff72315cb0b243ca9e835a4b5"}, - {file = "jiter-0.9.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3c7adb66f899ffa25e3c92bfcb593391ee1947dbdd6a9a970e0d7e713237d572"}, - {file = "jiter-0.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c98d27330fdfb77913c1097a7aab07f38ff2259048949f499c9901700789ac15"}, - {file = "jiter-0.9.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:eda3f8cc74df66892b1d06b5d41a71670c22d95a1ca2cbab73654745ce9d0419"}, - {file = "jiter-0.9.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:dd5ab5ddc11418dce28343123644a100f487eaccf1de27a459ab36d6cca31043"}, - {file = "jiter-0.9.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:42f8a68a69f047b310319ef8e2f52fdb2e7976fb3313ef27df495cf77bcad965"}, - {file = "jiter-0.9.0-cp311-cp311-win32.whl", hash = "sha256:a25519efb78a42254d59326ee417d6f5161b06f5da827d94cf521fed961b1ff2"}, - {file = "jiter-0.9.0-cp311-cp311-win_amd64.whl", hash = "sha256:923b54afdd697dfd00d368b7ccad008cccfeb1efb4e621f32860c75e9f25edbd"}, - {file = "jiter-0.9.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:7b46249cfd6c48da28f89eb0be3f52d6fdb40ab88e2c66804f546674e539ec11"}, - {file = "jiter-0.9.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:609cf3c78852f1189894383cf0b0b977665f54cb38788e3e6b941fa6d982c00e"}, - {file = "jiter-0.9.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d726a3890a54561e55a9c5faea1f7655eda7f105bd165067575ace6e65f80bb2"}, - {file = "jiter-0.9.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:2e89dc075c1fef8fa9be219e249f14040270dbc507df4215c324a1839522ea75"}, - {file = "jiter-0.9.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:04e8ffa3c353b1bc4134f96f167a2082494351e42888dfcf06e944f2729cbe1d"}, - {file = "jiter-0.9.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:203f28a72a05ae0e129b3ed1f75f56bc419d5f91dfacd057519a8bd137b00c42"}, - {file = "jiter-0.9.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fca1a02ad60ec30bb230f65bc01f611c8608b02d269f998bc29cca8619a919dc"}, - {file = "jiter-0.9.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:237e5cee4d5d2659aaf91bbf8ec45052cc217d9446070699441a91b386ae27dc"}, - {file = "jiter-0.9.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:528b6b71745e7326eed73c53d4aa57e2a522242320b6f7d65b9c5af83cf49b6e"}, - {file = "jiter-0.9.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:9f48e86b57bc711eb5acdfd12b6cb580a59cc9a993f6e7dcb6d8b50522dcd50d"}, - {file = "jiter-0.9.0-cp312-cp312-win32.whl", hash = "sha256:699edfde481e191d81f9cf6d2211debbfe4bd92f06410e7637dffb8dd5dfde06"}, - {file = "jiter-0.9.0-cp312-cp312-win_amd64.whl", hash = "sha256:099500d07b43f61d8bd780466d429c45a7b25411b334c60ca875fa775f68ccb0"}, - {file = "jiter-0.9.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:2764891d3f3e8b18dce2cff24949153ee30c9239da7c00f032511091ba688ff7"}, - {file = "jiter-0.9.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:387b22fbfd7a62418d5212b4638026d01723761c75c1c8232a8b8c37c2f1003b"}, - {file = "jiter-0.9.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:40d8da8629ccae3606c61d9184970423655fb4e33d03330bcdfe52d234d32f69"}, - {file = "jiter-0.9.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a1be73d8982bdc278b7b9377426a4b44ceb5c7952073dd7488e4ae96b88e1103"}, - {file = "jiter-0.9.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2228eaaaa111ec54b9e89f7481bffb3972e9059301a878d085b2b449fbbde635"}, - {file = "jiter-0.9.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:11509bfecbc319459647d4ac3fd391d26fdf530dad00c13c4dadabf5b81f01a4"}, - {file = "jiter-0.9.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3f22238da568be8bbd8e0650e12feeb2cfea15eda4f9fc271d3b362a4fa0604d"}, - {file = "jiter-0.9.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:17f5d55eb856597607562257c8e36c42bc87f16bef52ef7129b7da11afc779f3"}, - {file = "jiter-0.9.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:6a99bed9fbb02f5bed416d137944419a69aa4c423e44189bc49718859ea83bc5"}, - {file = "jiter-0.9.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:e057adb0cd1bd39606100be0eafe742de2de88c79df632955b9ab53a086b3c8d"}, - {file = "jiter-0.9.0-cp313-cp313-win32.whl", hash = "sha256:f7e6850991f3940f62d387ccfa54d1a92bd4bb9f89690b53aea36b4364bcab53"}, - {file = "jiter-0.9.0-cp313-cp313-win_amd64.whl", hash = "sha256:c8ae3bf27cd1ac5e6e8b7a27487bf3ab5f82318211ec2e1346a5b058756361f7"}, - {file = "jiter-0.9.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:f0b2827fb88dda2cbecbbc3e596ef08d69bda06c6f57930aec8e79505dc17001"}, - {file = "jiter-0.9.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:062b756ceb1d40b0b28f326cba26cfd575a4918415b036464a52f08632731e5a"}, - {file = "jiter-0.9.0-cp313-cp313t-win_amd64.whl", hash = "sha256:6f7838bc467ab7e8ef9f387bd6de195c43bad82a569c1699cb822f6609dd4cdf"}, - {file = "jiter-0.9.0-cp38-cp38-macosx_10_12_x86_64.whl", hash = "sha256:4a2d16360d0642cd68236f931b85fe50288834c383492e4279d9f1792e309571"}, - {file = "jiter-0.9.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:e84ed1c9c9ec10bbb8c37f450077cbe3c0d4e8c2b19f0a49a60ac7ace73c7452"}, - {file = "jiter-0.9.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9f3c848209ccd1bfa344a1240763975ca917de753c7875c77ec3034f4151d06c"}, - {file = "jiter-0.9.0-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7825f46e50646bee937e0f849d14ef3a417910966136f59cd1eb848b8b5bb3e4"}, - {file = "jiter-0.9.0-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d82a811928b26d1a6311a886b2566f68ccf2b23cf3bfed042e18686f1f22c2d7"}, - {file = "jiter-0.9.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0c058ecb51763a67f019ae423b1cbe3fa90f7ee6280c31a1baa6ccc0c0e2d06e"}, - {file = "jiter-0.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9897115ad716c48f0120c1f0c4efae348ec47037319a6c63b2d7838bb53aaef4"}, - {file = "jiter-0.9.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:351f4c90a24c4fb8c87c6a73af2944c440494ed2bea2094feecacb75c50398ae"}, - {file = "jiter-0.9.0-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:d45807b0f236c485e1e525e2ce3a854807dfe28ccf0d013dd4a563395e28008a"}, - {file = "jiter-0.9.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:1537a890724ba00fdba21787010ac6f24dad47f763410e9e1093277913592784"}, - {file = "jiter-0.9.0-cp38-cp38-win32.whl", hash = "sha256:e3630ec20cbeaddd4b65513fa3857e1b7c4190d4481ef07fb63d0fad59033321"}, - {file = "jiter-0.9.0-cp38-cp38-win_amd64.whl", hash = "sha256:2685f44bf80e95f8910553bf2d33b9c87bf25fceae6e9f0c1355f75d2922b0ee"}, - {file = "jiter-0.9.0-cp39-cp39-macosx_10_12_x86_64.whl", hash = "sha256:9ef340fae98065071ccd5805fe81c99c8f80484e820e40043689cf97fb66b3e2"}, - {file = "jiter-0.9.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:efb767d92c63b2cd9ec9f24feeb48f49574a713870ec87e9ba0c2c6e9329c3e2"}, - {file = "jiter-0.9.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:113f30f87fb1f412510c6d7ed13e91422cfd329436364a690c34c8b8bd880c42"}, - {file = "jiter-0.9.0-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8793b6df019b988526f5a633fdc7456ea75e4a79bd8396a3373c371fc59f5c9b"}, - {file = "jiter-0.9.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7a9aaa5102dba4e079bb728076fadd5a2dca94c05c04ce68004cfd96f128ea34"}, - {file = "jiter-0.9.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d838650f6ebaf4ccadfb04522463e74a4c378d7e667e0eb1865cfe3990bfac49"}, - {file = "jiter-0.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c0194f813efdf4b8865ad5f5c5f50f8566df7d770a82c51ef593d09e0b347020"}, - {file = "jiter-0.9.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a7954a401d0a8a0b8bc669199db78af435aae1e3569187c2939c477c53cb6a0a"}, - {file = "jiter-0.9.0-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:4feafe787eb8a8d98168ab15637ca2577f6ddf77ac6c8c66242c2d028aa5420e"}, - {file = "jiter-0.9.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:27cd1f2e8bb377f31d3190b34e4328d280325ad7ef55c6ac9abde72f79e84d2e"}, - {file = "jiter-0.9.0-cp39-cp39-win32.whl", hash = "sha256:161d461dcbe658cf0bd0aa375b30a968b087cdddc624fc585f3867c63c6eca95"}, - {file = "jiter-0.9.0-cp39-cp39-win_amd64.whl", hash = "sha256:e8b36d8a16a61993be33e75126ad3d8aa29cf450b09576f3c427d27647fcb4aa"}, - {file = "jiter-0.9.0.tar.gz", hash = "sha256:aadba0964deb424daa24492abc3d229c60c4a31bfee205aedbf1acc7639d7893"}, + {file = "jiter-0.10.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:cd2fb72b02478f06a900a5782de2ef47e0396b3e1f7d5aba30daeb1fce66f303"}, + {file = "jiter-0.10.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:32bb468e3af278f095d3fa5b90314728a6916d89ba3d0ffb726dd9bf7367285e"}, + {file = "jiter-0.10.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:aa8b3e0068c26ddedc7abc6fac37da2d0af16b921e288a5a613f4b86f050354f"}, + {file = "jiter-0.10.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:286299b74cc49e25cd42eea19b72aa82c515d2f2ee12d11392c56d8701f52224"}, + {file = "jiter-0.10.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6ed5649ceeaeffc28d87fb012d25a4cd356dcd53eff5acff1f0466b831dda2a7"}, + {file = "jiter-0.10.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b2ab0051160cb758a70716448908ef14ad476c3774bd03ddce075f3c1f90a3d6"}, + {file = "jiter-0.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:03997d2f37f6b67d2f5c475da4412be584e1cec273c1cfc03d642c46db43f8cf"}, + {file = "jiter-0.10.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c404a99352d839fed80d6afd6c1d66071f3bacaaa5c4268983fc10f769112e90"}, + {file = "jiter-0.10.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:66e989410b6666d3ddb27a74c7e50d0829704ede652fd4c858e91f8d64b403d0"}, + {file = "jiter-0.10.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:b532d3af9ef4f6374609a3bcb5e05a1951d3bf6190dc6b176fdb277c9bbf15ee"}, + {file = "jiter-0.10.0-cp310-cp310-win32.whl", hash = "sha256:da9be20b333970e28b72edc4dff63d4fec3398e05770fb3205f7fb460eb48dd4"}, + {file = "jiter-0.10.0-cp310-cp310-win_amd64.whl", hash = "sha256:f59e533afed0c5b0ac3eba20d2548c4a550336d8282ee69eb07b37ea526ee4e5"}, + {file = "jiter-0.10.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:3bebe0c558e19902c96e99217e0b8e8b17d570906e72ed8a87170bc290b1e978"}, + {file = "jiter-0.10.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:558cc7e44fd8e507a236bee6a02fa17199ba752874400a0ca6cd6e2196cdb7dc"}, + {file = "jiter-0.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4d613e4b379a07d7c8453c5712ce7014e86c6ac93d990a0b8e7377e18505e98d"}, + {file = "jiter-0.10.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f62cf8ba0618eda841b9bf61797f21c5ebd15a7a1e19daab76e4e4b498d515b2"}, + {file = "jiter-0.10.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:919d139cdfa8ae8945112398511cb7fca58a77382617d279556b344867a37e61"}, + {file = "jiter-0.10.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:13ddbc6ae311175a3b03bd8994881bc4635c923754932918e18da841632349db"}, + {file = "jiter-0.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4c440ea003ad10927a30521a9062ce10b5479592e8a70da27f21eeb457b4a9c5"}, + {file = "jiter-0.10.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:dc347c87944983481e138dea467c0551080c86b9d21de6ea9306efb12ca8f606"}, + {file = "jiter-0.10.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:13252b58c1f4d8c5b63ab103c03d909e8e1e7842d302473f482915d95fefd605"}, + {file = "jiter-0.10.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:7d1bbf3c465de4a24ab12fb7766a0003f6f9bce48b8b6a886158c4d569452dc5"}, + {file = "jiter-0.10.0-cp311-cp311-win32.whl", hash = "sha256:db16e4848b7e826edca4ccdd5b145939758dadf0dc06e7007ad0e9cfb5928ae7"}, + {file = "jiter-0.10.0-cp311-cp311-win_amd64.whl", hash = "sha256:9c9c1d5f10e18909e993f9641f12fe1c77b3e9b533ee94ffa970acc14ded3812"}, + {file = "jiter-0.10.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:1e274728e4a5345a6dde2d343c8da018b9d4bd4350f5a472fa91f66fda44911b"}, + {file = "jiter-0.10.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7202ae396446c988cb2a5feb33a543ab2165b786ac97f53b59aafb803fef0744"}, + {file = "jiter-0.10.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:23ba7722d6748b6920ed02a8f1726fb4b33e0fd2f3f621816a8b486c66410ab2"}, + {file = "jiter-0.10.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:371eab43c0a288537d30e1f0b193bc4eca90439fc08a022dd83e5e07500ed026"}, + {file = "jiter-0.10.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6c675736059020365cebc845a820214765162728b51ab1e03a1b7b3abb70f74c"}, + {file = "jiter-0.10.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0c5867d40ab716e4684858e4887489685968a47e3ba222e44cde6e4a2154f959"}, + {file = "jiter-0.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:395bb9a26111b60141757d874d27fdea01b17e8fac958b91c20128ba8f4acc8a"}, + {file = "jiter-0.10.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6842184aed5cdb07e0c7e20e5bdcfafe33515ee1741a6835353bb45fe5d1bd95"}, + {file = "jiter-0.10.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:62755d1bcea9876770d4df713d82606c8c1a3dca88ff39046b85a048566d56ea"}, + {file = "jiter-0.10.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:533efbce2cacec78d5ba73a41756beff8431dfa1694b6346ce7af3a12c42202b"}, + {file = "jiter-0.10.0-cp312-cp312-win32.whl", hash = "sha256:8be921f0cadd245e981b964dfbcd6fd4bc4e254cdc069490416dd7a2632ecc01"}, + {file = "jiter-0.10.0-cp312-cp312-win_amd64.whl", hash = "sha256:a7c7d785ae9dda68c2678532a5a1581347e9c15362ae9f6e68f3fdbfb64f2e49"}, + {file = "jiter-0.10.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:e0588107ec8e11b6f5ef0e0d656fb2803ac6cf94a96b2b9fc675c0e3ab5e8644"}, + {file = "jiter-0.10.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cafc4628b616dc32530c20ee53d71589816cf385dd9449633e910d596b1f5c8a"}, + {file = "jiter-0.10.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:520ef6d981172693786a49ff5b09eda72a42e539f14788124a07530f785c3ad6"}, + {file = "jiter-0.10.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:554dedfd05937f8fc45d17ebdf298fe7e0c77458232bcb73d9fbbf4c6455f5b3"}, + {file = "jiter-0.10.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5bc299da7789deacf95f64052d97f75c16d4fc8c4c214a22bf8d859a4288a1c2"}, + {file = "jiter-0.10.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:5161e201172de298a8a1baad95eb85db4fb90e902353b1f6a41d64ea64644e25"}, + {file = "jiter-0.10.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2e2227db6ba93cb3e2bf67c87e594adde0609f146344e8207e8730364db27041"}, + {file = "jiter-0.10.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:15acb267ea5e2c64515574b06a8bf393fbfee6a50eb1673614aa45f4613c0cca"}, + {file = "jiter-0.10.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:901b92f2e2947dc6dfcb52fd624453862e16665ea909a08398dde19c0731b7f4"}, + {file = "jiter-0.10.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:d0cb9a125d5a3ec971a094a845eadde2db0de85b33c9f13eb94a0c63d463879e"}, + {file = "jiter-0.10.0-cp313-cp313-win32.whl", hash = "sha256:48a403277ad1ee208fb930bdf91745e4d2d6e47253eedc96e2559d1e6527006d"}, + {file = "jiter-0.10.0-cp313-cp313-win_amd64.whl", hash = "sha256:75f9eb72ecb640619c29bf714e78c9c46c9c4eaafd644bf78577ede459f330d4"}, + {file = "jiter-0.10.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:28ed2a4c05a1f32ef0e1d24c2611330219fed727dae01789f4a335617634b1ca"}, + {file = "jiter-0.10.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:14a4c418b1ec86a195f1ca69da8b23e8926c752b685af665ce30777233dfe070"}, + {file = "jiter-0.10.0-cp313-cp313t-win_amd64.whl", hash = "sha256:d7bfed2fe1fe0e4dda6ef682cee888ba444b21e7a6553e03252e4feb6cf0adca"}, + {file = "jiter-0.10.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:5e9251a5e83fab8d87799d3e1a46cb4b7f2919b895c6f4483629ed2446f66522"}, + {file = "jiter-0.10.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:023aa0204126fe5b87ccbcd75c8a0d0261b9abdbbf46d55e7ae9f8e22424eeb8"}, + {file = "jiter-0.10.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3c189c4f1779c05f75fc17c0c1267594ed918996a231593a21a5ca5438445216"}, + {file = "jiter-0.10.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:15720084d90d1098ca0229352607cd68256c76991f6b374af96f36920eae13c4"}, + {file = "jiter-0.10.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e4f2fb68e5f1cfee30e2b2a09549a00683e0fde4c6a2ab88c94072fc33cb7426"}, + {file = "jiter-0.10.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ce541693355fc6da424c08b7edf39a2895f58d6ea17d92cc2b168d20907dee12"}, + {file = "jiter-0.10.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:31c50c40272e189d50006ad5c73883caabb73d4e9748a688b216e85a9a9ca3b9"}, + {file = "jiter-0.10.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:fa3402a2ff9815960e0372a47b75c76979d74402448509ccd49a275fa983ef8a"}, + {file = "jiter-0.10.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:1956f934dca32d7bb647ea21d06d93ca40868b505c228556d3373cbd255ce853"}, + {file = "jiter-0.10.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:fcedb049bdfc555e261d6f65a6abe1d5ad68825b7202ccb9692636c70fcced86"}, + {file = "jiter-0.10.0-cp314-cp314-win32.whl", hash = "sha256:ac509f7eccca54b2a29daeb516fb95b6f0bd0d0d8084efaf8ed5dfc7b9f0b357"}, + {file = "jiter-0.10.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:5ed975b83a2b8639356151cef5c0d597c68376fc4922b45d0eb384ac058cfa00"}, + {file = "jiter-0.10.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3aa96f2abba33dc77f79b4cf791840230375f9534e5fac927ccceb58c5e604a5"}, + {file = "jiter-0.10.0-cp39-cp39-macosx_10_12_x86_64.whl", hash = "sha256:bd6292a43c0fc09ce7c154ec0fa646a536b877d1e8f2f96c19707f65355b5a4d"}, + {file = "jiter-0.10.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:39de429dcaeb6808d75ffe9effefe96a4903c6a4b376b2f6d08d77c1aaee2f18"}, + {file = "jiter-0.10.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:52ce124f13a7a616fad3bb723f2bfb537d78239d1f7f219566dc52b6f2a9e48d"}, + {file = "jiter-0.10.0-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:166f3606f11920f9a1746b2eea84fa2c0a5d50fd313c38bdea4edc072000b0af"}, + {file = "jiter-0.10.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:28dcecbb4ba402916034fc14eba7709f250c4d24b0c43fc94d187ee0580af181"}, + {file = "jiter-0.10.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:86c5aa6910f9bebcc7bc4f8bc461aff68504388b43bfe5e5c0bd21efa33b52f4"}, + {file = "jiter-0.10.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ceeb52d242b315d7f1f74b441b6a167f78cea801ad7c11c36da77ff2d42e8a28"}, + {file = "jiter-0.10.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ff76d8887c8c8ee1e772274fcf8cc1071c2c58590d13e33bd12d02dc9a560397"}, + {file = "jiter-0.10.0-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:a9be4d0fa2b79f7222a88aa488bd89e2ae0a0a5b189462a12def6ece2faa45f1"}, + {file = "jiter-0.10.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:9ab7fd8738094139b6c1ab1822d6f2000ebe41515c537235fd45dabe13ec9324"}, + {file = "jiter-0.10.0-cp39-cp39-win32.whl", hash = "sha256:5f51e048540dd27f204ff4a87f5d79294ea0aa3aa552aca34934588cf27023cf"}, + {file = "jiter-0.10.0-cp39-cp39-win_amd64.whl", hash = "sha256:1b28302349dc65703a9e4ead16f163b1c339efffbe1049c30a44b001a2a4fff9"}, + {file = "jiter-0.10.0.tar.gz", hash = "sha256:07a7142c38aacc85194391108dc91b5b57093c978a9932bd86a36862759d9500"}, ] [[package]] @@ -1489,6 +1673,43 @@ files = [ {file = "jsonpointer-3.0.0.tar.gz", hash = "sha256:2b2d729f2091522d61c3b31f82e11870f60b68f43fbc705cb76bf4b832af59ef"}, ] +[[package]] +name = "jsonschema" +version = "4.24.1" +description = "An implementation of JSON Schema validation for Python" +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "jsonschema-4.24.1-py3-none-any.whl", hash = "sha256:6b916866aa0b61437785f1277aa2cbd63512e8d4b47151072ef13292049b4627"}, + {file = "jsonschema-4.24.1.tar.gz", hash = "sha256:fe45a130cc7f67cd0d67640b4e7e3e2e666919462ae355eda238296eafeb4b5d"}, +] + +[package.dependencies] +attrs = ">=22.2.0" +jsonschema-specifications = ">=2023.03.6" +referencing = ">=0.28.4" +rpds-py = ">=0.7.1" + +[package.extras] +format = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3987", "uri-template", "webcolors (>=1.11)"] +format-nongpl = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3986-validator (>0.1.0)", "uri-template", "webcolors (>=24.6.0)"] + +[[package]] +name = "jsonschema-specifications" +version = "2025.4.1" +description = "The JSON Schema meta-schemas and vocabularies, exposed as a Registry" +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "jsonschema_specifications-2025.4.1-py3-none-any.whl", hash = "sha256:4653bffbd6584f7de83a67e0d620ef16900b390ddc7939d56684d6c81e33f1af"}, + {file = "jsonschema_specifications-2025.4.1.tar.gz", hash = "sha256:630159c9f4dbea161a6a2205c3011cc4f18ff381b189fff48bb39b9bf26ae608"}, +] + +[package.dependencies] +referencing = ">=0.31.0" + [[package]] name = "kiwisolver" version = "1.4.8" @@ -1639,20 +1860,20 @@ pydantic = ">=2.7.4,<3.0.0" [[package]] name = "langchain-core" -version = "0.3.68" +version = "0.3.69" description = "Building applications with LLMs through composability" optional = false python-versions = ">=3.9" groups = ["eval"] files = [ - {file = "langchain_core-0.3.68-py3-none-any.whl", hash = "sha256:5e5c1fbef419590537c91b8c2d86af896fbcbaf0d5ed7fdcdd77f7d8f3467ba0"}, - {file = "langchain_core-0.3.68.tar.gz", hash = "sha256:312e1932ac9aa2eaf111b70fdc171776fa571d1a86c1f873dcac88a094b19c6f"}, + {file = "langchain_core-0.3.69-py3-none-any.whl", hash = "sha256:383e9cb4919f7ef4b24bf8552ef42e4323c064924fea88b28dd5d7ddb740d3b8"}, + {file = "langchain_core-0.3.69.tar.gz", hash = "sha256:c132961117cc7f0227a4c58dd3e209674a6dd5b7e74abc61a0df93b0d736e283"}, ] [package.dependencies] jsonpatch = ">=1.33,<2.0" langsmith = ">=0.3.45" -packaging = ">=23.2,<25" +packaging = ">=23.2" pydantic = ">=2.7.4" PyYAML = ">=5.3" tenacity = ">=8.1.0,<8.4.0 || >8.4.0,<10.0.0" @@ -1660,19 +1881,19 @@ typing-extensions = ">=4.7" [[package]] name = "langchain-openai" -version = "0.3.23" +version = "0.3.28" description = "An integration package connecting OpenAI and LangChain" optional = false python-versions = ">=3.9" groups = ["eval"] files = [ - {file = "langchain_openai-0.3.23-py3-none-any.whl", hash = "sha256:624794394482c0923823f0aac44979968d77fdcfa810e42d4b0abd8096199a40"}, - {file = "langchain_openai-0.3.23.tar.gz", hash = "sha256:73411c06e04bc145db7146a6fcf33dd0f1a85130499dcae988829a4441ddaa66"}, + {file = "langchain_openai-0.3.28-py3-none-any.whl", hash = "sha256:4cd6d80a5b2ae471a168017bc01b2e0f01548328d83532400a001623624ede67"}, + {file = "langchain_openai-0.3.28.tar.gz", hash = "sha256:6c669548dbdea325c034ae5ef699710e2abd054c7354fdb3ef7bf909dc739d9e"}, ] [package.dependencies] -langchain-core = ">=0.3.65,<1.0.0" -openai = ">=1.68.2,<2.0.0" +langchain-core = ">=0.3.68,<1.0.0" +openai = ">=1.86.0,<2.0.0" tiktoken = ">=0.7,<1" [[package]] @@ -1692,14 +1913,14 @@ langchain-core = ">=0.3.51,<1.0.0" [[package]] name = "langgraph" -version = "0.5.1" +version = "0.5.3" description = "Building stateful, multi-actor applications with LLMs" optional = false python-versions = ">=3.9" groups = ["eval"] files = [ - {file = "langgraph-0.5.1-py3-none-any.whl", hash = "sha256:707f0cc0d2713011fff4578bf57de8226cd96bcc0679868be2f41eb0984bb5af"}, - {file = "langgraph-0.5.1.tar.gz", hash = "sha256:312d341979e38034dde60e08df53505b8a196619df1266d6eacf6b002a0a65a8"}, + {file = "langgraph-0.5.3-py3-none-any.whl", hash = "sha256:9819b88a6ef6134a0fa6d6121a81b202dc3d17b25cf7ea3fe4d7669b9b252b5d"}, + {file = "langgraph-0.5.3.tar.gz", hash = "sha256:36d4b67f984ff2649d447826fc99b1a2af3e97599a590058f20750048e4f548f"}, ] [package.dependencies] @@ -1712,14 +1933,14 @@ xxhash = ">=3.5.0" [[package]] name = "langgraph-checkpoint" -version = "2.1.0" +version = "2.1.1" description = "Library with base interfaces for LangGraph checkpoint savers." optional = false python-versions = ">=3.9" groups = ["eval"] files = [ - {file = "langgraph_checkpoint-2.1.0-py3-none-any.whl", hash = "sha256:4cea3e512081da1241396a519cbfe4c5d92836545e2c64e85b6f5c34a1b8bc61"}, - {file = "langgraph_checkpoint-2.1.0.tar.gz", hash = "sha256:cdaa2f0b49aa130ab185c02d82f02b40299a1fbc9ac59ac20cecce09642a1abe"}, + {file = "langgraph_checkpoint-2.1.1-py3-none-any.whl", hash = "sha256:5a779134fd28134a9a83d078be4450bbf0e0c79fdf5e992549658899e6fc5ea7"}, + {file = "langgraph_checkpoint-2.1.1.tar.gz", hash = "sha256:72038c0f9e22260cb9bff1f3ebe5eb06d940b7ee5c1e4765019269d4f21cf92d"}, ] [package.dependencies] @@ -1744,14 +1965,14 @@ langgraph-checkpoint = ">=2.1.0" [[package]] name = "langgraph-sdk" -version = "0.1.72" +version = "0.1.73" description = "SDK for interacting with LangGraph API" optional = false python-versions = ">=3.9" groups = ["eval"] files = [ - {file = "langgraph_sdk-0.1.72-py3-none-any.whl", hash = "sha256:925d3fcc7a26361db04f9c4beb3ec05bc36361b2a836d181ff2ab145071ec3ce"}, - {file = "langgraph_sdk-0.1.72.tar.gz", hash = "sha256:396d8195881830700e2d54a0a9ee273e8b1173428e667502ef9c182a3cec7ab7"}, + {file = "langgraph_sdk-0.1.73-py3-none-any.whl", hash = "sha256:a60ac33f70688ad07051edff1d5ed8089c8f0de1f69dc900be46e095ca20eed8"}, + {file = "langgraph_sdk-0.1.73.tar.gz", hash = "sha256:6e6dcdf66bcf8710739899616856527a72a605ce15beb76fbac7f4ce0e2ad080"}, ] [package.dependencies] @@ -1782,14 +2003,14 @@ trustcall = ">=0.0.39" [[package]] name = "langsmith" -version = "0.4.4" +version = "0.4.7" description = "Client library to connect to the LangSmith LLM Tracing and Evaluation Platform." optional = false python-versions = ">=3.9" groups = ["eval"] files = [ - {file = "langsmith-0.4.4-py3-none-any.whl", hash = "sha256:014c68329bd085bd6c770a6405c61bb6881f82eb554ce8c4d1984b0035fd1716"}, - {file = "langsmith-0.4.4.tar.gz", hash = "sha256:70c53bbff24a7872e88e6fa0af98270f4986a6e364f9e85db1cc5636defa4d66"}, + {file = "langsmith-0.4.7-py3-none-any.whl", hash = "sha256:de91f1abdd65da369996f8eedb5201f442110c9c3bde5babc6f5300f07da65df"}, + {file = "langsmith-0.4.7.tar.gz", hash = "sha256:3864cf29295c2565c578e93d1533f5b39e2b4af616545ace30f069635a319890"}, ] [package.dependencies] @@ -1809,144 +2030,107 @@ pytest = ["pytest (>=7.0.0)", "rich (>=13.9.4,<14.0.0)"] [[package]] name = "lxml" -version = "5.4.0" +version = "6.0.0" description = "Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API." -optional = false -python-versions = ">=3.6" +optional = true +python-versions = ">=3.8" groups = ["main"] -files = [ - {file = "lxml-5.4.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:e7bc6df34d42322c5289e37e9971d6ed114e3776b45fa879f734bded9d1fea9c"}, - {file = "lxml-5.4.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:6854f8bd8a1536f8a1d9a3655e6354faa6406621cf857dc27b681b69860645c7"}, - {file = "lxml-5.4.0-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:696ea9e87442467819ac22394ca36cb3d01848dad1be6fac3fb612d3bd5a12cf"}, - {file = "lxml-5.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6ef80aeac414f33c24b3815ecd560cee272786c3adfa5f31316d8b349bfade28"}, - {file = "lxml-5.4.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3b9c2754cef6963f3408ab381ea55f47dabc6f78f4b8ebb0f0b25cf1ac1f7609"}, - {file = "lxml-5.4.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7a62cc23d754bb449d63ff35334acc9f5c02e6dae830d78dab4dd12b78a524f4"}, - {file = "lxml-5.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8f82125bc7203c5ae8633a7d5d20bcfdff0ba33e436e4ab0abc026a53a8960b7"}, - {file = "lxml-5.4.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:b67319b4aef1a6c56576ff544b67a2a6fbd7eaee485b241cabf53115e8908b8f"}, - {file = "lxml-5.4.0-cp310-cp310-manylinux_2_28_ppc64le.whl", hash = "sha256:a8ef956fce64c8551221f395ba21d0724fed6b9b6242ca4f2f7beb4ce2f41997"}, - {file = "lxml-5.4.0-cp310-cp310-manylinux_2_28_s390x.whl", hash = "sha256:0a01ce7d8479dce84fc03324e3b0c9c90b1ece9a9bb6a1b6c9025e7e4520e78c"}, - {file = "lxml-5.4.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:91505d3ddebf268bb1588eb0f63821f738d20e1e7f05d3c647a5ca900288760b"}, - {file = "lxml-5.4.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:a3bcdde35d82ff385f4ede021df801b5c4a5bcdfb61ea87caabcebfc4945dc1b"}, - {file = "lxml-5.4.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:aea7c06667b987787c7d1f5e1dfcd70419b711cdb47d6b4bb4ad4b76777a0563"}, - {file = "lxml-5.4.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:a7fb111eef4d05909b82152721a59c1b14d0f365e2be4c742a473c5d7372f4f5"}, - {file = "lxml-5.4.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:43d549b876ce64aa18b2328faff70f5877f8c6dede415f80a2f799d31644d776"}, - {file = "lxml-5.4.0-cp310-cp310-win32.whl", hash = "sha256:75133890e40d229d6c5837b0312abbe5bac1c342452cf0e12523477cd3aa21e7"}, - {file = "lxml-5.4.0-cp310-cp310-win_amd64.whl", hash = "sha256:de5b4e1088523e2b6f730d0509a9a813355b7f5659d70eb4f319c76beea2e250"}, - {file = "lxml-5.4.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:98a3912194c079ef37e716ed228ae0dcb960992100461b704aea4e93af6b0bb9"}, - {file = "lxml-5.4.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:0ea0252b51d296a75f6118ed0d8696888e7403408ad42345d7dfd0d1e93309a7"}, - {file = "lxml-5.4.0-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b92b69441d1bd39f4940f9eadfa417a25862242ca2c396b406f9272ef09cdcaa"}, - {file = "lxml-5.4.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:20e16c08254b9b6466526bc1828d9370ee6c0d60a4b64836bc3ac2917d1e16df"}, - {file = "lxml-5.4.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7605c1c32c3d6e8c990dd28a0970a3cbbf1429d5b92279e37fda05fb0c92190e"}, - {file = "lxml-5.4.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ecf4c4b83f1ab3d5a7ace10bafcb6f11df6156857a3c418244cef41ca9fa3e44"}, - {file = "lxml-5.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0cef4feae82709eed352cd7e97ae062ef6ae9c7b5dbe3663f104cd2c0e8d94ba"}, - {file = "lxml-5.4.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:df53330a3bff250f10472ce96a9af28628ff1f4efc51ccba351a8820bca2a8ba"}, - {file = "lxml-5.4.0-cp311-cp311-manylinux_2_28_ppc64le.whl", hash = "sha256:aefe1a7cb852fa61150fcb21a8c8fcea7b58c4cb11fbe59c97a0a4b31cae3c8c"}, - {file = "lxml-5.4.0-cp311-cp311-manylinux_2_28_s390x.whl", hash = "sha256:ef5a7178fcc73b7d8c07229e89f8eb45b2908a9238eb90dcfc46571ccf0383b8"}, - {file = "lxml-5.4.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:d2ed1b3cb9ff1c10e6e8b00941bb2e5bb568b307bfc6b17dffbbe8be5eecba86"}, - {file = "lxml-5.4.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:72ac9762a9f8ce74c9eed4a4e74306f2f18613a6b71fa065495a67ac227b3056"}, - {file = "lxml-5.4.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:f5cb182f6396706dc6cc1896dd02b1c889d644c081b0cdec38747573db88a7d7"}, - {file = "lxml-5.4.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:3a3178b4873df8ef9457a4875703488eb1622632a9cee6d76464b60e90adbfcd"}, - {file = "lxml-5.4.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:e094ec83694b59d263802ed03a8384594fcce477ce484b0cbcd0008a211ca751"}, - {file = "lxml-5.4.0-cp311-cp311-win32.whl", hash = "sha256:4329422de653cdb2b72afa39b0aa04252fca9071550044904b2e7036d9d97fe4"}, - {file = "lxml-5.4.0-cp311-cp311-win_amd64.whl", hash = "sha256:fd3be6481ef54b8cfd0e1e953323b7aa9d9789b94842d0e5b142ef4bb7999539"}, - {file = "lxml-5.4.0-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:b5aff6f3e818e6bdbbb38e5967520f174b18f539c2b9de867b1e7fde6f8d95a4"}, - {file = "lxml-5.4.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:942a5d73f739ad7c452bf739a62a0f83e2578afd6b8e5406308731f4ce78b16d"}, - {file = "lxml-5.4.0-cp312-cp312-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:460508a4b07364d6abf53acaa0a90b6d370fafde5693ef37602566613a9b0779"}, - {file = "lxml-5.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:529024ab3a505fed78fe3cc5ddc079464e709f6c892733e3f5842007cec8ac6e"}, - {file = "lxml-5.4.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7ca56ebc2c474e8f3d5761debfd9283b8b18c76c4fc0967b74aeafba1f5647f9"}, - {file = "lxml-5.4.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a81e1196f0a5b4167a8dafe3a66aa67c4addac1b22dc47947abd5d5c7a3f24b5"}, - {file = "lxml-5.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00b8686694423ddae324cf614e1b9659c2edb754de617703c3d29ff568448df5"}, - {file = "lxml-5.4.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:c5681160758d3f6ac5b4fea370495c48aac0989d6a0f01bb9a72ad8ef5ab75c4"}, - {file = "lxml-5.4.0-cp312-cp312-manylinux_2_28_ppc64le.whl", hash = "sha256:2dc191e60425ad70e75a68c9fd90ab284df64d9cd410ba8d2b641c0c45bc006e"}, - {file = "lxml-5.4.0-cp312-cp312-manylinux_2_28_s390x.whl", hash = "sha256:67f779374c6b9753ae0a0195a892a1c234ce8416e4448fe1e9f34746482070a7"}, - {file = "lxml-5.4.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:79d5bfa9c1b455336f52343130b2067164040604e41f6dc4d8313867ed540079"}, - {file = "lxml-5.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3d3c30ba1c9b48c68489dc1829a6eede9873f52edca1dda900066542528d6b20"}, - {file = "lxml-5.4.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1af80c6316ae68aded77e91cd9d80648f7dd40406cef73df841aa3c36f6907c8"}, - {file = "lxml-5.4.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:4d885698f5019abe0de3d352caf9466d5de2baded00a06ef3f1216c1a58ae78f"}, - {file = "lxml-5.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:aea53d51859b6c64e7c51d522c03cc2c48b9b5d6172126854cc7f01aa11f52bc"}, - {file = "lxml-5.4.0-cp312-cp312-win32.whl", hash = "sha256:d90b729fd2732df28130c064aac9bb8aff14ba20baa4aee7bd0795ff1187545f"}, - {file = "lxml-5.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:1dc4ca99e89c335a7ed47d38964abcb36c5910790f9bd106f2a8fa2ee0b909d2"}, - {file = "lxml-5.4.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:773e27b62920199c6197130632c18fb7ead3257fce1ffb7d286912e56ddb79e0"}, - {file = "lxml-5.4.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ce9c671845de9699904b1e9df95acfe8dfc183f2310f163cdaa91a3535af95de"}, - {file = "lxml-5.4.0-cp313-cp313-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:9454b8d8200ec99a224df8854786262b1bd6461f4280064c807303c642c05e76"}, - {file = "lxml-5.4.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cccd007d5c95279e529c146d095f1d39ac05139de26c098166c4beb9374b0f4d"}, - {file = "lxml-5.4.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0fce1294a0497edb034cb416ad3e77ecc89b313cff7adbee5334e4dc0d11f422"}, - {file = "lxml-5.4.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:24974f774f3a78ac12b95e3a20ef0931795ff04dbb16db81a90c37f589819551"}, - {file = "lxml-5.4.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:497cab4d8254c2a90bf988f162ace2ddbfdd806fce3bda3f581b9d24c852e03c"}, - {file = "lxml-5.4.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:e794f698ae4c5084414efea0f5cc9f4ac562ec02d66e1484ff822ef97c2cadff"}, - {file = "lxml-5.4.0-cp313-cp313-manylinux_2_28_ppc64le.whl", hash = "sha256:2c62891b1ea3094bb12097822b3d44b93fc6c325f2043c4d2736a8ff09e65f60"}, - {file = "lxml-5.4.0-cp313-cp313-manylinux_2_28_s390x.whl", hash = "sha256:142accb3e4d1edae4b392bd165a9abdee8a3c432a2cca193df995bc3886249c8"}, - {file = "lxml-5.4.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:1a42b3a19346e5601d1b8296ff6ef3d76038058f311902edd574461e9c036982"}, - {file = "lxml-5.4.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4291d3c409a17febf817259cb37bc62cb7eb398bcc95c1356947e2871911ae61"}, - {file = "lxml-5.4.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4f5322cf38fe0e21c2d73901abf68e6329dc02a4994e483adbcf92b568a09a54"}, - {file = "lxml-5.4.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:0be91891bdb06ebe65122aa6bf3fc94489960cf7e03033c6f83a90863b23c58b"}, - {file = "lxml-5.4.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:15a665ad90054a3d4f397bc40f73948d48e36e4c09f9bcffc7d90c87410e478a"}, - {file = "lxml-5.4.0-cp313-cp313-win32.whl", hash = "sha256:d5663bc1b471c79f5c833cffbc9b87d7bf13f87e055a5c86c363ccd2348d7e82"}, - {file = "lxml-5.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:bcb7a1096b4b6b24ce1ac24d4942ad98f983cd3810f9711bcd0293f43a9d8b9f"}, - {file = "lxml-5.4.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:7be701c24e7f843e6788353c055d806e8bd8466b52907bafe5d13ec6a6dbaecd"}, - {file = "lxml-5.4.0-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:fb54f7c6bafaa808f27166569b1511fc42701a7713858dddc08afdde9746849e"}, - {file = "lxml-5.4.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:97dac543661e84a284502e0cf8a67b5c711b0ad5fb661d1bd505c02f8cf716d7"}, - {file = "lxml-5.4.0-cp36-cp36m-manylinux_2_28_x86_64.whl", hash = "sha256:c70e93fba207106cb16bf852e421c37bbded92acd5964390aad07cb50d60f5cf"}, - {file = "lxml-5.4.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:9c886b481aefdf818ad44846145f6eaf373a20d200b5ce1a5c8e1bc2d8745410"}, - {file = "lxml-5.4.0-cp36-cp36m-musllinux_1_2_x86_64.whl", hash = "sha256:fa0e294046de09acd6146be0ed6727d1f42ded4ce3ea1e9a19c11b6774eea27c"}, - {file = "lxml-5.4.0-cp36-cp36m-win32.whl", hash = "sha256:61c7bbf432f09ee44b1ccaa24896d21075e533cd01477966a5ff5a71d88b2f56"}, - {file = "lxml-5.4.0-cp36-cp36m-win_amd64.whl", hash = "sha256:7ce1a171ec325192c6a636b64c94418e71a1964f56d002cc28122fceff0b6121"}, - {file = "lxml-5.4.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:795f61bcaf8770e1b37eec24edf9771b307df3af74d1d6f27d812e15a9ff3872"}, - {file = "lxml-5.4.0-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:29f451a4b614a7b5b6c2e043d7b64a15bd8304d7e767055e8ab68387a8cacf4e"}, - {file = "lxml-5.4.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:891f7f991a68d20c75cb13c5c9142b2a3f9eb161f1f12a9489c82172d1f133c0"}, - {file = "lxml-5.4.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4aa412a82e460571fad592d0f93ce9935a20090029ba08eca05c614f99b0cc92"}, - {file = "lxml-5.4.0-cp37-cp37m-manylinux_2_28_aarch64.whl", hash = "sha256:ac7ba71f9561cd7d7b55e1ea5511543c0282e2b6450f122672a2694621d63b7e"}, - {file = "lxml-5.4.0-cp37-cp37m-manylinux_2_28_x86_64.whl", hash = "sha256:c5d32f5284012deaccd37da1e2cd42f081feaa76981f0eaa474351b68df813c5"}, - {file = "lxml-5.4.0-cp37-cp37m-musllinux_1_2_aarch64.whl", hash = "sha256:ce31158630a6ac85bddd6b830cffd46085ff90498b397bd0a259f59d27a12188"}, - {file = "lxml-5.4.0-cp37-cp37m-musllinux_1_2_x86_64.whl", hash = "sha256:31e63621e073e04697c1b2d23fcb89991790eef370ec37ce4d5d469f40924ed6"}, - {file = "lxml-5.4.0-cp37-cp37m-win32.whl", hash = "sha256:be2ba4c3c5b7900246a8f866580700ef0d538f2ca32535e991027bdaba944063"}, - {file = "lxml-5.4.0-cp37-cp37m-win_amd64.whl", hash = "sha256:09846782b1ef650b321484ad429217f5154da4d6e786636c38e434fa32e94e49"}, - {file = "lxml-5.4.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:eaf24066ad0b30917186420d51e2e3edf4b0e2ea68d8cd885b14dc8afdcf6556"}, - {file = "lxml-5.4.0-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:2b31a3a77501d86d8ade128abb01082724c0dfd9524f542f2f07d693c9f1175f"}, - {file = "lxml-5.4.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0e108352e203c7afd0eb91d782582f00a0b16a948d204d4dec8565024fafeea5"}, - {file = "lxml-5.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a11a96c3b3f7551c8a8109aa65e8594e551d5a84c76bf950da33d0fb6dfafab7"}, - {file = "lxml-5.4.0-cp38-cp38-manylinux_2_28_aarch64.whl", hash = "sha256:ca755eebf0d9e62d6cb013f1261e510317a41bf4650f22963474a663fdfe02aa"}, - {file = "lxml-5.4.0-cp38-cp38-manylinux_2_28_x86_64.whl", hash = "sha256:4cd915c0fb1bed47b5e6d6edd424ac25856252f09120e3e8ba5154b6b921860e"}, - {file = "lxml-5.4.0-cp38-cp38-musllinux_1_2_aarch64.whl", hash = "sha256:226046e386556a45ebc787871d6d2467b32c37ce76c2680f5c608e25823ffc84"}, - {file = "lxml-5.4.0-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:b108134b9667bcd71236c5a02aad5ddd073e372fb5d48ea74853e009fe38acb6"}, - {file = "lxml-5.4.0-cp38-cp38-win32.whl", hash = "sha256:1320091caa89805df7dcb9e908add28166113dcd062590668514dbd510798c88"}, - {file = "lxml-5.4.0-cp38-cp38-win_amd64.whl", hash = "sha256:073eb6dcdf1f587d9b88c8c93528b57eccda40209cf9be549d469b942b41d70b"}, - {file = "lxml-5.4.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:bda3ea44c39eb74e2488297bb39d47186ed01342f0022c8ff407c250ac3f498e"}, - {file = "lxml-5.4.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9ceaf423b50ecfc23ca00b7f50b64baba85fb3fb91c53e2c9d00bc86150c7e40"}, - {file = "lxml-5.4.0-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:664cdc733bc87449fe781dbb1f309090966c11cc0c0cd7b84af956a02a8a4729"}, - {file = "lxml-5.4.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:67ed8a40665b84d161bae3181aa2763beea3747f748bca5874b4af4d75998f87"}, - {file = "lxml-5.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9b4a3bd174cc9cdaa1afbc4620c049038b441d6ba07629d89a83b408e54c35cd"}, - {file = "lxml-5.4.0-cp39-cp39-manylinux_2_28_aarch64.whl", hash = "sha256:b0989737a3ba6cf2a16efb857fb0dfa20bc5c542737fddb6d893fde48be45433"}, - {file = "lxml-5.4.0-cp39-cp39-manylinux_2_28_x86_64.whl", hash = "sha256:dc0af80267edc68adf85f2a5d9be1cdf062f973db6790c1d065e45025fa26140"}, - {file = "lxml-5.4.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:639978bccb04c42677db43c79bdaa23785dc7f9b83bfd87570da8207872f1ce5"}, - {file = "lxml-5.4.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:5a99d86351f9c15e4a901fc56404b485b1462039db59288b203f8c629260a142"}, - {file = "lxml-5.4.0-cp39-cp39-win32.whl", hash = "sha256:3e6d5557989cdc3ebb5302bbdc42b439733a841891762ded9514e74f60319ad6"}, - {file = "lxml-5.4.0-cp39-cp39-win_amd64.whl", hash = "sha256:a8c9b7f16b63e65bbba889acb436a1034a82d34fa09752d754f88d708eca80e1"}, - {file = "lxml-5.4.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:1b717b00a71b901b4667226bba282dd462c42ccf618ade12f9ba3674e1fabc55"}, - {file = "lxml-5.4.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:27a9ded0f0b52098ff89dd4c418325b987feed2ea5cc86e8860b0f844285d740"}, - {file = "lxml-5.4.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4b7ce10634113651d6f383aa712a194179dcd496bd8c41e191cec2099fa09de5"}, - {file = "lxml-5.4.0-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:53370c26500d22b45182f98847243efb518d268374a9570409d2e2276232fd37"}, - {file = "lxml-5.4.0-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:c6364038c519dffdbe07e3cf42e6a7f8b90c275d4d1617a69bb59734c1a2d571"}, - {file = "lxml-5.4.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:b12cb6527599808ada9eb2cd6e0e7d3d8f13fe7bbb01c6311255a15ded4c7ab4"}, - {file = "lxml-5.4.0-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:5f11a1526ebd0dee85e7b1e39e39a0cc0d9d03fb527f56d8457f6df48a10dc0c"}, - {file = "lxml-5.4.0-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:48b4afaf38bf79109bb060d9016fad014a9a48fb244e11b94f74ae366a64d252"}, - {file = "lxml-5.4.0-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:de6f6bb8a7840c7bf216fb83eec4e2f79f7325eca8858167b68708b929ab2172"}, - {file = "lxml-5.4.0-pp37-pypy37_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:5cca36a194a4eb4e2ed6be36923d3cffd03dcdf477515dea687185506583d4c9"}, - {file = "lxml-5.4.0-pp37-pypy37_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:b7c86884ad23d61b025989d99bfdd92a7351de956e01c61307cb87035960bcb1"}, - {file = "lxml-5.4.0-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:53d9469ab5460402c19553b56c3648746774ecd0681b1b27ea74d5d8a3ef5590"}, - {file = "lxml-5.4.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:56dbdbab0551532bb26c19c914848d7251d73edb507c3079d6805fa8bba5b706"}, - {file = "lxml-5.4.0-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:14479c2ad1cb08b62bb941ba8e0e05938524ee3c3114644df905d2331c76cd57"}, - {file = "lxml-5.4.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:32697d2ea994e0db19c1df9e40275ffe84973e4232b5c274f47e7c1ec9763cdd"}, - {file = "lxml-5.4.0-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:24f6df5f24fc3385f622c0c9d63fe34604893bc1a5bdbb2dbf5870f85f9a404a"}, - {file = "lxml-5.4.0-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:151d6c40bc9db11e960619d2bf2ec5829f0aaffb10b41dcf6ad2ce0f3c0b2325"}, - {file = "lxml-5.4.0-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:4025bf2884ac4370a3243c5aa8d66d3cb9e15d3ddd0af2d796eccc5f0244390e"}, - {file = "lxml-5.4.0-pp39-pypy39_pp73-macosx_10_15_x86_64.whl", hash = "sha256:9459e6892f59ecea2e2584ee1058f5d8f629446eab52ba2305ae13a32a059530"}, - {file = "lxml-5.4.0-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:47fb24cc0f052f0576ea382872b3fc7e1f7e3028e53299ea751839418ade92a6"}, - {file = "lxml-5.4.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:50441c9de951a153c698b9b99992e806b71c1f36d14b154592580ff4a9d0d877"}, - {file = "lxml-5.4.0-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:ab339536aa798b1e17750733663d272038bf28069761d5be57cb4a9b0137b4f8"}, - {file = "lxml-5.4.0-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:9776af1aad5a4b4a1317242ee2bea51da54b2a7b7b48674be736d463c999f37d"}, - {file = "lxml-5.4.0-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:63e7968ff83da2eb6fdda967483a7a023aa497d85ad8f05c3ad9b1f2e8c84987"}, - {file = "lxml-5.4.0.tar.gz", hash = "sha256:d12832e1dbea4be280b22fd0ea7c9b87f0d8fc51ba06e92dc62d52f804f78ebd"}, +markers = "extra == \"mem-reader\" or extra == \"all\"" +files = [ + {file = "lxml-6.0.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:35bc626eec405f745199200ccb5c6b36f202675d204aa29bb52e27ba2b71dea8"}, + {file = "lxml-6.0.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:246b40f8a4aec341cbbf52617cad8ab7c888d944bfe12a6abd2b1f6cfb6f6082"}, + {file = "lxml-6.0.0-cp310-cp310-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:2793a627e95d119e9f1e19720730472f5543a6d84c50ea33313ce328d870f2dd"}, + {file = "lxml-6.0.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:46b9ed911f36bfeb6338e0b482e7fe7c27d362c52fde29f221fddbc9ee2227e7"}, + {file = "lxml-6.0.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2b4790b558bee331a933e08883c423f65bbcd07e278f91b2272489e31ab1e2b4"}, + {file = "lxml-6.0.0-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e2030956cf4886b10be9a0285c6802e078ec2391e1dd7ff3eb509c2c95a69b76"}, + {file = "lxml-6.0.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4d23854ecf381ab1facc8f353dcd9adeddef3652268ee75297c1164c987c11dc"}, + {file = "lxml-6.0.0-cp310-cp310-manylinux_2_31_armv7l.whl", hash = "sha256:43fe5af2d590bf4691531b1d9a2495d7aab2090547eaacd224a3afec95706d76"}, + {file = "lxml-6.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:74e748012f8c19b47f7d6321ac929a9a94ee92ef12bc4298c47e8b7219b26541"}, + {file = "lxml-6.0.0-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:43cfbb7db02b30ad3926e8fceaef260ba2fb7df787e38fa2df890c1ca7966c3b"}, + {file = "lxml-6.0.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:34190a1ec4f1e84af256495436b2d196529c3f2094f0af80202947567fdbf2e7"}, + {file = "lxml-6.0.0-cp310-cp310-win32.whl", hash = "sha256:5967fe415b1920a3877a4195e9a2b779249630ee49ece22021c690320ff07452"}, + {file = "lxml-6.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:f3389924581d9a770c6caa4df4e74b606180869043b9073e2cec324bad6e306e"}, + {file = "lxml-6.0.0-cp310-cp310-win_arm64.whl", hash = "sha256:522fe7abb41309e9543b0d9b8b434f2b630c5fdaf6482bee642b34c8c70079c8"}, + {file = "lxml-6.0.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:4ee56288d0df919e4aac43b539dd0e34bb55d6a12a6562038e8d6f3ed07f9e36"}, + {file = "lxml-6.0.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:b8dd6dd0e9c1992613ccda2bcb74fc9d49159dbe0f0ca4753f37527749885c25"}, + {file = "lxml-6.0.0-cp311-cp311-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:d7ae472f74afcc47320238b5dbfd363aba111a525943c8a34a1b657c6be934c3"}, + {file = "lxml-6.0.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5592401cdf3dc682194727c1ddaa8aa0f3ddc57ca64fd03226a430b955eab6f6"}, + {file = "lxml-6.0.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:58ffd35bd5425c3c3b9692d078bf7ab851441434531a7e517c4984d5634cd65b"}, + {file = "lxml-6.0.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f720a14aa102a38907c6d5030e3d66b3b680c3e6f6bc95473931ea3c00c59967"}, + {file = "lxml-6.0.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c2a5e8d207311a0170aca0eb6b160af91adc29ec121832e4ac151a57743a1e1e"}, + {file = "lxml-6.0.0-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:2dd1cc3ea7e60bfb31ff32cafe07e24839df573a5e7c2d33304082a5019bcd58"}, + {file = "lxml-6.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:2cfcf84f1defed7e5798ef4f88aa25fcc52d279be731ce904789aa7ccfb7e8d2"}, + {file = "lxml-6.0.0-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:a52a4704811e2623b0324a18d41ad4b9fabf43ce5ff99b14e40a520e2190c851"}, + {file = "lxml-6.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:c16304bba98f48a28ae10e32a8e75c349dd742c45156f297e16eeb1ba9287a1f"}, + {file = "lxml-6.0.0-cp311-cp311-win32.whl", hash = "sha256:f8d19565ae3eb956d84da3ef367aa7def14a2735d05bd275cd54c0301f0d0d6c"}, + {file = "lxml-6.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:b2d71cdefda9424adff9a3607ba5bbfc60ee972d73c21c7e3c19e71037574816"}, + {file = "lxml-6.0.0-cp311-cp311-win_arm64.whl", hash = "sha256:8a2e76efbf8772add72d002d67a4c3d0958638696f541734304c7f28217a9cab"}, + {file = "lxml-6.0.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:78718d8454a6e928470d511bf8ac93f469283a45c354995f7d19e77292f26108"}, + {file = "lxml-6.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:84ef591495ffd3f9dcabffd6391db7bb70d7230b5c35ef5148354a134f56f2be"}, + {file = "lxml-6.0.0-cp312-cp312-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:2930aa001a3776c3e2601cb8e0a15d21b8270528d89cc308be4843ade546b9ab"}, + {file = "lxml-6.0.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:219e0431ea8006e15005767f0351e3f7f9143e793e58519dc97fe9e07fae5563"}, + {file = "lxml-6.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bd5913b4972681ffc9718bc2d4c53cde39ef81415e1671ff93e9aa30b46595e7"}, + {file = "lxml-6.0.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:390240baeb9f415a82eefc2e13285016f9c8b5ad71ec80574ae8fa9605093cd7"}, + {file = "lxml-6.0.0-cp312-cp312-manylinux_2_27_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d6e200909a119626744dd81bae409fc44134389e03fbf1d68ed2a55a2fb10991"}, + {file = "lxml-6.0.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ca50bd612438258a91b5b3788c6621c1f05c8c478e7951899f492be42defc0da"}, + {file = "lxml-6.0.0-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:c24b8efd9c0f62bad0439283c2c795ef916c5a6b75f03c17799775c7ae3c0c9e"}, + {file = "lxml-6.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:afd27d8629ae94c5d863e32ab0e1d5590371d296b87dae0a751fb22bf3685741"}, + {file = "lxml-6.0.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:54c4855eabd9fc29707d30141be99e5cd1102e7d2258d2892314cf4c110726c3"}, + {file = "lxml-6.0.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:c907516d49f77f6cd8ead1322198bdfd902003c3c330c77a1c5f3cc32a0e4d16"}, + {file = "lxml-6.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:36531f81c8214e293097cd2b7873f178997dae33d3667caaae8bdfb9666b76c0"}, + {file = "lxml-6.0.0-cp312-cp312-win32.whl", hash = "sha256:690b20e3388a7ec98e899fd54c924e50ba6693874aa65ef9cb53de7f7de9d64a"}, + {file = "lxml-6.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:310b719b695b3dd442cdfbbe64936b2f2e231bb91d998e99e6f0daf991a3eba3"}, + {file = "lxml-6.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:8cb26f51c82d77483cdcd2b4a53cda55bbee29b3c2f3ddeb47182a2a9064e4eb"}, + {file = "lxml-6.0.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:6da7cd4f405fd7db56e51e96bff0865b9853ae70df0e6720624049da76bde2da"}, + {file = "lxml-6.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:b34339898bb556a2351a1830f88f751679f343eabf9cf05841c95b165152c9e7"}, + {file = "lxml-6.0.0-cp313-cp313-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:51a5e4c61a4541bd1cd3ba74766d0c9b6c12d6a1a4964ef60026832aac8e79b3"}, + {file = "lxml-6.0.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d18a25b19ca7307045581b18b3ec9ead2b1db5ccd8719c291f0cd0a5cec6cb81"}, + {file = "lxml-6.0.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d4f0c66df4386b75d2ab1e20a489f30dc7fd9a06a896d64980541506086be1f1"}, + {file = "lxml-6.0.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9f4b481b6cc3a897adb4279216695150bbe7a44c03daba3c894f49d2037e0a24"}, + {file = "lxml-6.0.0-cp313-cp313-manylinux_2_27_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8a78d6c9168f5bcb20971bf3329c2b83078611fbe1f807baadc64afc70523b3a"}, + {file = "lxml-6.0.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2ae06fbab4f1bb7db4f7c8ca9897dc8db4447d1a2b9bee78474ad403437bcc29"}, + {file = "lxml-6.0.0-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:1fa377b827ca2023244a06554c6e7dc6828a10aaf74ca41965c5d8a4925aebb4"}, + {file = "lxml-6.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1676b56d48048a62ef77a250428d1f31f610763636e0784ba67a9740823988ca"}, + {file = "lxml-6.0.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:0e32698462aacc5c1cf6bdfebc9c781821b7e74c79f13e5ffc8bfe27c42b1abf"}, + {file = "lxml-6.0.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4d6036c3a296707357efb375cfc24bb64cd955b9ec731abf11ebb1e40063949f"}, + {file = "lxml-6.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7488a43033c958637b1a08cddc9188eb06d3ad36582cebc7d4815980b47e27ef"}, + {file = "lxml-6.0.0-cp313-cp313-win32.whl", hash = "sha256:5fcd7d3b1d8ecb91445bd71b9c88bdbeae528fefee4f379895becfc72298d181"}, + {file = "lxml-6.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:2f34687222b78fff795feeb799a7d44eca2477c3d9d3a46ce17d51a4f383e32e"}, + {file = "lxml-6.0.0-cp313-cp313-win_arm64.whl", hash = "sha256:21db1ec5525780fd07251636eb5f7acb84003e9382c72c18c542a87c416ade03"}, + {file = "lxml-6.0.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:4eb114a0754fd00075c12648d991ec7a4357f9cb873042cc9a77bf3a7e30c9db"}, + {file = "lxml-6.0.0-cp38-cp38-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:7da298e1659e45d151b4028ad5c7974917e108afb48731f4ed785d02b6818994"}, + {file = "lxml-6.0.0-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7bf61bc4345c1895221357af8f3e89f8c103d93156ef326532d35c707e2fb19d"}, + {file = "lxml-6.0.0-cp38-cp38-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:63b634facdfbad421d4b61c90735688465d4ab3a8853ac22c76ccac2baf98d97"}, + {file = "lxml-6.0.0-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:e380e85b93f148ad28ac15f8117e2fd8e5437aa7732d65e260134f83ce67911b"}, + {file = "lxml-6.0.0-cp38-cp38-win32.whl", hash = "sha256:185efc2fed89cdd97552585c624d3c908f0464090f4b91f7d92f8ed2f3b18f54"}, + {file = "lxml-6.0.0-cp38-cp38-win_amd64.whl", hash = "sha256:f97487996a39cb18278ca33f7be98198f278d0bc3c5d0fd4d7b3d63646ca3c8a"}, + {file = "lxml-6.0.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:85b14a4689d5cff426c12eefe750738648706ea2753b20c2f973b2a000d3d261"}, + {file = "lxml-6.0.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:f64ccf593916e93b8d36ed55401bb7fe9c7d5de3180ce2e10b08f82a8f397316"}, + {file = "lxml-6.0.0-cp39-cp39-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:b372d10d17a701b0945f67be58fae4664fd056b85e0ff0fbc1e6c951cdbc0512"}, + {file = "lxml-6.0.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a674c0948789e9136d69065cc28009c1b1874c6ea340253db58be7622ce6398f"}, + {file = "lxml-6.0.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:edf6e4c8fe14dfe316939711e3ece3f9a20760aabf686051b537a7562f4da91a"}, + {file = "lxml-6.0.0-cp39-cp39-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:048a930eb4572829604982e39a0c7289ab5dc8abc7fc9f5aabd6fbc08c154e93"}, + {file = "lxml-6.0.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c0b5fa5eda84057a4f1bbb4bb77a8c28ff20ae7ce211588d698ae453e13c6281"}, + {file = "lxml-6.0.0-cp39-cp39-manylinux_2_31_armv7l.whl", hash = "sha256:c352fc8f36f7e9727db17adbf93f82499457b3d7e5511368569b4c5bd155a922"}, + {file = "lxml-6.0.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:8db5dc617cb937ae17ff3403c3a70a7de9df4852a046f93e71edaec678f721d0"}, + {file = "lxml-6.0.0-cp39-cp39-musllinux_1_2_armv7l.whl", hash = "sha256:2181e4b1d07dde53986023482673c0f1fba5178ef800f9ab95ad791e8bdded6a"}, + {file = "lxml-6.0.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:b3c98d5b24c6095e89e03d65d5c574705be3d49c0d8ca10c17a8a4b5201b72f5"}, + {file = "lxml-6.0.0-cp39-cp39-win32.whl", hash = "sha256:04d67ceee6db4bcb92987ccb16e53bef6b42ced872509f333c04fb58a3315256"}, + {file = "lxml-6.0.0-cp39-cp39-win_amd64.whl", hash = "sha256:e0b1520ef900e9ef62e392dd3d7ae4f5fa224d1dd62897a792cf353eb20b6cae"}, + {file = "lxml-6.0.0-cp39-cp39-win_arm64.whl", hash = "sha256:e35e8aaaf3981489f42884b59726693de32dabfc438ac10ef4eb3409961fd402"}, + {file = "lxml-6.0.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:dbdd7679a6f4f08152818043dbb39491d1af3332128b3752c3ec5cebc0011a72"}, + {file = "lxml-6.0.0-pp310-pypy310_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:40442e2a4456e9910875ac12951476d36c0870dcb38a68719f8c4686609897c4"}, + {file = "lxml-6.0.0-pp310-pypy310_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:db0efd6bae1c4730b9c863fc4f5f3c0fa3e8f05cae2c44ae141cb9dfc7d091dc"}, + {file = "lxml-6.0.0-pp310-pypy310_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9ab542c91f5a47aaa58abdd8ea84b498e8e49fe4b883d67800017757a3eb78e8"}, + {file = "lxml-6.0.0-pp310-pypy310_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:013090383863b72c62a702d07678b658fa2567aa58d373d963cca245b017e065"}, + {file = "lxml-6.0.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:c86df1c9af35d903d2b52d22ea3e66db8058d21dc0f59842ca5deb0595921141"}, + {file = "lxml-6.0.0-pp39-pypy39_pp73-macosx_10_15_x86_64.whl", hash = "sha256:4337e4aec93b7c011f7ee2e357b0d30562edd1955620fdd4aeab6aacd90d43c5"}, + {file = "lxml-6.0.0-pp39-pypy39_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ae74f7c762270196d2dda56f8dd7309411f08a4084ff2dfcc0b095a218df2e06"}, + {file = "lxml-6.0.0-pp39-pypy39_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:059c4cbf3973a621b62ea3132934ae737da2c132a788e6cfb9b08d63a0ef73f9"}, + {file = "lxml-6.0.0-pp39-pypy39_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:17f090a9bc0ce8da51a5632092f98a7e7f84bca26f33d161a98b57f7fb0004ca"}, + {file = "lxml-6.0.0-pp39-pypy39_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9da022c14baeec36edfcc8daf0e281e2f55b950249a455776f0d1adeeada4734"}, + {file = "lxml-6.0.0-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:a55da151d0b0c6ab176b4e761670ac0e2667817a1e0dadd04a01d0561a219349"}, + {file = "lxml-6.0.0.tar.gz", hash = "sha256:032e65120339d44cdc3efc326c9f660f5f7205f3a535c1fdbf898b29ea01fb72"}, ] [package.extras] @@ -1954,15 +2138,15 @@ cssselect = ["cssselect (>=0.7)"] html-clean = ["lxml_html_clean"] html5 = ["html5lib"] htmlsoup = ["BeautifulSoup4"] -source = ["Cython (>=3.0.11,<3.1.0)"] [[package]] name = "magika" version = "0.6.2" description = "A tool to determine the content type of a file with deep learning" -optional = false +optional = true python-versions = ">=3.8" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "magika-0.6.2-py3-none-any.whl", hash = "sha256:5ef72fbc07723029b3684ef81454bc224ac5f60986aa0fc5a28f4456eebcb5b2"}, {file = "magika-0.6.2-py3-none-macosx_11_0_arm64.whl", hash = "sha256:9109309328a1553886c8ff36c2ee9a5e9cfd36893ad81b65bf61a57debdd9d0e"}, @@ -1983,14 +2167,15 @@ python-dotenv = ">=1.0.1" [[package]] name = "mammoth" -version = "1.9.0" +version = "1.9.1" description = "Convert Word documents from docx to simple and clean HTML and Markdown" -optional = false +optional = true python-versions = ">=3.7" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ - {file = "mammoth-1.9.0-py2.py3-none-any.whl", hash = "sha256:0eea277316586f0ca65d86834aec4de5a0572c83ec54b4991f9bb520a891150f"}, - {file = "mammoth-1.9.0.tar.gz", hash = "sha256:74f5dae10ca240fd9b7a0e1a6deaebe0aad23bc590633ef6f5e868aa9b7042a6"}, + {file = "mammoth-1.9.1-py2.py3-none-any.whl", hash = "sha256:f0569bd640cee6c77a07e7c75c5dc10d745dc4dc95d530cfcbb0a5d9536d636c"}, + {file = "mammoth-1.9.1.tar.gz", hash = "sha256:7924254ab8f03efe55fadc0fd5f7828db831190eb2679d63cb4372873e71c572"}, ] [package.dependencies] @@ -2025,9 +2210,10 @@ testing = ["coverage", "pytest", "pytest-cov", "pytest-regressions"] name = "markdownify" version = "1.1.0" description = "Convert HTML to markdown." -optional = false +optional = true python-versions = "*" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "markdownify-1.1.0-py3-none-any.whl", hash = "sha256:32a5a08e9af02c8a6528942224c91b933b4bd2c7d078f9012943776fc313eeef"}, {file = "markdownify-1.1.0.tar.gz", hash = "sha256:449c0bbbf1401c5112379619524f33b63490a8fa479456d41de9dc9e37560ebd"}, @@ -2039,19 +2225,22 @@ six = ">=1.15,<2" [[package]] name = "markitdown" -version = "0.1.1" +version = "0.1.2" description = "Utility tool for converting various files to Markdown" -optional = false +optional = true python-versions = ">=3.10" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ - {file = "markitdown-0.1.1-py3-none-any.whl", hash = "sha256:98ea8c009fe174b37ef933e00f4364214e8fed35691178b8521b13604d0c4a58"}, - {file = "markitdown-0.1.1.tar.gz", hash = "sha256:da97a55a45a3d775ea758e88a344d5cac94ee97115fb0293f99027d32c2fc3f6"}, + {file = "markitdown-0.1.2-py3-none-any.whl", hash = "sha256:4881f0768794ffccb52d09dd86498813a6896ba9639b4fc15512817f56ed9d74"}, + {file = "markitdown-0.1.2.tar.gz", hash = "sha256:85fe108a92bd18f317e75a36cf567a6fa812072612a898abf8c156d5d74c13c4"}, ] [package.dependencies] beautifulsoup4 = "*" charset-normalizer = "*" +defusedxml = "*" +lxml = {version = "*", optional = true, markers = "extra == \"docx\""} magika = ">=0.6.1,<0.7.0" mammoth = {version = "*", optional = true, markers = "extra == \"docx\""} markdownify = "*" @@ -2063,10 +2252,10 @@ requests = "*" xlrd = {version = "*", optional = true, markers = "extra == \"xls\""} [package.extras] -all = ["azure-ai-documentintelligence", "azure-identity", "mammoth", "olefile", "openpyxl", "pandas", "pdfminer-six", "pydub", "python-pptx", "speechrecognition", "xlrd", "youtube-transcript-api (>=1.0.0,<1.1.0)"] +all = ["azure-ai-documentintelligence", "azure-identity", "lxml", "mammoth", "olefile", "openpyxl", "pandas", "pdfminer-six", "pydub", "python-pptx", "speechrecognition", "xlrd", "youtube-transcript-api (>=1.0.0,<1.1.0)"] audio-transcription = ["pydub", "speechrecognition"] az-doc-intel = ["azure-ai-documentintelligence", "azure-identity"] -docx = ["mammoth"] +docx = ["lxml", "mammoth"] outlook = ["olefile"] pdf = ["pdfminer-six"] pptx = ["python-pptx"] @@ -2203,6 +2392,36 @@ python-dateutil = ">=2.7" [package.extras] dev = ["meson-python (>=0.13.1,<0.17.0)", "pybind11 (>=2.13.2,!=2.13.3)", "setuptools (>=64)", "setuptools_scm (>=7)"] +[[package]] +name = "mcp" +version = "1.12.0" +description = "Model Context Protocol SDK" +optional = false +python-versions = ">=3.10" +groups = ["main"] +files = [ + {file = "mcp-1.12.0-py3-none-any.whl", hash = "sha256:19a498b2bf273283e463b4dd1ed83f791fbba5c25bfa16b8b34cfd5571673e7f"}, + {file = "mcp-1.12.0.tar.gz", hash = "sha256:853f6b17a3f31ea6e2f278c2ec7d3b38457bc80c7c2c675260dd7f04a6fd0e70"}, +] + +[package.dependencies] +anyio = ">=4.5" +httpx = ">=0.27" +httpx-sse = ">=0.4" +jsonschema = ">=4.20.0" +pydantic = ">=2.8.0,<3.0.0" +pydantic-settings = ">=2.5.2" +python-multipart = ">=0.0.9" +pywin32 = {version = ">=310", markers = "sys_platform == \"win32\""} +sse-starlette = ">=1.6.1" +starlette = ">=0.27" +uvicorn = {version = ">=0.23.1", markers = "sys_platform != \"emscripten\""} + +[package.extras] +cli = ["python-dotenv (>=1.0.0)", "typer (>=0.16.0)"] +rich = ["rich (>=13.9.4)"] +ws = ["websockets (>=15.0.1)"] + [[package]] name = "mdurl" version = "0.1.2" @@ -2217,14 +2436,14 @@ files = [ [[package]] name = "mem0ai" -version = "0.1.109" +version = "0.1.114" description = "Long-term memory for AI Agents" optional = false python-versions = "<4.0,>=3.9" groups = ["eval"] files = [ - {file = "mem0ai-0.1.109-py3-none-any.whl", hash = "sha256:b173687a40a4aa2b59277666c180efcad66a751ebbaa2c1f0eea3d671fa0e07a"}, - {file = "mem0ai-0.1.109.tar.gz", hash = "sha256:7c8ebbfa10d2949c23ac55721241303f982853e9a0527c6f31f6c66477632bef"}, + {file = "mem0ai-0.1.114-py3-none-any.whl", hash = "sha256:dfb7f0079ee282f5d9782e220f6f09707bcf5e107925d1901dbca30d8dd83f9b"}, + {file = "mem0ai-0.1.114.tar.gz", hash = "sha256:b27886132eaec78544e8b8b54f0b14a36728f3c99da54cb7cb417150e2fad7e1"}, ] [package.dependencies] @@ -2237,11 +2456,11 @@ sqlalchemy = ">=2.0.31" [package.extras] dev = ["isort (>=5.13.2)", "pytest (>=8.2.2)", "ruff (>=0.6.5)"] -extras = ["boto3 (>=1.34.0)", "elasticsearch (>=8.0.0)", "langchain-community (>=0.0.0)", "langchain-memgraph (>=0.1.0)", "opensearch-py (>=2.0.0)", "sentence-transformers (>=2.2.2)"] -graph = ["langchain-neo4j (>=0.4.0)", "neo4j (>=5.23.1)", "rank-bm25 (>=0.2.2)"] +extras = ["boto3 (>=1.34.0)", "elasticsearch (>=8.0.0)", "langchain-community (>=0.0.0)", "langchain-memgraph (>=0.1.0)", "opensearch-py (>=2.0.0)", "sentence-transformers (>=5.0.0)"] +graph = ["langchain-aws (>=0.2.23)", "langchain-neo4j (>=0.4.0)", "neo4j (>=5.23.1)", "rank-bm25 (>=0.2.2)"] llms = ["google-genai (>=1.0.0)", "google-generativeai (>=0.3.0)", "groq (>=0.3.0)", "litellm (>=0.1.0)", "ollama (>=0.1.0)", "together (>=0.2.10)", "vertexai (>=0.1.0)"] test = ["pytest (>=8.2.2)", "pytest-asyncio (>=0.23.7)", "pytest-mock (>=3.14.0)"] -vector-stores = ["azure-search-documents (>=11.4.0b8)", "chromadb (>=0.4.24)", "faiss-cpu (>=1.7.4)", "pinecone (<7.0.0)", "pinecone-text (>=0.1.1)", "upstash-vector (>=0.1.0)", "vecs (>=0.4.0)", "weaviate-client (>=4.4.0)"] +vector-stores = ["azure-search-documents (>=11.4.0b8)", "chromadb (>=0.4.24)", "faiss-cpu (>=1.7.4)", "pinecone (<=7.3.0)", "pinecone-text (>=0.10.0)", "pymochow (>=2.2.9)", "pymongo (>=4.13.2)", "upstash-vector (>=0.1.0)", "vecs (>=0.4.0)", "weaviate-client (>=4.4.0)"] [[package]] name = "mpmath" @@ -2254,6 +2473,7 @@ files = [ {file = "mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c"}, {file = "mpmath-1.3.0.tar.gz", hash = "sha256:7a28eb2a9774d00c7bc92411c19a89209d5da7c4c9a9e227be8330a23a25b91f"}, ] +markers = {main = "extra == \"mem-reader\" or extra == \"all\""} [package.extras] develop = ["codecov", "pycodestyle", "pytest (>=4.6)", "pytest-cov", "wheel"] @@ -2265,9 +2485,10 @@ tests = ["pytest (>=4.6)"] name = "neo4j" version = "5.28.1" description = "Neo4j Bolt driver for Python" -optional = false +optional = true python-versions = ">=3.7" groups = ["main"] +markers = "extra == \"tree-mem\" or extra == \"all\"" files = [ {file = "neo4j-5.28.1-py3-none-any.whl", hash = "sha256:6755ef9e5f4e14b403aef1138fb6315b120631a0075c138b5ddb2a06b87b09fd"}, {file = "neo4j-5.28.1.tar.gz", hash = "sha256:ae8e37a1d895099062c75bc359b2cce62099baac7be768d0eba7180c1298e214"}, @@ -2288,11 +2509,11 @@ description = "Python package for creating and manipulating graphs and networks" optional = false python-versions = ">=3.10" groups = ["main", "eval"] -markers = "python_version == \"3.10\"" files = [ {file = "networkx-3.4.2-py3-none-any.whl", hash = "sha256:df5d4365b724cf81b8c6a7312509d0c22386097011ad1abe274afd5e9d3bbc5f"}, {file = "networkx-3.4.2.tar.gz", hash = "sha256:307c3669428c5362aab27c8a1260aa8f47c4e91d3891f48be0141738d8d053e1"}, ] +markers = {main = "python_version == \"3.10\" and extra == \"all\"", eval = "python_version == \"3.10\""} [package.extras] default = ["matplotlib (>=3.7)", "numpy (>=1.24)", "pandas (>=2.0)", "scipy (>=1.10,!=1.11.0,!=1.11.1)"] @@ -2309,11 +2530,11 @@ description = "Python package for creating and manipulating graphs and networks" optional = false python-versions = ">=3.11" groups = ["main", "eval"] -markers = "python_version >= \"3.11\"" files = [ {file = "networkx-3.5-py3-none-any.whl", hash = "sha256:0030d386a9a06dee3565298b4a734b68589749a544acbb6c412dc9e2489ec6ec"}, {file = "networkx-3.5.tar.gz", hash = "sha256:d4c6f9cf81f52d69230866796b82afbccdec3db7ae4fbd1b65ea750feed50037"}, ] +markers = {main = "python_version >= \"3.11\" and extra == \"all\"", eval = "python_version >= \"3.11\""} [package.extras] default = ["matplotlib (>=3.8)", "numpy (>=1.25)", "pandas (>=2.0)", "scipy (>=1.11.2)"] @@ -2364,67 +2585,130 @@ files = [ [[package]] name = "numpy" -version = "2.2.5" +version = "2.2.6" description = "Fundamental package for array computing in Python" optional = false python-versions = ">=3.10" groups = ["main", "eval"] +markers = "python_version == \"3.10\"" files = [ - {file = "numpy-2.2.5-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:1f4a922da1729f4c40932b2af4fe84909c7a6e167e6e99f71838ce3a29f3fe26"}, - {file = "numpy-2.2.5-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:b6f91524d31b34f4a5fee24f5bc16dcd1491b668798b6d85585d836c1e633a6a"}, - {file = "numpy-2.2.5-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:19f4718c9012e3baea91a7dba661dcab2451cda2550678dc30d53acb91a7290f"}, - {file = "numpy-2.2.5-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:eb7fd5b184e5d277afa9ec0ad5e4eb562ecff541e7f60e69ee69c8d59e9aeaba"}, - {file = "numpy-2.2.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6413d48a9be53e183eb06495d8e3b006ef8f87c324af68241bbe7a39e8ff54c3"}, - {file = "numpy-2.2.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7451f92eddf8503c9b8aa4fe6aa7e87fd51a29c2cfc5f7dbd72efde6c65acf57"}, - {file = "numpy-2.2.5-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:0bcb1d057b7571334139129b7f941588f69ce7c4ed15a9d6162b2ea54ded700c"}, - {file = "numpy-2.2.5-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:36ab5b23915887543441efd0417e6a3baa08634308894316f446027611b53bf1"}, - {file = "numpy-2.2.5-cp310-cp310-win32.whl", hash = "sha256:422cc684f17bc963da5f59a31530b3936f57c95a29743056ef7a7903a5dbdf88"}, - {file = "numpy-2.2.5-cp310-cp310-win_amd64.whl", hash = "sha256:e4f0b035d9d0ed519c813ee23e0a733db81ec37d2e9503afbb6e54ccfdee0fa7"}, - {file = "numpy-2.2.5-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c42365005c7a6c42436a54d28c43fe0e01ca11eb2ac3cefe796c25a5f98e5e9b"}, - {file = "numpy-2.2.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:498815b96f67dc347e03b719ef49c772589fb74b8ee9ea2c37feae915ad6ebda"}, - {file = "numpy-2.2.5-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:6411f744f7f20081b1b4e7112e0f4c9c5b08f94b9f086e6f0adf3645f85d3a4d"}, - {file = "numpy-2.2.5-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:9de6832228f617c9ef45d948ec1cd8949c482238d68b2477e6f642c33a7b0a54"}, - {file = "numpy-2.2.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:369e0d4647c17c9363244f3468f2227d557a74b6781cb62ce57cf3ef5cc7c610"}, - {file = "numpy-2.2.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:262d23f383170f99cd9191a7c85b9a50970fe9069b2f8ab5d786eca8a675d60b"}, - {file = "numpy-2.2.5-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:aa70fdbdc3b169d69e8c59e65c07a1c9351ceb438e627f0fdcd471015cd956be"}, - {file = "numpy-2.2.5-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37e32e985f03c06206582a7323ef926b4e78bdaa6915095ef08070471865b906"}, - {file = "numpy-2.2.5-cp311-cp311-win32.whl", hash = "sha256:f5045039100ed58fa817a6227a356240ea1b9a1bc141018864c306c1a16d4175"}, - {file = "numpy-2.2.5-cp311-cp311-win_amd64.whl", hash = "sha256:b13f04968b46ad705f7c8a80122a42ae8f620536ea38cf4bdd374302926424dd"}, - {file = "numpy-2.2.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ee461a4eaab4f165b68780a6a1af95fb23a29932be7569b9fab666c407969051"}, - {file = "numpy-2.2.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ec31367fd6a255dc8de4772bd1658c3e926d8e860a0b6e922b615e532d320ddc"}, - {file = "numpy-2.2.5-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:47834cde750d3c9f4e52c6ca28a7361859fcaf52695c7dc3cc1a720b8922683e"}, - {file = "numpy-2.2.5-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:2c1a1c6ccce4022383583a6ded7bbcda22fc635eb4eb1e0a053336425ed36dfa"}, - {file = "numpy-2.2.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9d75f338f5f79ee23548b03d801d28a505198297534f62416391857ea0479571"}, - {file = "numpy-2.2.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3a801fef99668f309b88640e28d261991bfad9617c27beda4a3aec4f217ea073"}, - {file = "numpy-2.2.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:abe38cd8381245a7f49967a6010e77dbf3680bd3627c0fe4362dd693b404c7f8"}, - {file = "numpy-2.2.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5a0ac90e46fdb5649ab6369d1ab6104bfe5854ab19b645bf5cda0127a13034ae"}, - {file = "numpy-2.2.5-cp312-cp312-win32.whl", hash = "sha256:0cd48122a6b7eab8f06404805b1bd5856200e3ed6f8a1b9a194f9d9054631beb"}, - {file = "numpy-2.2.5-cp312-cp312-win_amd64.whl", hash = "sha256:ced69262a8278547e63409b2653b372bf4baff0870c57efa76c5703fd6543282"}, - {file = "numpy-2.2.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:059b51b658f4414fff78c6d7b1b4e18283ab5fa56d270ff212d5ba0c561846f4"}, - {file = "numpy-2.2.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:47f9ed103af0bc63182609044b0490747e03bd20a67e391192dde119bf43d52f"}, - {file = "numpy-2.2.5-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:261a1ef047751bb02f29dfe337230b5882b54521ca121fc7f62668133cb119c9"}, - {file = "numpy-2.2.5-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:4520caa3807c1ceb005d125a75e715567806fed67e315cea619d5ec6e75a4191"}, - {file = "numpy-2.2.5-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3d14b17b9be5f9c9301f43d2e2a4886a33b53f4e6fdf9ca2f4cc60aeeee76372"}, - {file = "numpy-2.2.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2ba321813a00e508d5421104464510cc962a6f791aa2fca1c97b1e65027da80d"}, - {file = "numpy-2.2.5-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a4cbdef3ddf777423060c6f81b5694bad2dc9675f110c4b2a60dc0181543fac7"}, - {file = "numpy-2.2.5-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:54088a5a147ab71a8e7fdfd8c3601972751ded0739c6b696ad9cb0343e21ab73"}, - {file = "numpy-2.2.5-cp313-cp313-win32.whl", hash = "sha256:c8b82a55ef86a2d8e81b63da85e55f5537d2157165be1cb2ce7cfa57b6aef38b"}, - {file = "numpy-2.2.5-cp313-cp313-win_amd64.whl", hash = "sha256:d8882a829fd779f0f43998e931c466802a77ca1ee0fe25a3abe50278616b1471"}, - {file = "numpy-2.2.5-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:e8b025c351b9f0e8b5436cf28a07fa4ac0204d67b38f01433ac7f9b870fa38c6"}, - {file = "numpy-2.2.5-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:8dfa94b6a4374e7851bbb6f35e6ded2120b752b063e6acdd3157e4d2bb922eba"}, - {file = "numpy-2.2.5-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:97c8425d4e26437e65e1d189d22dff4a079b747ff9c2788057bfb8114ce1e133"}, - {file = "numpy-2.2.5-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:352d330048c055ea6db701130abc48a21bec690a8d38f8284e00fab256dc1376"}, - {file = "numpy-2.2.5-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8b4c0773b6ada798f51f0f8e30c054d32304ccc6e9c5d93d46cb26f3d385ab19"}, - {file = "numpy-2.2.5-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:55f09e00d4dccd76b179c0f18a44f041e5332fd0e022886ba1c0bbf3ea4a18d0"}, - {file = "numpy-2.2.5-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:02f226baeefa68f7d579e213d0f3493496397d8f1cff5e2b222af274c86a552a"}, - {file = "numpy-2.2.5-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c26843fd58f65da9491165072da2cccc372530681de481ef670dcc8e27cfb066"}, - {file = "numpy-2.2.5-cp313-cp313t-win32.whl", hash = "sha256:1a161c2c79ab30fe4501d5a2bbfe8b162490757cf90b7f05be8b80bc02f7bb8e"}, - {file = "numpy-2.2.5-cp313-cp313t-win_amd64.whl", hash = "sha256:d403c84991b5ad291d3809bace5e85f4bbf44a04bdc9a88ed2bb1807b3360bb8"}, - {file = "numpy-2.2.5-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:b4ea7e1cff6784e58fe281ce7e7f05036b3e1c89c6f922a6bfbc0a7e8768adbe"}, - {file = "numpy-2.2.5-pp310-pypy310_pp73-macosx_14_0_x86_64.whl", hash = "sha256:d7543263084a85fbc09c704b515395398d31d6395518446237eac219eab9e55e"}, - {file = "numpy-2.2.5-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0255732338c4fdd00996c0421884ea8a3651eea555c3a56b84892b66f696eb70"}, - {file = "numpy-2.2.5-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:d2e3bdadaba0e040d1e7ab39db73e0afe2c74ae277f5614dad53eadbecbbb169"}, - {file = "numpy-2.2.5.tar.gz", hash = "sha256:a9c0d994680cd991b1cb772e8b297340085466a6fe964bc9d4e80f5e2f43c291"}, + {file = "numpy-2.2.6-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:b412caa66f72040e6d268491a59f2c43bf03eb6c96dd8f0307829feb7fa2b6fb"}, + {file = "numpy-2.2.6-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8e41fd67c52b86603a91c1a505ebaef50b3314de0213461c7a6e99c9a3beff90"}, + {file = "numpy-2.2.6-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:37e990a01ae6ec7fe7fa1c26c55ecb672dd98b19c3d0e1d1f326fa13cb38d163"}, + {file = "numpy-2.2.6-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:5a6429d4be8ca66d889b7cf70f536a397dc45ba6faeb5f8c5427935d9592e9cf"}, + {file = "numpy-2.2.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:efd28d4e9cd7d7a8d39074a4d44c63eda73401580c5c76acda2ce969e0a38e83"}, + {file = "numpy-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fc7b73d02efb0e18c000e9ad8b83480dfcd5dfd11065997ed4c6747470ae8915"}, + {file = "numpy-2.2.6-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:74d4531beb257d2c3f4b261bfb0fc09e0f9ebb8842d82a7b4209415896adc680"}, + {file = "numpy-2.2.6-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8fc377d995680230e83241d8a96def29f204b5782f371c532579b4f20607a289"}, + {file = "numpy-2.2.6-cp310-cp310-win32.whl", hash = "sha256:b093dd74e50a8cba3e873868d9e93a85b78e0daf2e98c6797566ad8044e8363d"}, + {file = "numpy-2.2.6-cp310-cp310-win_amd64.whl", hash = "sha256:f0fd6321b839904e15c46e0d257fdd101dd7f530fe03fd6359c1ea63738703f3"}, + {file = "numpy-2.2.6-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:f9f1adb22318e121c5c69a09142811a201ef17ab257a1e66ca3025065b7f53ae"}, + {file = "numpy-2.2.6-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c820a93b0255bc360f53eca31a0e676fd1101f673dda8da93454a12e23fc5f7a"}, + {file = "numpy-2.2.6-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:3d70692235e759f260c3d837193090014aebdf026dfd167834bcba43e30c2a42"}, + {file = "numpy-2.2.6-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:481b49095335f8eed42e39e8041327c05b0f6f4780488f61286ed3c01368d491"}, + {file = "numpy-2.2.6-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b64d8d4d17135e00c8e346e0a738deb17e754230d7e0810ac5012750bbd85a5a"}, + {file = "numpy-2.2.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ba10f8411898fc418a521833e014a77d3ca01c15b0c6cdcce6a0d2897e6dbbdf"}, + {file = "numpy-2.2.6-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:bd48227a919f1bafbdda0583705e547892342c26fb127219d60a5c36882609d1"}, + {file = "numpy-2.2.6-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:9551a499bf125c1d4f9e250377c1ee2eddd02e01eac6644c080162c0c51778ab"}, + {file = "numpy-2.2.6-cp311-cp311-win32.whl", hash = "sha256:0678000bb9ac1475cd454c6b8c799206af8107e310843532b04d49649c717a47"}, + {file = "numpy-2.2.6-cp311-cp311-win_amd64.whl", hash = "sha256:e8213002e427c69c45a52bbd94163084025f533a55a59d6f9c5b820774ef3303"}, + {file = "numpy-2.2.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:41c5a21f4a04fa86436124d388f6ed60a9343a6f767fced1a8a71c3fbca038ff"}, + {file = "numpy-2.2.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:de749064336d37e340f640b05f24e9e3dd678c57318c7289d222a8a2f543e90c"}, + {file = "numpy-2.2.6-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:894b3a42502226a1cac872f840030665f33326fc3dac8e57c607905773cdcde3"}, + {file = "numpy-2.2.6-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:71594f7c51a18e728451bb50cc60a3ce4e6538822731b2933209a1f3614e9282"}, + {file = "numpy-2.2.6-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f2618db89be1b4e05f7a1a847a9c1c0abd63e63a1607d892dd54668dd92faf87"}, + {file = "numpy-2.2.6-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd83c01228a688733f1ded5201c678f0c53ecc1006ffbc404db9f7a899ac6249"}, + {file = "numpy-2.2.6-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:37c0ca431f82cd5fa716eca9506aefcabc247fb27ba69c5062a6d3ade8cf8f49"}, + {file = "numpy-2.2.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fe27749d33bb772c80dcd84ae7e8df2adc920ae8297400dabec45f0dedb3f6de"}, + {file = "numpy-2.2.6-cp312-cp312-win32.whl", hash = "sha256:4eeaae00d789f66c7a25ac5f34b71a7035bb474e679f410e5e1a94deb24cf2d4"}, + {file = "numpy-2.2.6-cp312-cp312-win_amd64.whl", hash = "sha256:c1f9540be57940698ed329904db803cf7a402f3fc200bfe599334c9bd84a40b2"}, + {file = "numpy-2.2.6-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0811bb762109d9708cca4d0b13c4f67146e3c3b7cf8d34018c722adb2d957c84"}, + {file = "numpy-2.2.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:287cc3162b6f01463ccd86be154f284d0893d2b3ed7292439ea97eafa8170e0b"}, + {file = "numpy-2.2.6-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:f1372f041402e37e5e633e586f62aa53de2eac8d98cbfb822806ce4bbefcb74d"}, + {file = "numpy-2.2.6-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:55a4d33fa519660d69614a9fad433be87e5252f4b03850642f88993f7b2ca566"}, + {file = "numpy-2.2.6-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f92729c95468a2f4f15e9bb94c432a9229d0d50de67304399627a943201baa2f"}, + {file = "numpy-2.2.6-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1bc23a79bfabc5d056d106f9befb8d50c31ced2fbc70eedb8155aec74a45798f"}, + {file = "numpy-2.2.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:e3143e4451880bed956e706a3220b4e5cf6172ef05fcc397f6f36a550b1dd868"}, + {file = "numpy-2.2.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b4f13750ce79751586ae2eb824ba7e1e8dba64784086c98cdbbcc6a42112ce0d"}, + {file = "numpy-2.2.6-cp313-cp313-win32.whl", hash = "sha256:5beb72339d9d4fa36522fc63802f469b13cdbe4fdab4a288f0c441b74272ebfd"}, + {file = "numpy-2.2.6-cp313-cp313-win_amd64.whl", hash = "sha256:b0544343a702fa80c95ad5d3d608ea3599dd54d4632df855e4c8d24eb6ecfa1c"}, + {file = "numpy-2.2.6-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:0bca768cd85ae743b2affdc762d617eddf3bcf8724435498a1e80132d04879e6"}, + {file = "numpy-2.2.6-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:fc0c5673685c508a142ca65209b4e79ed6740a4ed6b2267dbba90f34b0b3cfda"}, + {file = "numpy-2.2.6-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:5bd4fc3ac8926b3819797a7c0e2631eb889b4118a9898c84f585a54d475b7e40"}, + {file = "numpy-2.2.6-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:fee4236c876c4e8369388054d02d0e9bb84821feb1a64dd59e137e6511a551f8"}, + {file = "numpy-2.2.6-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e1dda9c7e08dc141e0247a5b8f49cf05984955246a327d4c48bda16821947b2f"}, + {file = "numpy-2.2.6-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f447e6acb680fd307f40d3da4852208af94afdfab89cf850986c3ca00562f4fa"}, + {file = "numpy-2.2.6-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:389d771b1623ec92636b0786bc4ae56abafad4a4c513d36a55dce14bd9ce8571"}, + {file = "numpy-2.2.6-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8e9ace4a37db23421249ed236fdcdd457d671e25146786dfc96835cd951aa7c1"}, + {file = "numpy-2.2.6-cp313-cp313t-win32.whl", hash = "sha256:038613e9fb8c72b0a41f025a7e4c3f0b7a1b5d768ece4796b674c8f3fe13efff"}, + {file = "numpy-2.2.6-cp313-cp313t-win_amd64.whl", hash = "sha256:6031dd6dfecc0cf9f668681a37648373bddd6421fff6c66ec1624eed0180ee06"}, + {file = "numpy-2.2.6-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:0b605b275d7bd0c640cad4e5d30fa701a8d59302e127e5f79138ad62762c3e3d"}, + {file = "numpy-2.2.6-pp310-pypy310_pp73-macosx_14_0_x86_64.whl", hash = "sha256:7befc596a7dc9da8a337f79802ee8adb30a552a94f792b9c9d18c840055907db"}, + {file = "numpy-2.2.6-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ce47521a4754c8f4593837384bd3424880629f718d87c5d44f8ed763edd63543"}, + {file = "numpy-2.2.6-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:d042d24c90c41b54fd506da306759e06e568864df8ec17ccc17e9e884634fd00"}, + {file = "numpy-2.2.6.tar.gz", hash = "sha256:e29554e2bef54a90aa5cc07da6ce955accb83f21ab5de01a62c8478897b264fd"}, +] + +[[package]] +name = "numpy" +version = "2.3.1" +description = "Fundamental package for array computing in Python" +optional = false +python-versions = ">=3.11" +groups = ["main", "eval"] +markers = "python_version >= \"3.11\"" +files = [ + {file = "numpy-2.3.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6ea9e48336a402551f52cd8f593343699003d2353daa4b72ce8d34f66b722070"}, + {file = "numpy-2.3.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:5ccb7336eaf0e77c1635b232c141846493a588ec9ea777a7c24d7166bb8533ae"}, + {file = "numpy-2.3.1-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:0bb3a4a61e1d327e035275d2a993c96fa786e4913aa089843e6a2d9dd205c66a"}, + {file = "numpy-2.3.1-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:e344eb79dab01f1e838ebb67aab09965fb271d6da6b00adda26328ac27d4a66e"}, + {file = "numpy-2.3.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:467db865b392168ceb1ef1ffa6f5a86e62468c43e0cfb4ab6da667ede10e58db"}, + {file = "numpy-2.3.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:afed2ce4a84f6b0fc6c1ce734ff368cbf5a5e24e8954a338f3bdffa0718adffb"}, + {file = "numpy-2.3.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:0025048b3c1557a20bc80d06fdeb8cc7fc193721484cca82b2cfa072fec71a93"}, + {file = "numpy-2.3.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:a5ee121b60aa509679b682819c602579e1df14a5b07fe95671c8849aad8f2115"}, + {file = "numpy-2.3.1-cp311-cp311-win32.whl", hash = "sha256:a8b740f5579ae4585831b3cf0e3b0425c667274f82a484866d2adf9570539369"}, + {file = "numpy-2.3.1-cp311-cp311-win_amd64.whl", hash = "sha256:d4580adadc53311b163444f877e0789f1c8861e2698f6b2a4ca852fda154f3ff"}, + {file = "numpy-2.3.1-cp311-cp311-win_arm64.whl", hash = "sha256:ec0bdafa906f95adc9a0c6f26a4871fa753f25caaa0e032578a30457bff0af6a"}, + {file = "numpy-2.3.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2959d8f268f3d8ee402b04a9ec4bb7604555aeacf78b360dc4ec27f1d508177d"}, + {file = "numpy-2.3.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:762e0c0c6b56bdedfef9a8e1d4538556438288c4276901ea008ae44091954e29"}, + {file = "numpy-2.3.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:867ef172a0976aaa1f1d1b63cf2090de8b636a7674607d514505fb7276ab08fc"}, + {file = "numpy-2.3.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:4e602e1b8682c2b833af89ba641ad4176053aaa50f5cacda1a27004352dde943"}, + {file = "numpy-2.3.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:8e333040d069eba1652fb08962ec5b76af7f2c7bce1df7e1418c8055cf776f25"}, + {file = "numpy-2.3.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:e7cbf5a5eafd8d230a3ce356d892512185230e4781a361229bd902ff403bc660"}, + {file = "numpy-2.3.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5f1b8f26d1086835f442286c1d9b64bb3974b0b1e41bb105358fd07d20872952"}, + {file = "numpy-2.3.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ee8340cb48c9b7a5899d1149eece41ca535513a9698098edbade2a8e7a84da77"}, + {file = "numpy-2.3.1-cp312-cp312-win32.whl", hash = "sha256:e772dda20a6002ef7061713dc1e2585bc1b534e7909b2030b5a46dae8ff077ab"}, + {file = "numpy-2.3.1-cp312-cp312-win_amd64.whl", hash = "sha256:cfecc7822543abdea6de08758091da655ea2210b8ffa1faf116b940693d3df76"}, + {file = "numpy-2.3.1-cp312-cp312-win_arm64.whl", hash = "sha256:7be91b2239af2658653c5bb6f1b8bccafaf08226a258caf78ce44710a0160d30"}, + {file = "numpy-2.3.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:25a1992b0a3fdcdaec9f552ef10d8103186f5397ab45e2d25f8ac51b1a6b97e8"}, + {file = "numpy-2.3.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7dea630156d39b02a63c18f508f85010230409db5b2927ba59c8ba4ab3e8272e"}, + {file = "numpy-2.3.1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:bada6058dd886061f10ea15f230ccf7dfff40572e99fef440a4a857c8728c9c0"}, + {file = "numpy-2.3.1-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:a894f3816eb17b29e4783e5873f92faf55b710c2519e5c351767c51f79d8526d"}, + {file = "numpy-2.3.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:18703df6c4a4fee55fd3d6e5a253d01c5d33a295409b03fda0c86b3ca2ff41a1"}, + {file = "numpy-2.3.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:5902660491bd7a48b2ec16c23ccb9124b8abfd9583c5fdfa123fe6b421e03de1"}, + {file = "numpy-2.3.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:36890eb9e9d2081137bd78d29050ba63b8dab95dff7912eadf1185e80074b2a0"}, + {file = "numpy-2.3.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a780033466159c2270531e2b8ac063704592a0bc62ec4a1b991c7c40705eb0e8"}, + {file = "numpy-2.3.1-cp313-cp313-win32.whl", hash = "sha256:39bff12c076812595c3a306f22bfe49919c5513aa1e0e70fac756a0be7c2a2b8"}, + {file = "numpy-2.3.1-cp313-cp313-win_amd64.whl", hash = "sha256:8d5ee6eec45f08ce507a6570e06f2f879b374a552087a4179ea7838edbcbfa42"}, + {file = "numpy-2.3.1-cp313-cp313-win_arm64.whl", hash = "sha256:0c4d9e0a8368db90f93bd192bfa771ace63137c3488d198ee21dfb8e7771916e"}, + {file = "numpy-2.3.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:b0b5397374f32ec0649dd98c652a1798192042e715df918c20672c62fb52d4b8"}, + {file = "numpy-2.3.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:c5bdf2015ccfcee8253fb8be695516ac4457c743473a43290fd36eba6a1777eb"}, + {file = "numpy-2.3.1-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:d70f20df7f08b90a2062c1f07737dd340adccf2068d0f1b9b3d56e2038979fee"}, + {file = "numpy-2.3.1-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:2fb86b7e58f9ac50e1e9dd1290154107e47d1eef23a0ae9145ded06ea606f992"}, + {file = "numpy-2.3.1-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:23ab05b2d241f76cb883ce8b9a93a680752fbfcbd51c50eff0b88b979e471d8c"}, + {file = "numpy-2.3.1-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:ce2ce9e5de4703a673e705183f64fd5da5bf36e7beddcb63a25ee2286e71ca48"}, + {file = "numpy-2.3.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c4913079974eeb5c16ccfd2b1f09354b8fed7e0d6f2cab933104a09a6419b1ee"}, + {file = "numpy-2.3.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:010ce9b4f00d5c036053ca684c77441f2f2c934fd23bee058b4d6f196efd8280"}, + {file = "numpy-2.3.1-cp313-cp313t-win32.whl", hash = "sha256:6269b9edfe32912584ec496d91b00b6d34282ca1d07eb10e82dfc780907d6c2e"}, + {file = "numpy-2.3.1-cp313-cp313t-win_amd64.whl", hash = "sha256:2a809637460e88a113e186e87f228d74ae2852a2e0c44de275263376f17b5bdc"}, + {file = "numpy-2.3.1-cp313-cp313t-win_arm64.whl", hash = "sha256:eccb9a159db9aed60800187bc47a6d3451553f0e1b08b068d8b277ddfbb9b244"}, + {file = "numpy-2.3.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:ad506d4b09e684394c42c966ec1527f6ebc25da7f4da4b1b056606ffe446b8a3"}, + {file = "numpy-2.3.1-pp311-pypy311_pp73-macosx_14_0_arm64.whl", hash = "sha256:ebb8603d45bc86bbd5edb0d63e52c5fd9e7945d3a503b77e486bd88dde67a19b"}, + {file = "numpy-2.3.1-pp311-pypy311_pp73-macosx_14_0_x86_64.whl", hash = "sha256:15aa4c392ac396e2ad3d0a2680c0f0dee420f9fed14eef09bdb9450ee6dcb7b7"}, + {file = "numpy-2.3.1-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:c6e0bf9d1a2f50d2b65a7cf56db37c095af17b59f6c132396f7c6d5dd76484df"}, + {file = "numpy-2.3.1-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:eabd7e8740d494ce2b4ea0ff05afa1b7b291e978c0ae075487c51e8bd93c0c68"}, + {file = "numpy-2.3.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:e610832418a2bc09d974cc9fecebfa51e9532d6190223bc5ef6a7402ebf3b5cb"}, + {file = "numpy-2.3.1.tar.gz", hash = "sha256:1ec9ae20a4226da374362cca3c62cd753faf2f951440b0e3b98e93c235441d2b"}, ] [[package]] @@ -2434,12 +2718,12 @@ description = "CUBLAS native runtime libraries" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cublas_cu12-12.6.4.1-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:08ed2686e9875d01b58e3cb379c6896df8e76c75e0d4a7f7dace3d7b6d9ef8eb"}, {file = "nvidia_cublas_cu12-12.6.4.1-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:235f728d6e2a409eddf1df58d5b0921cf80cfa9e72b9f2775ccb7b4a87984668"}, {file = "nvidia_cublas_cu12-12.6.4.1-py3-none-win_amd64.whl", hash = "sha256:9e4fa264f4d8a4eb0cdbd34beadc029f453b3bafae02401e999cf3d5a5af75f8"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "nvidia-cuda-cupti-cu12" @@ -2448,7 +2732,6 @@ description = "CUDA profiling tools runtime libs." optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:166ee35a3ff1587f2490364f90eeeb8da06cd867bd5b701bf7f9a02b78bc63fc"}, {file = "nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_aarch64.whl", hash = "sha256:358b4a1d35370353d52e12f0a7d1769fc01ff74a191689d3870b2123156184c4"}, @@ -2456,6 +2739,7 @@ files = [ {file = "nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_x86_64.whl", hash = "sha256:a3eff6cdfcc6a4c35db968a06fcadb061cbc7d6dde548609a941ff8701b98b73"}, {file = "nvidia_cuda_cupti_cu12-12.6.80-py3-none-win_amd64.whl", hash = "sha256:bbe6ae76e83ce5251b56e8c8e61a964f757175682bbad058b170b136266ab00a"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "nvidia-cuda-nvrtc-cu12" @@ -2464,12 +2748,12 @@ description = "NVRTC native runtime libraries" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cuda_nvrtc_cu12-12.6.77-py3-none-manylinux2014_aarch64.whl", hash = "sha256:5847f1d6e5b757f1d2b3991a01082a44aad6f10ab3c5c0213fa3e25bddc25a13"}, {file = "nvidia_cuda_nvrtc_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl", hash = "sha256:35b0cc6ee3a9636d5409133e79273ce1f3fd087abb0532d2d2e8fff1fe9efc53"}, {file = "nvidia_cuda_nvrtc_cu12-12.6.77-py3-none-win_amd64.whl", hash = "sha256:f7007dbd914c56bd80ea31bc43e8e149da38f68158f423ba845fc3292684e45a"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "nvidia-cuda-runtime-cu12" @@ -2478,7 +2762,6 @@ description = "CUDA Runtime native Libraries" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6116fad3e049e04791c0256a9778c16237837c08b27ed8c8401e2e45de8d60cd"}, {file = "nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_aarch64.whl", hash = "sha256:d461264ecb429c84c8879a7153499ddc7b19b5f8d84c204307491989a365588e"}, @@ -2486,6 +2769,7 @@ files = [ {file = "nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl", hash = "sha256:a84d15d5e1da416dd4774cb42edf5e954a3e60cc945698dc1d5be02321c44dc8"}, {file = "nvidia_cuda_runtime_cu12-12.6.77-py3-none-win_amd64.whl", hash = "sha256:86c58044c824bf3c173c49a2dbc7a6c8b53cb4e4dca50068be0bf64e9dab3f7f"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "nvidia-cudnn-cu12" @@ -2494,12 +2778,12 @@ description = "cuDNN runtime libraries" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cudnn_cu12-9.5.1.17-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:9fd4584468533c61873e5fda8ca41bac3a38bcb2d12350830c69b0a96a7e4def"}, {file = "nvidia_cudnn_cu12-9.5.1.17-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:30ac3869f6db17d170e0e556dd6cc5eee02647abc31ca856634d5a40f82c15b2"}, {file = "nvidia_cudnn_cu12-9.5.1.17-py3-none-win_amd64.whl", hash = "sha256:d7af0f8a4f3b4b9dbb3122f2ef553b45694ed9c384d5a75bab197b8eefb79ab8"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [package.dependencies] nvidia-cublas-cu12 = "*" @@ -2511,7 +2795,6 @@ description = "CUFFT native runtime libraries" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d16079550df460376455cba121db6564089176d9bac9e4f360493ca4741b22a6"}, {file = "nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_aarch64.whl", hash = "sha256:8510990de9f96c803a051822618d42bf6cb8f069ff3f48d93a8486efdacb48fb"}, @@ -2519,6 +2802,7 @@ files = [ {file = "nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_x86_64.whl", hash = "sha256:768160ac89f6f7b459bee747e8d175dbf53619cfe74b2a5636264163138013ca"}, {file = "nvidia_cufft_cu12-11.3.0.4-py3-none-win_amd64.whl", hash = "sha256:6048ebddfb90d09d2707efb1fd78d4e3a77cb3ae4dc60e19aab6be0ece2ae464"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [package.dependencies] nvidia-nvjitlink-cu12 = "*" @@ -2530,11 +2814,11 @@ description = "cuFile GPUDirect libraries" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cufile_cu12-1.11.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cc23469d1c7e52ce6c1d55253273d32c565dd22068647f3aa59b3c6b005bf159"}, {file = "nvidia_cufile_cu12-1.11.1.6-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:8f57a0051dcf2543f6dc2b98a98cb2719c37d3cee1baba8965d57f3bbc90d4db"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "nvidia-curand-cu12" @@ -2543,7 +2827,6 @@ description = "CURAND native runtime libraries" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_curand_cu12-10.3.7.77-py3-none-manylinux2014_aarch64.whl", hash = "sha256:6e82df077060ea28e37f48a3ec442a8f47690c7499bff392a5938614b56c98d8"}, {file = "nvidia_curand_cu12-10.3.7.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a42cd1344297f70b9e39a1e4f467a4e1c10f1da54ff7a85c12197f6c652c8bdf"}, @@ -2551,6 +2834,7 @@ files = [ {file = "nvidia_curand_cu12-10.3.7.77-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:7b2ed8e95595c3591d984ea3603dd66fe6ce6812b886d59049988a712ed06b6e"}, {file = "nvidia_curand_cu12-10.3.7.77-py3-none-win_amd64.whl", hash = "sha256:6d6d935ffba0f3d439b7cd968192ff068fafd9018dbf1b85b37261b13cfc9905"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "nvidia-cusolver-cu12" @@ -2559,7 +2843,6 @@ description = "CUDA solver native runtime libraries" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_aarch64.whl", hash = "sha256:0ce237ef60acde1efc457335a2ddadfd7610b892d94efee7b776c64bb1cac9e0"}, {file = "nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e9e49843a7707e42022babb9bcfa33c29857a93b88020c4e4434656a655b698c"}, @@ -2567,6 +2850,7 @@ files = [ {file = "nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:dbbe4fc38ec1289c7e5230e16248365e375c3673c9c8bac5796e2e20db07f56e"}, {file = "nvidia_cusolver_cu12-11.7.1.2-py3-none-win_amd64.whl", hash = "sha256:6813f9d8073f555444a8705f3ab0296d3e1cb37a16d694c5fc8b862a0d8706d7"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [package.dependencies] nvidia-cublas-cu12 = "*" @@ -2580,7 +2864,6 @@ description = "CUSPARSE native runtime libraries" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d25b62fb18751758fe3c93a4a08eff08effedfe4edf1c6bb5afd0890fe88f887"}, {file = "nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_aarch64.whl", hash = "sha256:7aa32fa5470cf754f72d1116c7cbc300b4e638d3ae5304cfa4a638a5b87161b1"}, @@ -2588,6 +2871,7 @@ files = [ {file = "nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.whl", hash = "sha256:23749a6571191a215cb74d1cdbff4a86e7b19f1200c071b3fcf844a5bea23a2f"}, {file = "nvidia_cusparse_cu12-12.5.4.2-py3-none-win_amd64.whl", hash = "sha256:4acb8c08855a26d737398cba8fb6f8f5045d93f82612b4cfd84645a2332ccf20"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [package.dependencies] nvidia-nvjitlink-cu12 = "*" @@ -2599,12 +2883,12 @@ description = "NVIDIA cuSPARSELt" optional = false python-versions = "*" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_cusparselt_cu12-0.6.3-py3-none-manylinux2014_aarch64.whl", hash = "sha256:8371549623ba601a06322af2133c4a44350575f5a3108fb75f3ef20b822ad5f1"}, {file = "nvidia_cusparselt_cu12-0.6.3-py3-none-manylinux2014_x86_64.whl", hash = "sha256:e5c8a26c36445dd2e6812f1177978a24e2d37cacce7e090f297a688d1ec44f46"}, {file = "nvidia_cusparselt_cu12-0.6.3-py3-none-win_amd64.whl", hash = "sha256:3b325bcbd9b754ba43df5a311488fca11a6b5dc3d11df4d190c000cf1a0765c7"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "nvidia-nccl-cu12" @@ -2613,11 +2897,11 @@ description = "NVIDIA Collective Communication Library (NCCL) Runtime" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_nccl_cu12-2.26.2-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5c196e95e832ad30fbbb50381eb3cbd1fadd5675e587a548563993609af19522"}, {file = "nvidia_nccl_cu12-2.26.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:694cf3879a206553cc9d7dbda76b13efaf610fdb70a50cba303de1b0d1530ac6"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "nvidia-nvjitlink-cu12" @@ -2626,12 +2910,12 @@ description = "Nvidia JIT LTO Library" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_nvjitlink_cu12-12.6.85-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:eedc36df9e88b682efe4309aa16b5b4e78c2407eac59e8c10a6a47535164369a"}, {file = "nvidia_nvjitlink_cu12-12.6.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cf4eaa7d4b6b543ffd69d6abfb11efdeb2db48270d94dfd3a452c24150829e41"}, {file = "nvidia_nvjitlink_cu12-12.6.85-py3-none-win_amd64.whl", hash = "sha256:e61120e52ed675747825cdd16febc6a0730537451d867ee58bee3853b1b13d1c"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "nvidia-nvtx-cu12" @@ -2640,7 +2924,6 @@ description = "NVIDIA Tools Extension" optional = false python-versions = ">=3" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f44f8d86bb7d5629988d61c8d3ae61dddb2015dee142740536bc7481b022fe4b"}, {file = "nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_aarch64.whl", hash = "sha256:adcaabb9d436c9761fca2b13959a2d237c5f9fd406c8e4b723c695409ff88059"}, @@ -2648,49 +2931,51 @@ files = [ {file = "nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl", hash = "sha256:6574241a3ec5fdc9334353ab8c479fe75841dbe8f4532a8fc97ce63503330ba1"}, {file = "nvidia_nvtx_cu12-12.6.77-py3-none-win_amd64.whl", hash = "sha256:2fb11a4af04a5e6c84073e6404d26588a34afd35379f0855a99797897efa75c0"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [[package]] name = "ollama" -version = "0.4.8" +version = "0.4.9" description = "The official Python client for Ollama." optional = false -python-versions = "<4.0,>=3.8" +python-versions = ">=3.8" groups = ["main"] files = [ - {file = "ollama-0.4.8-py3-none-any.whl", hash = "sha256:04312af2c5e72449aaebac4a2776f52ef010877c554103419d3f36066fe8af4c"}, - {file = "ollama-0.4.8.tar.gz", hash = "sha256:1121439d49b96fa8339842965d0616eba5deb9f8c790786cdf4c0b3df4833802"}, + {file = "ollama-0.4.9-py3-none-any.whl", hash = "sha256:18c8c85358c54d7f73d6a66cda495b0e3ba99fdb88f824ae470d740fbb211a50"}, + {file = "ollama-0.4.9.tar.gz", hash = "sha256:5266d4d29b5089a01489872b8e8f980f018bccbdd1082b3903448af1d5615ce7"}, ] [package.dependencies] -httpx = ">=0.27,<0.29" -pydantic = ">=2.9.0,<3.0.0" +httpx = ">=0.27" +pydantic = ">=2.9" [[package]] name = "onnxruntime" -version = "1.22.0" +version = "1.22.1" description = "ONNX Runtime is a runtime accelerator for Machine Learning models" -optional = false +optional = true python-versions = ">=3.10" groups = ["main"] -files = [ - {file = "onnxruntime-1.22.0-cp310-cp310-macosx_13_0_universal2.whl", hash = "sha256:85d8826cc8054e4d6bf07f779dc742a363c39094015bdad6a08b3c18cfe0ba8c"}, - {file = "onnxruntime-1.22.0-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:468c9502a12f6f49ec335c2febd22fdceecc1e4cc96dfc27e419ba237dff5aff"}, - {file = "onnxruntime-1.22.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:681fe356d853630a898ee05f01ddb95728c9a168c9460e8361d0a240c9b7cb97"}, - {file = "onnxruntime-1.22.0-cp310-cp310-win_amd64.whl", hash = "sha256:20bca6495d06925631e201f2b257cc37086752e8fe7b6c83a67c6509f4759bc9"}, - {file = "onnxruntime-1.22.0-cp311-cp311-macosx_13_0_universal2.whl", hash = "sha256:8d6725c5b9a681d8fe72f2960c191a96c256367887d076b08466f52b4e0991df"}, - {file = "onnxruntime-1.22.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fef17d665a917866d1f68f09edc98223b9a27e6cb167dec69da4c66484ad12fd"}, - {file = "onnxruntime-1.22.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b978aa63a9a22095479c38371a9b359d4c15173cbb164eaad5f2cd27d666aa65"}, - {file = "onnxruntime-1.22.0-cp311-cp311-win_amd64.whl", hash = "sha256:03d3ef7fb11adf154149d6e767e21057e0e577b947dd3f66190b212528e1db31"}, - {file = "onnxruntime-1.22.0-cp312-cp312-macosx_13_0_universal2.whl", hash = "sha256:f3c0380f53c1e72a41b3f4d6af2ccc01df2c17844072233442c3a7e74851ab97"}, - {file = "onnxruntime-1.22.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c8601128eaef79b636152aea76ae6981b7c9fc81a618f584c15d78d42b310f1c"}, - {file = "onnxruntime-1.22.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6964a975731afc19dc3418fad8d4e08c48920144ff590149429a5ebe0d15fb3c"}, - {file = "onnxruntime-1.22.0-cp312-cp312-win_amd64.whl", hash = "sha256:c0d534a43d1264d1273c2d4f00a5a588fa98d21117a3345b7104fa0bbcaadb9a"}, - {file = "onnxruntime-1.22.0-cp313-cp313-macosx_13_0_universal2.whl", hash = "sha256:fe7c051236aae16d8e2e9ffbfc1e115a0cc2450e873a9c4cb75c0cc96c1dae07"}, - {file = "onnxruntime-1.22.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6a6bbed10bc5e770c04d422893d3045b81acbbadc9fb759a2cd1ca00993da919"}, - {file = "onnxruntime-1.22.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9fe45ee3e756300fccfd8d61b91129a121d3d80e9d38e01f03ff1295badc32b8"}, - {file = "onnxruntime-1.22.0-cp313-cp313-win_amd64.whl", hash = "sha256:5a31d84ef82b4b05d794a4ce8ba37b0d9deb768fd580e36e17b39e0b4840253b"}, - {file = "onnxruntime-1.22.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0a2ac5bd9205d831541db4e508e586e764a74f14efdd3f89af7fd20e1bf4a1ed"}, - {file = "onnxruntime-1.22.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:64845709f9e8a2809e8e009bc4c8f73b788cee9c6619b7d9930344eae4c9cd36"}, +markers = "extra == \"mem-reader\" or extra == \"all\"" +files = [ + {file = "onnxruntime-1.22.1-cp310-cp310-macosx_13_0_universal2.whl", hash = "sha256:80e7f51da1f5201c1379b8d6ef6170505cd800e40da216290f5e06be01aadf95"}, + {file = "onnxruntime-1.22.1-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b89ddfdbbdaf7e3a59515dee657f6515601d55cb21a0f0f48c81aefc54ff1b73"}, + {file = "onnxruntime-1.22.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bddc75868bcf6f9ed76858a632f65f7b1846bdcefc6d637b1e359c2c68609964"}, + {file = "onnxruntime-1.22.1-cp310-cp310-win_amd64.whl", hash = "sha256:01e2f21b2793eb0c8642d2be3cee34cc7d96b85f45f6615e4e220424158877ce"}, + {file = "onnxruntime-1.22.1-cp311-cp311-macosx_13_0_universal2.whl", hash = "sha256:f4581bccb786da68725d8eac7c63a8f31a89116b8761ff8b4989dc58b61d49a0"}, + {file = "onnxruntime-1.22.1-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7ae7526cf10f93454beb0f751e78e5cb7619e3b92f9fc3bd51aa6f3b7a8977e5"}, + {file = "onnxruntime-1.22.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f6effa1299ac549a05c784d50292e3378dbbf010346ded67400193b09ddc2f04"}, + {file = "onnxruntime-1.22.1-cp311-cp311-win_amd64.whl", hash = "sha256:f28a42bb322b4ca6d255531bb334a2b3e21f172e37c1741bd5e66bc4b7b61f03"}, + {file = "onnxruntime-1.22.1-cp312-cp312-macosx_13_0_universal2.whl", hash = "sha256:a938d11c0dc811badf78e435daa3899d9af38abee950d87f3ab7430eb5b3cf5a"}, + {file = "onnxruntime-1.22.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:984cea2a02fcc5dfea44ade9aca9fe0f7a8a2cd6f77c258fc4388238618f3928"}, + {file = "onnxruntime-1.22.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2d39a530aff1ec8d02e365f35e503193991417788641b184f5b1e8c9a6d5ce8d"}, + {file = "onnxruntime-1.22.1-cp312-cp312-win_amd64.whl", hash = "sha256:6a64291d57ea966a245f749eb970f4fa05a64d26672e05a83fdb5db6b7d62f87"}, + {file = "onnxruntime-1.22.1-cp313-cp313-macosx_13_0_universal2.whl", hash = "sha256:d29c7d87b6cbed8fecfd09dca471832384d12a69e1ab873e5effbb94adc3e966"}, + {file = "onnxruntime-1.22.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:460487d83b7056ba98f1f7bac80287224c31d8149b15712b0d6f5078fcc33d0f"}, + {file = "onnxruntime-1.22.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b0c37070268ba4e02a1a9d28560cd00cd1e94f0d4f275cbef283854f861a65fa"}, + {file = "onnxruntime-1.22.1-cp313-cp313-win_amd64.whl", hash = "sha256:70980d729145a36a05f74b573435531f55ef9503bcda81fc6c3d6b9306199982"}, + {file = "onnxruntime-1.22.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:33a7980bbc4b7f446bac26c3785652fe8730ed02617d765399e89ac7d44e0f7d"}, + {file = "onnxruntime-1.22.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6e7e823624b015ea879d976cbef8bfaed2f7e2cc233d7506860a76dd37f8f381"}, ] [package.dependencies] @@ -2703,14 +2988,14 @@ sympy = "*" [[package]] name = "openai" -version = "1.78.0" +version = "1.97.0" description = "The official Python library for the openai API" optional = false python-versions = ">=3.8" groups = ["main", "eval"] files = [ - {file = "openai-1.78.0-py3-none-any.whl", hash = "sha256:1ade6a48cd323ad8a7715e7e1669bb97a17e1a5b8a916644261aaef4bf284778"}, - {file = "openai-1.78.0.tar.gz", hash = "sha256:254aef4980688468e96cbddb1f348ed01d274d02c64c6c69b0334bf001fb62b3"}, + {file = "openai-1.97.0-py3-none-any.whl", hash = "sha256:a1c24d96f4609f3f7f51c9e1c2606d97cc6e334833438659cfd687e9c972c610"}, + {file = "openai-1.97.0.tar.gz", hash = "sha256:0be349569ccaa4fb54f97bb808423fd29ccaeb1246ee1be762e0c81a47bae0aa"}, ] [package.dependencies] @@ -2724,17 +3009,34 @@ tqdm = ">4" typing-extensions = ">=4.11,<5" [package.extras] +aiohttp = ["aiohttp", "httpx-aiohttp (>=0.1.8)"] datalib = ["numpy (>=1)", "pandas (>=1.2.3)", "pandas-stubs (>=1.1.0.11)"] realtime = ["websockets (>=13,<16)"] voice-helpers = ["numpy (>=2.0.2)", "sounddevice (>=0.5.1)"] +[[package]] +name = "openapi-pydantic" +version = "0.5.1" +description = "Pydantic OpenAPI schema implementation" +optional = false +python-versions = "<4.0,>=3.8" +groups = ["main"] +files = [ + {file = "openapi_pydantic-0.5.1-py3-none-any.whl", hash = "sha256:a3a09ef4586f5bd760a8df7f43028b60cafb6d9f61de2acba9574766255ab146"}, + {file = "openapi_pydantic-0.5.1.tar.gz", hash = "sha256:ff6835af6bde7a459fb93eb93bb92b8749b754fc6e51b2f1590a19dc3005ee0d"}, +] + +[package.dependencies] +pydantic = ">=1.8" + [[package]] name = "openpyxl" version = "3.1.5" description = "A Python library to read/write Excel 2010 xlsx/xlsm files" -optional = false +optional = true python-versions = ">=3.8" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "openpyxl-3.1.5-py2.py3-none-any.whl", hash = "sha256:5282c12b107bffeef825f4617dc029afaf41d0ea60823bbb665ef3079dc79de2"}, {file = "openpyxl-3.1.5.tar.gz", hash = "sha256:cf0e3cf56142039133628b5acffe8ef0c12bc902d2aadd3e0fe5878dc08d1050"}, @@ -2745,84 +3047,84 @@ et-xmlfile = "*" [[package]] name = "orjson" -version = "3.10.18" +version = "3.11.0" description = "Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy" optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "orjson-3.10.18-cp310-cp310-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:a45e5d68066b408e4bc383b6e4ef05e717c65219a9e1390abc6155a520cac402"}, - {file = "orjson-3.10.18-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:be3b9b143e8b9db05368b13b04c84d37544ec85bb97237b3a923f076265ec89c"}, - {file = "orjson-3.10.18-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:9b0aa09745e2c9b3bf779b096fa71d1cc2d801a604ef6dd79c8b1bfef52b2f92"}, - {file = "orjson-3.10.18-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:53a245c104d2792e65c8d225158f2b8262749ffe64bc7755b00024757d957a13"}, - {file = "orjson-3.10.18-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f9495ab2611b7f8a0a8a505bcb0f0cbdb5469caafe17b0e404c3c746f9900469"}, - {file = "orjson-3.10.18-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:73be1cbcebadeabdbc468f82b087df435843c809cd079a565fb16f0f3b23238f"}, - {file = "orjson-3.10.18-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fe8936ee2679e38903df158037a2f1c108129dee218975122e37847fb1d4ac68"}, - {file = "orjson-3.10.18-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:7115fcbc8525c74e4c2b608129bef740198e9a120ae46184dac7683191042056"}, - {file = "orjson-3.10.18-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:771474ad34c66bc4d1c01f645f150048030694ea5b2709b87d3bda273ffe505d"}, - {file = "orjson-3.10.18-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:7c14047dbbea52886dd87169f21939af5d55143dad22d10db6a7514f058156a8"}, - {file = "orjson-3.10.18-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:641481b73baec8db14fdf58f8967e52dc8bda1f2aba3aa5f5c1b07ed6df50b7f"}, - {file = "orjson-3.10.18-cp310-cp310-win32.whl", hash = "sha256:607eb3ae0909d47280c1fc657c4284c34b785bae371d007595633f4b1a2bbe06"}, - {file = "orjson-3.10.18-cp310-cp310-win_amd64.whl", hash = "sha256:8770432524ce0eca50b7efc2a9a5f486ee0113a5fbb4231526d414e6254eba92"}, - {file = "orjson-3.10.18-cp311-cp311-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:e0a183ac3b8e40471e8d843105da6fbe7c070faab023be3b08188ee3f85719b8"}, - {file = "orjson-3.10.18-cp311-cp311-macosx_15_0_arm64.whl", hash = "sha256:5ef7c164d9174362f85238d0cd4afdeeb89d9e523e4651add6a5d458d6f7d42d"}, - {file = "orjson-3.10.18-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:afd14c5d99cdc7bf93f22b12ec3b294931518aa019e2a147e8aa2f31fd3240f7"}, - {file = "orjson-3.10.18-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7b672502323b6cd133c4af6b79e3bea36bad2d16bca6c1f645903fce83909a7a"}, - {file = "orjson-3.10.18-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:51f8c63be6e070ec894c629186b1c0fe798662b8687f3d9fdfa5e401c6bd7679"}, - {file = "orjson-3.10.18-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3f9478ade5313d724e0495d167083c6f3be0dd2f1c9c8a38db9a9e912cdaf947"}, - {file = "orjson-3.10.18-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:187aefa562300a9d382b4b4eb9694806e5848b0cedf52037bb5c228c61bb66d4"}, - {file = "orjson-3.10.18-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9da552683bc9da222379c7a01779bddd0ad39dd699dd6300abaf43eadee38334"}, - {file = "orjson-3.10.18-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:e450885f7b47a0231979d9c49b567ed1c4e9f69240804621be87c40bc9d3cf17"}, - {file = "orjson-3.10.18-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:5e3c9cc2ba324187cd06287ca24f65528f16dfc80add48dc99fa6c836bb3137e"}, - {file = "orjson-3.10.18-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:50ce016233ac4bfd843ac5471e232b865271d7d9d44cf9d33773bcd883ce442b"}, - {file = "orjson-3.10.18-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:b3ceff74a8f7ffde0b2785ca749fc4e80e4315c0fd887561144059fb1c138aa7"}, - {file = "orjson-3.10.18-cp311-cp311-win32.whl", hash = "sha256:fdba703c722bd868c04702cac4cb8c6b8ff137af2623bc0ddb3b3e6a2c8996c1"}, - {file = "orjson-3.10.18-cp311-cp311-win_amd64.whl", hash = "sha256:c28082933c71ff4bc6ccc82a454a2bffcef6e1d7379756ca567c772e4fb3278a"}, - {file = "orjson-3.10.18-cp311-cp311-win_arm64.whl", hash = "sha256:a6c7c391beaedd3fa63206e5c2b7b554196f14debf1ec9deb54b5d279b1b46f5"}, - {file = "orjson-3.10.18-cp312-cp312-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:50c15557afb7f6d63bc6d6348e0337a880a04eaa9cd7c9d569bcb4e760a24753"}, - {file = "orjson-3.10.18-cp312-cp312-macosx_15_0_arm64.whl", hash = "sha256:356b076f1662c9813d5fa56db7d63ccceef4c271b1fb3dd522aca291375fcf17"}, - {file = "orjson-3.10.18-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:559eb40a70a7494cd5beab2d73657262a74a2c59aff2068fdba8f0424ec5b39d"}, - {file = "orjson-3.10.18-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f3c29eb9a81e2fbc6fd7ddcfba3e101ba92eaff455b8d602bf7511088bbc0eae"}, - {file = "orjson-3.10.18-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6612787e5b0756a171c7d81ba245ef63a3533a637c335aa7fcb8e665f4a0966f"}, - {file = "orjson-3.10.18-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7ac6bd7be0dcab5b702c9d43d25e70eb456dfd2e119d512447468f6405b4a69c"}, - {file = "orjson-3.10.18-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9f72f100cee8dde70100406d5c1abba515a7df926d4ed81e20a9730c062fe9ad"}, - {file = "orjson-3.10.18-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9dca85398d6d093dd41dc0983cbf54ab8e6afd1c547b6b8a311643917fbf4e0c"}, - {file = "orjson-3.10.18-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:22748de2a07fcc8781a70edb887abf801bb6142e6236123ff93d12d92db3d406"}, - {file = "orjson-3.10.18-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:3a83c9954a4107b9acd10291b7f12a6b29e35e8d43a414799906ea10e75438e6"}, - {file = "orjson-3.10.18-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:303565c67a6c7b1f194c94632a4a39918e067bd6176a48bec697393865ce4f06"}, - {file = "orjson-3.10.18-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:86314fdb5053a2f5a5d881f03fca0219bfdf832912aa88d18676a5175c6916b5"}, - {file = "orjson-3.10.18-cp312-cp312-win32.whl", hash = "sha256:187ec33bbec58c76dbd4066340067d9ece6e10067bb0cc074a21ae3300caa84e"}, - {file = "orjson-3.10.18-cp312-cp312-win_amd64.whl", hash = "sha256:f9f94cf6d3f9cd720d641f8399e390e7411487e493962213390d1ae45c7814fc"}, - {file = "orjson-3.10.18-cp312-cp312-win_arm64.whl", hash = "sha256:3d600be83fe4514944500fa8c2a0a77099025ec6482e8087d7659e891f23058a"}, - {file = "orjson-3.10.18-cp313-cp313-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:69c34b9441b863175cc6a01f2935de994025e773f814412030f269da4f7be147"}, - {file = "orjson-3.10.18-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:1ebeda919725f9dbdb269f59bc94f861afbe2a27dce5608cdba2d92772364d1c"}, - {file = "orjson-3.10.18-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5adf5f4eed520a4959d29ea80192fa626ab9a20b2ea13f8f6dc58644f6927103"}, - {file = "orjson-3.10.18-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7592bb48a214e18cd670974f289520f12b7aed1fa0b2e2616b8ed9e069e08595"}, - {file = "orjson-3.10.18-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f872bef9f042734110642b7a11937440797ace8c87527de25e0c53558b579ccc"}, - {file = "orjson-3.10.18-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0315317601149c244cb3ecef246ef5861a64824ccbcb8018d32c66a60a84ffbc"}, - {file = "orjson-3.10.18-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e0da26957e77e9e55a6c2ce2e7182a36a6f6b180ab7189315cb0995ec362e049"}, - {file = "orjson-3.10.18-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bb70d489bc79b7519e5803e2cc4c72343c9dc1154258adf2f8925d0b60da7c58"}, - {file = "orjson-3.10.18-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:e9e86a6af31b92299b00736c89caf63816f70a4001e750bda179e15564d7a034"}, - {file = "orjson-3.10.18-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:c382a5c0b5931a5fc5405053d36c1ce3fd561694738626c77ae0b1dfc0242ca1"}, - {file = "orjson-3.10.18-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:8e4b2ae732431127171b875cb2668f883e1234711d3c147ffd69fe5be51a8012"}, - {file = "orjson-3.10.18-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:2d808e34ddb24fc29a4d4041dcfafbae13e129c93509b847b14432717d94b44f"}, - {file = "orjson-3.10.18-cp313-cp313-win32.whl", hash = "sha256:ad8eacbb5d904d5591f27dee4031e2c1db43d559edb8f91778efd642d70e6bea"}, - {file = "orjson-3.10.18-cp313-cp313-win_amd64.whl", hash = "sha256:aed411bcb68bf62e85588f2a7e03a6082cc42e5a2796e06e72a962d7c6310b52"}, - {file = "orjson-3.10.18-cp313-cp313-win_arm64.whl", hash = "sha256:f54c1385a0e6aba2f15a40d703b858bedad36ded0491e55d35d905b2c34a4cc3"}, - {file = "orjson-3.10.18-cp39-cp39-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:c95fae14225edfd699454e84f61c3dd938df6629a00c6ce15e704f57b58433bb"}, - {file = "orjson-3.10.18-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5232d85f177f98e0cefabb48b5e7f60cff6f3f0365f9c60631fecd73849b2a82"}, - {file = "orjson-3.10.18-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:2783e121cafedf0d85c148c248a20470018b4ffd34494a68e125e7d5857655d1"}, - {file = "orjson-3.10.18-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e54ee3722caf3db09c91f442441e78f916046aa58d16b93af8a91500b7bbf273"}, - {file = "orjson-3.10.18-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2daf7e5379b61380808c24f6fc182b7719301739e4271c3ec88f2984a2d61f89"}, - {file = "orjson-3.10.18-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7f39b371af3add20b25338f4b29a8d6e79a8c7ed0e9dd49e008228a065d07781"}, - {file = "orjson-3.10.18-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2b819ed34c01d88c6bec290e6842966f8e9ff84b7694632e88341363440d4cc0"}, - {file = "orjson-3.10.18-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:2f6c57debaef0b1aa13092822cbd3698a1fb0209a9ea013a969f4efa36bdea57"}, - {file = "orjson-3.10.18-cp39-cp39-musllinux_1_2_armv7l.whl", hash = "sha256:755b6d61ffdb1ffa1e768330190132e21343757c9aa2308c67257cc81a1a6f5a"}, - {file = "orjson-3.10.18-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:ce8d0a875a85b4c8579eab5ac535fb4b2a50937267482be402627ca7e7570ee3"}, - {file = "orjson-3.10.18-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:57b5d0673cbd26781bebc2bf86f99dd19bd5a9cb55f71cc4f66419f6b50f3d77"}, - {file = "orjson-3.10.18-cp39-cp39-win32.whl", hash = "sha256:951775d8b49d1d16ca8818b1f20c4965cae9157e7b562a2ae34d3967b8f21c8e"}, - {file = "orjson-3.10.18-cp39-cp39-win_amd64.whl", hash = "sha256:fdd9d68f83f0bc4406610b1ac68bdcded8c5ee58605cc69e643a06f4d075f429"}, - {file = "orjson-3.10.18.tar.gz", hash = "sha256:e8da3947d92123eda795b68228cafe2724815621fe35e8e320a9e9593a4bcd53"}, + {file = "orjson-3.11.0-cp310-cp310-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:b8913baba9751f7400f8fa4ec18a8b618ff01177490842e39e47b66c1b04bc79"}, + {file = "orjson-3.11.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9d4d86910554de5c9c87bc560b3bdd315cc3988adbdc2acf5dda3797079407ed"}, + {file = "orjson-3.11.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:84ae3d329360cf18fb61b67c505c00dedb61b0ee23abfd50f377a58e7d7bed06"}, + {file = "orjson-3.11.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:47a54e660414baacd71ebf41a69bb17ea25abb3c5b69ce9e13e43be7ac20e342"}, + {file = "orjson-3.11.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2560b740604751854be146169c1de7e7ee1e6120b00c1788ec3f3a012c6a243f"}, + {file = "orjson-3.11.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:dd7f9cd995da9e46fbac0a371f0ff6e89a21d8ecb7a8a113c0acb147b0a32f73"}, + {file = "orjson-3.11.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7cf728cb3a013bdf9f4132575404bf885aa773d8bb4205656575e1890fc91990"}, + {file = "orjson-3.11.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:c27de273320294121200440cd5002b6aeb922d3cb9dab3357087c69f04ca6934"}, + {file = "orjson-3.11.0-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:4430ec6ff1a1f4595dd7e0fad991bdb2fed65401ed294984c490ffa025926325"}, + {file = "orjson-3.11.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:325be41a8d7c227d460a9795a181511ba0e731cf3fee088c63eb47e706ea7559"}, + {file = "orjson-3.11.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:d9760217b84d1aee393b4436fbe9c639e963ec7bc0f2c074581ce5fb3777e466"}, + {file = "orjson-3.11.0-cp310-cp310-win32.whl", hash = "sha256:fe36e5012f886ff91c68b87a499c227fa220e9668cea96335219874c8be5fab5"}, + {file = "orjson-3.11.0-cp310-cp310-win_amd64.whl", hash = "sha256:ebeecd5d5511b3ca9dc4e7db0ab95266afd41baf424cc2fad8c2d3a3cdae650a"}, + {file = "orjson-3.11.0-cp311-cp311-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:1785df7ada75c18411ff7e20ac822af904a40161ea9dfe8c55b3f6b66939add6"}, + {file = "orjson-3.11.0-cp311-cp311-macosx_15_0_arm64.whl", hash = "sha256:a57899bebbcea146616a2426d20b51b3562b4bc9f8039a3bd14fae361c23053d"}, + {file = "orjson-3.11.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9b6fbc2fc825aff1456dd358c11a0ad7912a4cb4537d3db92e5334af7463a967"}, + {file = "orjson-3.11.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4305a638f4cf9bed3746ca3b7c242f14e05177d5baec2527026e0f9ee6c24fb7"}, + {file = "orjson-3.11.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1235fe7bbc37164f69302199d46f29cfb874018738714dccc5a5a44042c79c77"}, + {file = "orjson-3.11.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a640e3954e7b4fcb160097551e54cafbde9966be3991932155b71071077881aa"}, + {file = "orjson-3.11.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:6d750b97d22d5566955e50b02c622f3a1d32744d7a578c878b29a873190ccb7a"}, + {file = "orjson-3.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4bfcfe498484161e011f8190a400591c52b026de96b3b3cbd3f21e8999b9dc0e"}, + {file = "orjson-3.11.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:feaed3ed43a1d2df75c039798eb5ec92c350c7d86be53369bafc4f3700ce7df2"}, + {file = "orjson-3.11.0-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:aa1120607ec8fc98acf8c54aac6fb0b7b003ba883401fa2d261833111e2fa071"}, + {file = "orjson-3.11.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:c4b48d9775b0cf1f0aca734f4c6b272cbfacfac38e6a455e6520662f9434afb7"}, + {file = "orjson-3.11.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f018ed1986d79434ac712ff19f951cd00b4dfcb767444410fbb834ebec160abf"}, + {file = "orjson-3.11.0-cp311-cp311-win32.whl", hash = "sha256:08e191f8a55ac2c00be48e98a5d10dca004cbe8abe73392c55951bfda60fc123"}, + {file = "orjson-3.11.0-cp311-cp311-win_amd64.whl", hash = "sha256:b5a4214ea59c8a3b56f8d484b28114af74e9fba0956f9be5c3ce388ae143bf1f"}, + {file = "orjson-3.11.0-cp311-cp311-win_arm64.whl", hash = "sha256:57e8e7198a679ab21241ab3f355a7990c7447559e35940595e628c107ef23736"}, + {file = "orjson-3.11.0-cp312-cp312-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:b4089f940c638bb1947d54e46c1cd58f4259072fcc97bc833ea9c78903150ac9"}, + {file = "orjson-3.11.0-cp312-cp312-macosx_15_0_arm64.whl", hash = "sha256:8335a0ba1c26359fb5c82d643b4c1abbee2bc62875e0f2b5bde6c8e9e25eb68c"}, + {file = "orjson-3.11.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:63c1c9772dafc811d16d6a7efa3369a739da15d1720d6e58ebe7562f54d6f4a2"}, + {file = "orjson-3.11.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:9457ccbd8b241fb4ba516417a4c5b95ba0059df4ac801309bcb4ec3870f45ad9"}, + {file = "orjson-3.11.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0846e13abe79daece94a00b92574f294acad1d362be766c04245b9b4dd0e47e1"}, + {file = "orjson-3.11.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5587c85ae02f608a3f377b6af9eb04829606f518257cbffa8f5081c1aacf2e2f"}, + {file = "orjson-3.11.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c7a1964a71c1567b4570c932a0084ac24ad52c8cf6253d1881400936565ed438"}, + {file = "orjson-3.11.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b5a8243e73690cc6e9151c9e1dd046a8f21778d775f7d478fa1eb4daa4897c61"}, + {file = "orjson-3.11.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:51646f6d995df37b6e1b628f092f41c0feccf1d47e3452c6e95e2474b547d842"}, + {file = "orjson-3.11.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:2fb8ca8f0b4e31b8aaec674c7540649b64ef02809410506a44dc68d31bd5647b"}, + {file = "orjson-3.11.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:64a6a3e94a44856c3f6557e6aa56a6686544fed9816ae0afa8df9077f5759791"}, + {file = "orjson-3.11.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:d69f95d484938d8fab5963e09131bcf9fbbb81fa4ec132e316eb2fb9adb8ce78"}, + {file = "orjson-3.11.0-cp312-cp312-win32.whl", hash = "sha256:8514f9f9c667ce7d7ef709ab1a73e7fcab78c297270e90b1963df7126d2b0e23"}, + {file = "orjson-3.11.0-cp312-cp312-win_amd64.whl", hash = "sha256:41b38a894520b8cb5344a35ffafdf6ae8042f56d16771b2c5eb107798cee85ee"}, + {file = "orjson-3.11.0-cp312-cp312-win_arm64.whl", hash = "sha256:5579acd235dd134467340b2f8a670c1c36023b5a69c6a3174c4792af7502bd92"}, + {file = "orjson-3.11.0-cp313-cp313-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:4a8ba9698655e16746fdf5266939427da0f9553305152aeb1a1cc14974a19cfb"}, + {file = "orjson-3.11.0-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:67133847f9a35a5ef5acfa3325d4a2f7fe05c11f1505c4117bb086fc06f2a58f"}, + {file = "orjson-3.11.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5f797d57814975b78f5f5423acb003db6f9be5186b72d48bd97a1000e89d331d"}, + {file = "orjson-3.11.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:28acd19822987c5163b9e03a6e60853a52acfee384af2b394d11cb413b889246"}, + {file = "orjson-3.11.0-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e8d38d9e1e2cf9729658e35956cf01e13e89148beb4cb9e794c9c10c5cb252f8"}, + {file = "orjson-3.11.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:05f094edd2b782650b0761fd78858d9254de1c1286f5af43145b3d08cdacfd51"}, + {file = "orjson-3.11.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:6d09176a4a9e04a5394a4a0edd758f645d53d903b306d02f2691b97d5c736a9e"}, + {file = "orjson-3.11.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2a585042104e90a61eda2564d11317b6a304eb4e71cd33e839f5af6be56c34d3"}, + {file = "orjson-3.11.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d2218629dbfdeeb5c9e0573d59f809d42f9d49ae6464d2f479e667aee14c3ef4"}, + {file = "orjson-3.11.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:613e54a2b10b51b656305c11235a9c4a5c5491ef5c283f86483d4e9e123ed5e4"}, + {file = "orjson-3.11.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:9dac7fbf3b8b05965986c5cfae051eb9a30fced7f15f1d13a5adc608436eb486"}, + {file = "orjson-3.11.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:93b64b254414e2be55ac5257124b5602c5f0b4d06b80bd27d1165efe8f36e836"}, + {file = "orjson-3.11.0-cp313-cp313-win32.whl", hash = "sha256:359cbe11bc940c64cb3848cf22000d2aef36aff7bfd09ca2c0b9cb309c387132"}, + {file = "orjson-3.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:0759b36428067dc777b202dd286fbdd33d7f261c6455c4238ea4e8474358b1e6"}, + {file = "orjson-3.11.0-cp313-cp313-win_arm64.whl", hash = "sha256:51cdca2f36e923126d0734efaf72ddbb5d6da01dbd20eab898bdc50de80d7b5a"}, + {file = "orjson-3.11.0-cp39-cp39-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:d79c180cfb3ae68f13245d0ff551dca03d96258aa560830bf8a223bd68d8272c"}, + {file = "orjson-3.11.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:105bca887532dc71ce4b05a5de95dea447a310409d7a8cf0cb1c4a120469e9ad"}, + {file = "orjson-3.11.0-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:acf5a63ae9cdb88274126af85913ceae554d8fd71122effa24a53227abbeee16"}, + {file = "orjson-3.11.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:894635df36c0be32f1c8c8607e853b8865edb58e7618e57892e85d06418723eb"}, + {file = "orjson-3.11.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:02dd4f0a1a2be943a104ce5f3ec092631ee3e9f0b4bb9eeee3400430bd94ddef"}, + {file = "orjson-3.11.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:720b4bb5e1b971960a62c2fa254c2d2a14e7eb791e350d05df8583025aa59d15"}, + {file = "orjson-3.11.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8bf058105a8aed144e0d1cfe7ac4174748c3fc7203f225abaeac7f4121abccb0"}, + {file = "orjson-3.11.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:a2788f741e5a0e885e5eaf1d91d0c9106e03cb9575b0c55ba36fd3d48b0b1e9b"}, + {file = "orjson-3.11.0-cp39-cp39-musllinux_1_2_armv7l.whl", hash = "sha256:c60c99fe1e15894367b0340b2ff16c7c69f9c3f3a54aa3961a58c102b292ad94"}, + {file = "orjson-3.11.0-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:99d17aab984f4d029b8f3c307e6be3c63d9ee5ef55e30d761caf05e883009949"}, + {file = "orjson-3.11.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:e98f02e23611763c9e5dfcb83bd33219231091589f0d1691e721aea9c52bf329"}, + {file = "orjson-3.11.0-cp39-cp39-win32.whl", hash = "sha256:923301f33ea866b18f8836cf41d9c6d33e3b5cab8577d20fed34ec29f0e13a0d"}, + {file = "orjson-3.11.0-cp39-cp39-win_amd64.whl", hash = "sha256:475491bb78af2a0170f49e90013f1a0f1286527f3617491f8940d7e5da862da7"}, + {file = "orjson-3.11.0.tar.gz", hash = "sha256:2e4c129da624f291bcc607016a99e7f04a353f6874f3bd8d9b47b88597d5f700"}, ] [[package]] @@ -2878,67 +3180,68 @@ files = [ [[package]] name = "packaging" -version = "24.2" +version = "25.0" description = "Core utilities for Python packages" optional = false python-versions = ">=3.8" groups = ["main", "eval", "test"] files = [ - {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"}, - {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"}, + {file = "packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484"}, + {file = "packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f"}, ] [[package]] name = "pandas" -version = "2.2.3" +version = "2.3.1" description = "Powerful data structures for data analysis, time series, and statistics" optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "pandas-2.2.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:1948ddde24197a0f7add2bdc4ca83bf2b1ef84a1bc8ccffd95eda17fd836ecb5"}, - {file = "pandas-2.2.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:381175499d3802cde0eabbaf6324cce0c4f5d52ca6f8c377c29ad442f50f6348"}, - {file = "pandas-2.2.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d9c45366def9a3dd85a6454c0e7908f2b3b8e9c138f5dc38fed7ce720d8453ed"}, - {file = "pandas-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:86976a1c5b25ae3f8ccae3a5306e443569ee3c3faf444dfd0f41cda24667ad57"}, - {file = "pandas-2.2.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:b8661b0238a69d7aafe156b7fa86c44b881387509653fdf857bebc5e4008ad42"}, - {file = "pandas-2.2.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:37e0aced3e8f539eccf2e099f65cdb9c8aa85109b0be6e93e2baff94264bdc6f"}, - {file = "pandas-2.2.3-cp310-cp310-win_amd64.whl", hash = "sha256:56534ce0746a58afaf7942ba4863e0ef81c9c50d3f0ae93e9497d6a41a057645"}, - {file = "pandas-2.2.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:66108071e1b935240e74525006034333f98bcdb87ea116de573a6a0dccb6c039"}, - {file = "pandas-2.2.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:7c2875855b0ff77b2a64a0365e24455d9990730d6431b9e0ee18ad8acee13dbd"}, - {file = "pandas-2.2.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cd8d0c3be0515c12fed0bdbae072551c8b54b7192c7b1fda0ba56059a0179698"}, - {file = "pandas-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c124333816c3a9b03fbeef3a9f230ba9a737e9e5bb4060aa2107a86cc0a497fc"}, - {file = "pandas-2.2.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:63cc132e40a2e084cf01adf0775b15ac515ba905d7dcca47e9a251819c575ef3"}, - {file = "pandas-2.2.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:29401dbfa9ad77319367d36940cd8a0b3a11aba16063e39632d98b0e931ddf32"}, - {file = "pandas-2.2.3-cp311-cp311-win_amd64.whl", hash = "sha256:3fc6873a41186404dad67245896a6e440baacc92f5b716ccd1bc9ed2995ab2c5"}, - {file = "pandas-2.2.3-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:b1d432e8d08679a40e2a6d8b2f9770a5c21793a6f9f47fdd52c5ce1948a5a8a9"}, - {file = "pandas-2.2.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a5a1595fe639f5988ba6a8e5bc9649af3baf26df3998a0abe56c02609392e0a4"}, - {file = "pandas-2.2.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5de54125a92bb4d1c051c0659e6fcb75256bf799a732a87184e5ea503965bce3"}, - {file = "pandas-2.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fffb8ae78d8af97f849404f21411c95062db1496aeb3e56f146f0355c9989319"}, - {file = "pandas-2.2.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6dfcb5ee8d4d50c06a51c2fffa6cff6272098ad6540aed1a76d15fb9318194d8"}, - {file = "pandas-2.2.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:062309c1b9ea12a50e8ce661145c6aab431b1e99530d3cd60640e255778bd43a"}, - {file = "pandas-2.2.3-cp312-cp312-win_amd64.whl", hash = "sha256:59ef3764d0fe818125a5097d2ae867ca3fa64df032331b7e0917cf5d7bf66b13"}, - {file = "pandas-2.2.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:f00d1345d84d8c86a63e476bb4955e46458b304b9575dcf71102b5c705320015"}, - {file = "pandas-2.2.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3508d914817e153ad359d7e069d752cdd736a247c322d932eb89e6bc84217f28"}, - {file = "pandas-2.2.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:22a9d949bfc9a502d320aa04e5d02feab689d61da4e7764b62c30b991c42c5f0"}, - {file = "pandas-2.2.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f3a255b2c19987fbbe62a9dfd6cff7ff2aa9ccab3fc75218fd4b7530f01efa24"}, - {file = "pandas-2.2.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:800250ecdadb6d9c78eae4990da62743b857b470883fa27f652db8bdde7f6659"}, - {file = "pandas-2.2.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6374c452ff3ec675a8f46fd9ab25c4ad0ba590b71cf0656f8b6daa5202bca3fb"}, - {file = "pandas-2.2.3-cp313-cp313-win_amd64.whl", hash = "sha256:61c5ad4043f791b61dd4752191d9f07f0ae412515d59ba8f005832a532f8736d"}, - {file = "pandas-2.2.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:3b71f27954685ee685317063bf13c7709a7ba74fc996b84fc6821c59b0f06468"}, - {file = "pandas-2.2.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:38cf8125c40dae9d5acc10fa66af8ea6fdf760b2714ee482ca691fc66e6fcb18"}, - {file = "pandas-2.2.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ba96630bc17c875161df3818780af30e43be9b166ce51c9a18c1feae342906c2"}, - {file = "pandas-2.2.3-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1db71525a1538b30142094edb9adc10be3f3e176748cd7acc2240c2f2e5aa3a4"}, - {file = "pandas-2.2.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:15c0e1e02e93116177d29ff83e8b1619c93ddc9c49083f237d4312337a61165d"}, - {file = "pandas-2.2.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:ad5b65698ab28ed8d7f18790a0dc58005c7629f227be9ecc1072aa74c0c1d43a"}, - {file = "pandas-2.2.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:bc6b93f9b966093cb0fd62ff1a7e4c09e6d546ad7c1de191767baffc57628f39"}, - {file = "pandas-2.2.3-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:5dbca4c1acd72e8eeef4753eeca07de9b1db4f398669d5994086f788a5d7cc30"}, - {file = "pandas-2.2.3-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8cd6d7cc958a3910f934ea8dbdf17b2364827bb4dafc38ce6eef6bb3d65ff09c"}, - {file = "pandas-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:99df71520d25fade9db7c1076ac94eb994f4d2673ef2aa2e86ee039b6746d20c"}, - {file = "pandas-2.2.3-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:31d0ced62d4ea3e231a9f228366919a5ea0b07440d9d4dac345376fd8e1477ea"}, - {file = "pandas-2.2.3-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:7eee9e7cea6adf3e3d24e304ac6b8300646e2a5d1cd3a3c2abed9101b0846761"}, - {file = "pandas-2.2.3-cp39-cp39-win_amd64.whl", hash = "sha256:4850ba03528b6dd51d6c5d273c46f183f39a9baf3f0143e566b89450965b105e"}, - {file = "pandas-2.2.3.tar.gz", hash = "sha256:4f18ba62b61d7e192368b84517265a99b4d7ee8912f8708660fb4a366cc82667"}, -] + {file = "pandas-2.3.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:22c2e866f7209ebc3a8f08d75766566aae02bcc91d196935a1d9e59c7b990ac9"}, + {file = "pandas-2.3.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:3583d348546201aff730c8c47e49bc159833f971c2899d6097bce68b9112a4f1"}, + {file = "pandas-2.3.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0f951fbb702dacd390561e0ea45cdd8ecfa7fb56935eb3dd78e306c19104b9b0"}, + {file = "pandas-2.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:cd05b72ec02ebfb993569b4931b2e16fbb4d6ad6ce80224a3ee838387d83a191"}, + {file = "pandas-2.3.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:1b916a627919a247d865aed068eb65eb91a344b13f5b57ab9f610b7716c92de1"}, + {file = "pandas-2.3.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:fe67dc676818c186d5a3d5425250e40f179c2a89145df477dd82945eaea89e97"}, + {file = "pandas-2.3.1-cp310-cp310-win_amd64.whl", hash = "sha256:2eb789ae0274672acbd3c575b0598d213345660120a257b47b5dafdc618aec83"}, + {file = "pandas-2.3.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:2b0540963d83431f5ce8870ea02a7430adca100cec8a050f0811f8e31035541b"}, + {file = "pandas-2.3.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:fe7317f578c6a153912bd2292f02e40c1d8f253e93c599e82620c7f69755c74f"}, + {file = "pandas-2.3.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e6723a27ad7b244c0c79d8e7007092d7c8f0f11305770e2f4cd778b3ad5f9f85"}, + {file = "pandas-2.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3462c3735fe19f2638f2c3a40bd94ec2dc5ba13abbb032dd2fa1f540a075509d"}, + {file = "pandas-2.3.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:98bcc8b5bf7afed22cc753a28bc4d9e26e078e777066bc53fac7904ddef9a678"}, + {file = "pandas-2.3.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:4d544806b485ddf29e52d75b1f559142514e60ef58a832f74fb38e48d757b299"}, + {file = "pandas-2.3.1-cp311-cp311-win_amd64.whl", hash = "sha256:b3cd4273d3cb3707b6fffd217204c52ed92859533e31dc03b7c5008aa933aaab"}, + {file = "pandas-2.3.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:689968e841136f9e542020698ee1c4fbe9caa2ed2213ae2388dc7b81721510d3"}, + {file = "pandas-2.3.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:025e92411c16cbe5bb2a4abc99732a6b132f439b8aab23a59fa593eb00704232"}, + {file = "pandas-2.3.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9b7ff55f31c4fcb3e316e8f7fa194566b286d6ac430afec0d461163312c5841e"}, + {file = "pandas-2.3.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7dcb79bf373a47d2a40cf7232928eb7540155abbc460925c2c96d2d30b006eb4"}, + {file = "pandas-2.3.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:56a342b231e8862c96bdb6ab97170e203ce511f4d0429589c8ede1ee8ece48b8"}, + {file = "pandas-2.3.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ca7ed14832bce68baef331f4d7f294411bed8efd032f8109d690df45e00c4679"}, + {file = "pandas-2.3.1-cp312-cp312-win_amd64.whl", hash = "sha256:ac942bfd0aca577bef61f2bc8da8147c4ef6879965ef883d8e8d5d2dc3e744b8"}, + {file = "pandas-2.3.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:9026bd4a80108fac2239294a15ef9003c4ee191a0f64b90f170b40cfb7cf2d22"}, + {file = "pandas-2.3.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6de8547d4fdb12421e2d047a2c446c623ff4c11f47fddb6b9169eb98ffba485a"}, + {file = "pandas-2.3.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:782647ddc63c83133b2506912cc6b108140a38a37292102aaa19c81c83db2928"}, + {file = "pandas-2.3.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2ba6aff74075311fc88504b1db890187a3cd0f887a5b10f5525f8e2ef55bfdb9"}, + {file = "pandas-2.3.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:e5635178b387bd2ba4ac040f82bc2ef6e6b500483975c4ebacd34bec945fda12"}, + {file = "pandas-2.3.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6f3bf5ec947526106399a9e1d26d40ee2b259c66422efdf4de63c848492d91bb"}, + {file = "pandas-2.3.1-cp313-cp313-win_amd64.whl", hash = "sha256:1c78cf43c8fde236342a1cb2c34bcff89564a7bfed7e474ed2fffa6aed03a956"}, + {file = "pandas-2.3.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:8dfc17328e8da77be3cf9f47509e5637ba8f137148ed0e9b5241e1baf526e20a"}, + {file = "pandas-2.3.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:ec6c851509364c59a5344458ab935e6451b31b818be467eb24b0fe89bd05b6b9"}, + {file = "pandas-2.3.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:911580460fc4884d9b05254b38a6bfadddfcc6aaef856fb5859e7ca202e45275"}, + {file = "pandas-2.3.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2f4d6feeba91744872a600e6edbbd5b033005b431d5ae8379abee5bcfa479fab"}, + {file = "pandas-2.3.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:fe37e757f462d31a9cd7580236a82f353f5713a80e059a29753cf938c6775d96"}, + {file = "pandas-2.3.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:5db9637dbc24b631ff3707269ae4559bce4b7fd75c1c4d7e13f40edc42df4444"}, + {file = "pandas-2.3.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:4645f770f98d656f11c69e81aeb21c6fca076a44bed3dcbb9396a4311bc7f6d8"}, + {file = "pandas-2.3.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:342e59589cc454aaff7484d75b816a433350b3d7964d7847327edda4d532a2e3"}, + {file = "pandas-2.3.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1d12f618d80379fde6af007f65f0c25bd3e40251dbd1636480dfffce2cf1e6da"}, + {file = "pandas-2.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dd71c47a911da120d72ef173aeac0bf5241423f9bfea57320110a978457e069e"}, + {file = "pandas-2.3.1-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:09e3b1587f0f3b0913e21e8b32c3119174551deb4a4eba4a89bc7377947977e7"}, + {file = "pandas-2.3.1-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:2323294c73ed50f612f67e2bf3ae45aea04dce5690778e08a09391897f35ff88"}, + {file = "pandas-2.3.1-cp39-cp39-win_amd64.whl", hash = "sha256:b4b0de34dc8499c2db34000ef8baad684cfa4cbd836ecee05f323ebfba348c7d"}, + {file = "pandas-2.3.1.tar.gz", hash = "sha256:0a95b9ac964fe83ce317827f80304d37388ea77616b1425f0ae41c9d2d0d7bb2"}, +] +markers = {main = "extra == \"mem-reader\" or extra == \"all\""} [package.dependencies] numpy = [ @@ -2979,9 +3282,10 @@ xml = ["lxml (>=4.9.2)"] name = "pdfminer-six" version = "20250506" description = "PDF parser and analyzer" -optional = false +optional = true python-versions = ">=3.9" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "pdfminer_six-20250506-py3-none-any.whl", hash = "sha256:d81ad173f62e5f841b53a8ba63af1a4a355933cfc0ffabd608e568b9193909e3"}, {file = "pdfminer_six-20250506.tar.gz", hash = "sha256:b03cc8df09cf3c7aba8246deae52e0bca7ebb112a38895b5e1d4f5dd2b8ca2e7"}, @@ -2996,103 +3300,147 @@ dev = ["atheris ; python_version < \"3.12\"", "black", "mypy (==0.931)", "nox", docs = ["sphinx", "sphinx-argparse"] image = ["Pillow"] +[[package]] +name = "pika" +version = "1.3.2" +description = "Pika Python AMQP Client Library" +optional = true +python-versions = ">=3.7" +groups = ["main"] +markers = "extra == \"mem-scheduler\" or extra == \"all\"" +files = [ + {file = "pika-1.3.2-py3-none-any.whl", hash = "sha256:0779a7c1fafd805672796085560d290213a465e4f6f76a6fb19e378d8041a14f"}, + {file = "pika-1.3.2.tar.gz", hash = "sha256:b2a327ddddf8570b4965b3576ac77091b850262d34ce8c1d8cb4e4146aa4145f"}, +] + +[package.extras] +gevent = ["gevent"] +tornado = ["tornado"] +twisted = ["twisted"] + [[package]] name = "pillow" -version = "11.2.1" +version = "11.3.0" description = "Python Imaging Library (Fork)" optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "pillow-11.2.1-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:d57a75d53922fc20c165016a20d9c44f73305e67c351bbc60d1adaf662e74047"}, - {file = "pillow-11.2.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:127bf6ac4a5b58b3d32fc8289656f77f80567d65660bc46f72c0d77e6600cc95"}, - {file = "pillow-11.2.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b4ba4be812c7a40280629e55ae0b14a0aafa150dd6451297562e1764808bbe61"}, - {file = "pillow-11.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c8bd62331e5032bc396a93609982a9ab6b411c05078a52f5fe3cc59234a3abd1"}, - {file = "pillow-11.2.1-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:562d11134c97a62fe3af29581f083033179f7ff435f78392565a1ad2d1c2c45c"}, - {file = "pillow-11.2.1-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:c97209e85b5be259994eb5b69ff50c5d20cca0f458ef9abd835e262d9d88b39d"}, - {file = "pillow-11.2.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:0c3e6d0f59171dfa2e25d7116217543310908dfa2770aa64b8f87605f8cacc97"}, - {file = "pillow-11.2.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:cc1c3bc53befb6096b84165956e886b1729634a799e9d6329a0c512ab651e579"}, - {file = "pillow-11.2.1-cp310-cp310-win32.whl", hash = "sha256:312c77b7f07ab2139924d2639860e084ec2a13e72af54d4f08ac843a5fc9c79d"}, - {file = "pillow-11.2.1-cp310-cp310-win_amd64.whl", hash = "sha256:9bc7ae48b8057a611e5fe9f853baa88093b9a76303937449397899385da06fad"}, - {file = "pillow-11.2.1-cp310-cp310-win_arm64.whl", hash = "sha256:2728567e249cdd939f6cc3d1f049595c66e4187f3c34078cbc0a7d21c47482d2"}, - {file = "pillow-11.2.1-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:35ca289f712ccfc699508c4658a1d14652e8033e9b69839edf83cbdd0ba39e70"}, - {file = "pillow-11.2.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:e0409af9f829f87a2dfb7e259f78f317a5351f2045158be321fd135973fff7bf"}, - {file = "pillow-11.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d4e5c5edee874dce4f653dbe59db7c73a600119fbea8d31f53423586ee2aafd7"}, - {file = "pillow-11.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b93a07e76d13bff9444f1a029e0af2964e654bfc2e2c2d46bfd080df5ad5f3d8"}, - {file = "pillow-11.2.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:e6def7eed9e7fa90fde255afaf08060dc4b343bbe524a8f69bdd2a2f0018f600"}, - {file = "pillow-11.2.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:8f4f3724c068be008c08257207210c138d5f3731af6c155a81c2b09a9eb3a788"}, - {file = "pillow-11.2.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:a0a6709b47019dff32e678bc12c63008311b82b9327613f534e496dacaefb71e"}, - {file = "pillow-11.2.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f6b0c664ccb879109ee3ca702a9272d877f4fcd21e5eb63c26422fd6e415365e"}, - {file = "pillow-11.2.1-cp311-cp311-win32.whl", hash = "sha256:cc5d875d56e49f112b6def6813c4e3d3036d269c008bf8aef72cd08d20ca6df6"}, - {file = "pillow-11.2.1-cp311-cp311-win_amd64.whl", hash = "sha256:0f5c7eda47bf8e3c8a283762cab94e496ba977a420868cb819159980b6709193"}, - {file = "pillow-11.2.1-cp311-cp311-win_arm64.whl", hash = "sha256:4d375eb838755f2528ac8cbc926c3e31cc49ca4ad0cf79cff48b20e30634a4a7"}, - {file = "pillow-11.2.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:78afba22027b4accef10dbd5eed84425930ba41b3ea0a86fa8d20baaf19d807f"}, - {file = "pillow-11.2.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:78092232a4ab376a35d68c4e6d5e00dfd73454bd12b230420025fbe178ee3b0b"}, - {file = "pillow-11.2.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:25a5f306095c6780c52e6bbb6109624b95c5b18e40aab1c3041da3e9e0cd3e2d"}, - {file = "pillow-11.2.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c7b29dbd4281923a2bfe562acb734cee96bbb129e96e6972d315ed9f232bef4"}, - {file = "pillow-11.2.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:3e645b020f3209a0181a418bffe7b4a93171eef6c4ef6cc20980b30bebf17b7d"}, - {file = "pillow-11.2.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:b2dbea1012ccb784a65349f57bbc93730b96e85b42e9bf7b01ef40443db720b4"}, - {file = "pillow-11.2.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:da3104c57bbd72948d75f6a9389e6727d2ab6333c3617f0a89d72d4940aa0443"}, - {file = "pillow-11.2.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:598174aef4589af795f66f9caab87ba4ff860ce08cd5bb447c6fc553ffee603c"}, - {file = "pillow-11.2.1-cp312-cp312-win32.whl", hash = "sha256:1d535df14716e7f8776b9e7fee118576d65572b4aad3ed639be9e4fa88a1cad3"}, - {file = "pillow-11.2.1-cp312-cp312-win_amd64.whl", hash = "sha256:14e33b28bf17c7a38eede290f77db7c664e4eb01f7869e37fa98a5aa95978941"}, - {file = "pillow-11.2.1-cp312-cp312-win_arm64.whl", hash = "sha256:21e1470ac9e5739ff880c211fc3af01e3ae505859392bf65458c224d0bf283eb"}, - {file = "pillow-11.2.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:fdec757fea0b793056419bca3e9932eb2b0ceec90ef4813ea4c1e072c389eb28"}, - {file = "pillow-11.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:b0e130705d568e2f43a17bcbe74d90958e8a16263868a12c3e0d9c8162690830"}, - {file = "pillow-11.2.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7bdb5e09068332578214cadd9c05e3d64d99e0e87591be22a324bdbc18925be0"}, - {file = "pillow-11.2.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d189ba1bebfbc0c0e529159631ec72bb9e9bc041f01ec6d3233d6d82eb823bc1"}, - {file = "pillow-11.2.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:191955c55d8a712fab8934a42bfefbf99dd0b5875078240943f913bb66d46d9f"}, - {file = "pillow-11.2.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:ad275964d52e2243430472fc5d2c2334b4fc3ff9c16cb0a19254e25efa03a155"}, - {file = "pillow-11.2.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:750f96efe0597382660d8b53e90dd1dd44568a8edb51cb7f9d5d918b80d4de14"}, - {file = "pillow-11.2.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:fe15238d3798788d00716637b3d4e7bb6bde18b26e5d08335a96e88564a36b6b"}, - {file = "pillow-11.2.1-cp313-cp313-win32.whl", hash = "sha256:3fe735ced9a607fee4f481423a9c36701a39719252a9bb251679635f99d0f7d2"}, - {file = "pillow-11.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:74ee3d7ecb3f3c05459ba95eed5efa28d6092d751ce9bf20e3e253a4e497e691"}, - {file = "pillow-11.2.1-cp313-cp313-win_arm64.whl", hash = "sha256:5119225c622403afb4b44bad4c1ca6c1f98eed79db8d3bc6e4e160fc6339d66c"}, - {file = "pillow-11.2.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:8ce2e8411c7aaef53e6bb29fe98f28cd4fbd9a1d9be2eeea434331aac0536b22"}, - {file = "pillow-11.2.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:9ee66787e095127116d91dea2143db65c7bb1e232f617aa5957c0d9d2a3f23a7"}, - {file = "pillow-11.2.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9622e3b6c1d8b551b6e6f21873bdcc55762b4b2126633014cea1803368a9aa16"}, - {file = "pillow-11.2.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:63b5dff3a68f371ea06025a1a6966c9a1e1ee452fc8020c2cd0ea41b83e9037b"}, - {file = "pillow-11.2.1-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:31df6e2d3d8fc99f993fd253e97fae451a8db2e7207acf97859732273e108406"}, - {file = "pillow-11.2.1-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:062b7a42d672c45a70fa1f8b43d1d38ff76b63421cbbe7f88146b39e8a558d91"}, - {file = "pillow-11.2.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:4eb92eca2711ef8be42fd3f67533765d9fd043b8c80db204f16c8ea62ee1a751"}, - {file = "pillow-11.2.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f91ebf30830a48c825590aede79376cb40f110b387c17ee9bd59932c961044f9"}, - {file = "pillow-11.2.1-cp313-cp313t-win32.whl", hash = "sha256:e0b55f27f584ed623221cfe995c912c61606be8513bfa0e07d2c674b4516d9dd"}, - {file = "pillow-11.2.1-cp313-cp313t-win_amd64.whl", hash = "sha256:36d6b82164c39ce5482f649b437382c0fb2395eabc1e2b1702a6deb8ad647d6e"}, - {file = "pillow-11.2.1-cp313-cp313t-win_arm64.whl", hash = "sha256:225c832a13326e34f212d2072982bb1adb210e0cc0b153e688743018c94a2681"}, - {file = "pillow-11.2.1-cp39-cp39-macosx_10_10_x86_64.whl", hash = "sha256:7491cf8a79b8eb867d419648fff2f83cb0b3891c8b36da92cc7f1931d46108c8"}, - {file = "pillow-11.2.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:8b02d8f9cb83c52578a0b4beadba92e37d83a4ef11570a8688bbf43f4ca50909"}, - {file = "pillow-11.2.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:014ca0050c85003620526b0ac1ac53f56fc93af128f7546623cc8e31875ab928"}, - {file = "pillow-11.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3692b68c87096ac6308296d96354eddd25f98740c9d2ab54e1549d6c8aea9d79"}, - {file = "pillow-11.2.1-cp39-cp39-manylinux_2_28_aarch64.whl", hash = "sha256:f781dcb0bc9929adc77bad571b8621ecb1e4cdef86e940fe2e5b5ee24fd33b35"}, - {file = "pillow-11.2.1-cp39-cp39-manylinux_2_28_x86_64.whl", hash = "sha256:2b490402c96f907a166615e9a5afacf2519e28295f157ec3a2bb9bd57de638cb"}, - {file = "pillow-11.2.1-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:dd6b20b93b3ccc9c1b597999209e4bc5cf2853f9ee66e3fc9a400a78733ffc9a"}, - {file = "pillow-11.2.1-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:4b835d89c08a6c2ee7781b8dd0a30209a8012b5f09c0a665b65b0eb3560b6f36"}, - {file = "pillow-11.2.1-cp39-cp39-win32.whl", hash = "sha256:b10428b3416d4f9c61f94b494681280be7686bda15898a3a9e08eb66a6d92d67"}, - {file = "pillow-11.2.1-cp39-cp39-win_amd64.whl", hash = "sha256:6ebce70c3f486acf7591a3d73431fa504a4e18a9b97ff27f5f47b7368e4b9dd1"}, - {file = "pillow-11.2.1-cp39-cp39-win_arm64.whl", hash = "sha256:c27476257b2fdcd7872d54cfd119b3a9ce4610fb85c8e32b70b42e3680a29a1e"}, - {file = "pillow-11.2.1-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:9b7b0d4fd2635f54ad82785d56bc0d94f147096493a79985d0ab57aedd563156"}, - {file = "pillow-11.2.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:aa442755e31c64037aa7c1cb186e0b369f8416c567381852c63444dd666fb772"}, - {file = "pillow-11.2.1-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f0d3348c95b766f54b76116d53d4cb171b52992a1027e7ca50c81b43b9d9e363"}, - {file = "pillow-11.2.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:85d27ea4c889342f7e35f6d56e7e1cb345632ad592e8c51b693d7b7556043ce0"}, - {file = "pillow-11.2.1-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:bf2c33d6791c598142f00c9c4c7d47f6476731c31081331664eb26d6ab583e01"}, - {file = "pillow-11.2.1-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:e616e7154c37669fc1dfc14584f11e284e05d1c650e1c0f972f281c4ccc53193"}, - {file = "pillow-11.2.1-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:39ad2e0f424394e3aebc40168845fee52df1394a4673a6ee512d840d14ab3013"}, - {file = "pillow-11.2.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:80f1df8dbe9572b4b7abdfa17eb5d78dd620b1d55d9e25f834efdbee872d3aed"}, - {file = "pillow-11.2.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:ea926cfbc3957090becbcbbb65ad177161a2ff2ad578b5a6ec9bb1e1cd78753c"}, - {file = "pillow-11.2.1-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:738db0e0941ca0376804d4de6a782c005245264edaa253ffce24e5a15cbdc7bd"}, - {file = "pillow-11.2.1-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9db98ab6565c69082ec9b0d4e40dd9f6181dab0dd236d26f7a50b8b9bfbd5076"}, - {file = "pillow-11.2.1-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:036e53f4170e270ddb8797d4c590e6dd14d28e15c7da375c18978045f7e6c37b"}, - {file = "pillow-11.2.1-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:14f73f7c291279bd65fda51ee87affd7c1e097709f7fdd0188957a16c264601f"}, - {file = "pillow-11.2.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:208653868d5c9ecc2b327f9b9ef34e0e42a4cdd172c2988fd81d62d2bc9bc044"}, - {file = "pillow-11.2.1.tar.gz", hash = "sha256:a64dd61998416367b7ef979b73d3a85853ba9bec4c2925f74e588879a58716b6"}, -] + {file = "pillow-11.3.0-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:1b9c17fd4ace828b3003dfd1e30bff24863e0eb59b535e8f80194d9cc7ecf860"}, + {file = "pillow-11.3.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:65dc69160114cdd0ca0f35cb434633c75e8e7fad4cf855177a05bf38678f73ad"}, + {file = "pillow-11.3.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7107195ddc914f656c7fc8e4a5e1c25f32e9236ea3ea860f257b0436011fddd0"}, + {file = "pillow-11.3.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cc3e831b563b3114baac7ec2ee86819eb03caa1a2cef0b481a5675b59c4fe23b"}, + {file = "pillow-11.3.0-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f1f182ebd2303acf8c380a54f615ec883322593320a9b00438eb842c1f37ae50"}, + {file = "pillow-11.3.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4445fa62e15936a028672fd48c4c11a66d641d2c05726c7ec1f8ba6a572036ae"}, + {file = "pillow-11.3.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:71f511f6b3b91dd543282477be45a033e4845a40278fa8dcdbfdb07109bf18f9"}, + {file = "pillow-11.3.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:040a5b691b0713e1f6cbe222e0f4f74cd233421e105850ae3b3c0ceda520f42e"}, + {file = "pillow-11.3.0-cp310-cp310-win32.whl", hash = "sha256:89bd777bc6624fe4115e9fac3352c79ed60f3bb18651420635f26e643e3dd1f6"}, + {file = "pillow-11.3.0-cp310-cp310-win_amd64.whl", hash = "sha256:19d2ff547c75b8e3ff46f4d9ef969a06c30ab2d4263a9e287733aa8b2429ce8f"}, + {file = "pillow-11.3.0-cp310-cp310-win_arm64.whl", hash = "sha256:819931d25e57b513242859ce1876c58c59dc31587847bf74cfe06b2e0cb22d2f"}, + {file = "pillow-11.3.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:1cd110edf822773368b396281a2293aeb91c90a2db00d78ea43e7e861631b722"}, + {file = "pillow-11.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:9c412fddd1b77a75aa904615ebaa6001f169b26fd467b4be93aded278266b288"}, + {file = "pillow-11.3.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7d1aa4de119a0ecac0a34a9c8bde33f34022e2e8f99104e47a3ca392fd60e37d"}, + {file = "pillow-11.3.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:91da1d88226663594e3f6b4b8c3c8d85bd504117d043740a8e0ec449087cc494"}, + {file = "pillow-11.3.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:643f189248837533073c405ec2f0bb250ba54598cf80e8c1e043381a60632f58"}, + {file = "pillow-11.3.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:106064daa23a745510dabce1d84f29137a37224831d88eb4ce94bb187b1d7e5f"}, + {file = "pillow-11.3.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:cd8ff254faf15591e724dc7c4ddb6bf4793efcbe13802a4ae3e863cd300b493e"}, + {file = "pillow-11.3.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:932c754c2d51ad2b2271fd01c3d121daaa35e27efae2a616f77bf164bc0b3e94"}, + {file = "pillow-11.3.0-cp311-cp311-win32.whl", hash = "sha256:b4b8f3efc8d530a1544e5962bd6b403d5f7fe8b9e08227c6b255f98ad82b4ba0"}, + {file = "pillow-11.3.0-cp311-cp311-win_amd64.whl", hash = "sha256:1a992e86b0dd7aeb1f053cd506508c0999d710a8f07b4c791c63843fc6a807ac"}, + {file = "pillow-11.3.0-cp311-cp311-win_arm64.whl", hash = "sha256:30807c931ff7c095620fe04448e2c2fc673fcbb1ffe2a7da3fb39613489b1ddd"}, + {file = "pillow-11.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fdae223722da47b024b867c1ea0be64e0df702c5e0a60e27daad39bf960dd1e4"}, + {file = "pillow-11.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:921bd305b10e82b4d1f5e802b6850677f965d8394203d182f078873851dada69"}, + {file = "pillow-11.3.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb76541cba2f958032d79d143b98a3a6b3ea87f0959bbe256c0b5e416599fd5d"}, + {file = "pillow-11.3.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:67172f2944ebba3d4a7b54f2e95c786a3a50c21b88456329314caaa28cda70f6"}, + {file = "pillow-11.3.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97f07ed9f56a3b9b5f49d3661dc9607484e85c67e27f3e8be2c7d28ca032fec7"}, + {file = "pillow-11.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:676b2815362456b5b3216b4fd5bd89d362100dc6f4945154ff172e206a22c024"}, + {file = "pillow-11.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3e184b2f26ff146363dd07bde8b711833d7b0202e27d13540bfe2e35a323a809"}, + {file = "pillow-11.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6be31e3fc9a621e071bc17bb7de63b85cbe0bfae91bb0363c893cbe67247780d"}, + {file = "pillow-11.3.0-cp312-cp312-win32.whl", hash = "sha256:7b161756381f0918e05e7cb8a371fff367e807770f8fe92ecb20d905d0e1c149"}, + {file = "pillow-11.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:a6444696fce635783440b7f7a9fc24b3ad10a9ea3f0ab66c5905be1c19ccf17d"}, + {file = "pillow-11.3.0-cp312-cp312-win_arm64.whl", hash = "sha256:2aceea54f957dd4448264f9bf40875da0415c83eb85f55069d89c0ed436e3542"}, + {file = "pillow-11.3.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:1c627742b539bba4309df89171356fcb3cc5a9178355b2727d1b74a6cf155fbd"}, + {file = "pillow-11.3.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:30b7c02f3899d10f13d7a48163c8969e4e653f8b43416d23d13d1bbfdc93b9f8"}, + {file = "pillow-11.3.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:7859a4cc7c9295f5838015d8cc0a9c215b77e43d07a25e460f35cf516df8626f"}, + {file = "pillow-11.3.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ec1ee50470b0d050984394423d96325b744d55c701a439d2bd66089bff963d3c"}, + {file = "pillow-11.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7db51d222548ccfd274e4572fdbf3e810a5e66b00608862f947b163e613b67dd"}, + {file = "pillow-11.3.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2d6fcc902a24ac74495df63faad1884282239265c6839a0a6416d33faedfae7e"}, + {file = "pillow-11.3.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f0f5d8f4a08090c6d6d578351a2b91acf519a54986c055af27e7a93feae6d3f1"}, + {file = "pillow-11.3.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c37d8ba9411d6003bba9e518db0db0c58a680ab9fe5179f040b0463644bc9805"}, + {file = "pillow-11.3.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:13f87d581e71d9189ab21fe0efb5a23e9f28552d5be6979e84001d3b8505abe8"}, + {file = "pillow-11.3.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:023f6d2d11784a465f09fd09a34b150ea4672e85fb3d05931d89f373ab14abb2"}, + {file = "pillow-11.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:45dfc51ac5975b938e9809451c51734124e73b04d0f0ac621649821a63852e7b"}, + {file = "pillow-11.3.0-cp313-cp313-win32.whl", hash = "sha256:a4d336baed65d50d37b88ca5b60c0fa9d81e3a87d4a7930d3880d1624d5b31f3"}, + {file = "pillow-11.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:0bce5c4fd0921f99d2e858dc4d4d64193407e1b99478bc5cacecba2311abde51"}, + {file = "pillow-11.3.0-cp313-cp313-win_arm64.whl", hash = "sha256:1904e1264881f682f02b7f8167935cce37bc97db457f8e7849dc3a6a52b99580"}, + {file = "pillow-11.3.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:4c834a3921375c48ee6b9624061076bc0a32a60b5532b322cc0ea64e639dd50e"}, + {file = "pillow-11.3.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:5e05688ccef30ea69b9317a9ead994b93975104a677a36a8ed8106be9260aa6d"}, + {file = "pillow-11.3.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1019b04af07fc0163e2810167918cb5add8d74674b6267616021ab558dc98ced"}, + {file = "pillow-11.3.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f944255db153ebb2b19c51fe85dd99ef0ce494123f21b9db4877ffdfc5590c7c"}, + {file = "pillow-11.3.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1f85acb69adf2aaee8b7da124efebbdb959a104db34d3a2cb0f3793dbae422a8"}, + {file = "pillow-11.3.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:05f6ecbeff5005399bb48d198f098a9b4b6bdf27b8487c7f38ca16eeb070cd59"}, + {file = "pillow-11.3.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:a7bc6e6fd0395bc052f16b1a8670859964dbd7003bd0af2ff08342eb6e442cfe"}, + {file = "pillow-11.3.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:83e1b0161c9d148125083a35c1c5a89db5b7054834fd4387499e06552035236c"}, + {file = "pillow-11.3.0-cp313-cp313t-win32.whl", hash = "sha256:2a3117c06b8fb646639dce83694f2f9eac405472713fcb1ae887469c0d4f6788"}, + {file = "pillow-11.3.0-cp313-cp313t-win_amd64.whl", hash = "sha256:857844335c95bea93fb39e0fa2726b4d9d758850b34075a7e3ff4f4fa3aa3b31"}, + {file = "pillow-11.3.0-cp313-cp313t-win_arm64.whl", hash = "sha256:8797edc41f3e8536ae4b10897ee2f637235c94f27404cac7297f7b607dd0716e"}, + {file = "pillow-11.3.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:d9da3df5f9ea2a89b81bb6087177fb1f4d1c7146d583a3fe5c672c0d94e55e12"}, + {file = "pillow-11.3.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0b275ff9b04df7b640c59ec5a3cb113eefd3795a8df80bac69646ef699c6981a"}, + {file = "pillow-11.3.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:0743841cabd3dba6a83f38a92672cccbd69af56e3e91777b0ee7f4dba4385632"}, + {file = "pillow-11.3.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2465a69cf967b8b49ee1b96d76718cd98c4e925414ead59fdf75cf0fd07df673"}, + {file = "pillow-11.3.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:41742638139424703b4d01665b807c6468e23e699e8e90cffefe291c5832b027"}, + {file = "pillow-11.3.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:93efb0b4de7e340d99057415c749175e24c8864302369e05914682ba642e5d77"}, + {file = "pillow-11.3.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7966e38dcd0fa11ca390aed7c6f20454443581d758242023cf36fcb319b1a874"}, + {file = "pillow-11.3.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:98a9afa7b9007c67ed84c57c9e0ad86a6000da96eaa638e4f8abe5b65ff83f0a"}, + {file = "pillow-11.3.0-cp314-cp314-win32.whl", hash = "sha256:02a723e6bf909e7cea0dac1b0e0310be9d7650cd66222a5f1c571455c0a45214"}, + {file = "pillow-11.3.0-cp314-cp314-win_amd64.whl", hash = "sha256:a418486160228f64dd9e9efcd132679b7a02a5f22c982c78b6fc7dab3fefb635"}, + {file = "pillow-11.3.0-cp314-cp314-win_arm64.whl", hash = "sha256:155658efb5e044669c08896c0c44231c5e9abcaadbc5cd3648df2f7c0b96b9a6"}, + {file = "pillow-11.3.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:59a03cdf019efbfeeed910bf79c7c93255c3d54bc45898ac2a4140071b02b4ae"}, + {file = "pillow-11.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f8a5827f84d973d8636e9dc5764af4f0cf2318d26744b3d902931701b0d46653"}, + {file = "pillow-11.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ee92f2fd10f4adc4b43d07ec5e779932b4eb3dbfbc34790ada5a6669bc095aa6"}, + {file = "pillow-11.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c96d333dcf42d01f47b37e0979b6bd73ec91eae18614864622d9b87bbd5bbf36"}, + {file = "pillow-11.3.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4c96f993ab8c98460cd0c001447bff6194403e8b1d7e149ade5f00594918128b"}, + {file = "pillow-11.3.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:41342b64afeba938edb034d122b2dda5db2139b9a4af999729ba8818e0056477"}, + {file = "pillow-11.3.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:068d9c39a2d1b358eb9f245ce7ab1b5c3246c7c8c7d9ba58cfa5b43146c06e50"}, + {file = "pillow-11.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a1bc6ba083b145187f648b667e05a2534ecc4b9f2784c2cbe3089e44868f2b9b"}, + {file = "pillow-11.3.0-cp314-cp314t-win32.whl", hash = "sha256:118ca10c0d60b06d006be10a501fd6bbdfef559251ed31b794668ed569c87e12"}, + {file = "pillow-11.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:8924748b688aa210d79883357d102cd64690e56b923a186f35a82cbc10f997db"}, + {file = "pillow-11.3.0-cp314-cp314t-win_arm64.whl", hash = "sha256:79ea0d14d3ebad43ec77ad5272e6ff9bba5b679ef73375ea760261207fa8e0aa"}, + {file = "pillow-11.3.0-cp39-cp39-macosx_10_10_x86_64.whl", hash = "sha256:48d254f8a4c776de343051023eb61ffe818299eeac478da55227d96e241de53f"}, + {file = "pillow-11.3.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:7aee118e30a4cf54fdd873bd3a29de51e29105ab11f9aad8c32123f58c8f8081"}, + {file = "pillow-11.3.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:23cff760a9049c502721bdb743a7cb3e03365fafcdfc2ef9784610714166e5a4"}, + {file = "pillow-11.3.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6359a3bc43f57d5b375d1ad54a0074318a0844d11b76abccf478c37c986d3cfc"}, + {file = "pillow-11.3.0-cp39-cp39-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:092c80c76635f5ecb10f3f83d76716165c96f5229addbd1ec2bdbbda7d496e06"}, + {file = "pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cadc9e0ea0a2431124cde7e1697106471fc4c1da01530e679b2391c37d3fbb3a"}, + {file = "pillow-11.3.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:6a418691000f2a418c9135a7cf0d797c1bb7d9a485e61fe8e7722845b95ef978"}, + {file = "pillow-11.3.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:97afb3a00b65cc0804d1c7abddbf090a81eaac02768af58cbdcaaa0a931e0b6d"}, + {file = "pillow-11.3.0-cp39-cp39-win32.whl", hash = "sha256:ea944117a7974ae78059fcc1800e5d3295172bb97035c0c1d9345fca1419da71"}, + {file = "pillow-11.3.0-cp39-cp39-win_amd64.whl", hash = "sha256:e5c5858ad8ec655450a7c7df532e9842cf8df7cc349df7225c60d5d348c8aada"}, + {file = "pillow-11.3.0-cp39-cp39-win_arm64.whl", hash = "sha256:6abdbfd3aea42be05702a8dd98832329c167ee84400a1d1f61ab11437f1717eb"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:3cee80663f29e3843b68199b9d6f4f54bd1d4a6b59bdd91bceefc51238bcb967"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:b5f56c3f344f2ccaf0dd875d3e180f631dc60a51b314295a3e681fe8cf851fbe"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e67d793d180c9df62f1f40aee3accca4829d3794c95098887edc18af4b8b780c"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d000f46e2917c705e9fb93a3606ee4a819d1e3aa7a9b442f6444f07e77cf5e25"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:527b37216b6ac3a12d7838dc3bd75208ec57c1c6d11ef01902266a5a0c14fc27"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:be5463ac478b623b9dd3937afd7fb7ab3d79dd290a28e2b6df292dc75063eb8a"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:8dc70ca24c110503e16918a658b869019126ecfe03109b754c402daff12b3d9f"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:7c8ec7a017ad1bd562f93dbd8505763e688d388cde6e4a010ae1486916e713e6"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:9ab6ae226de48019caa8074894544af5b53a117ccb9d3b3dcb2871464c829438"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fe27fb049cdcca11f11a7bfda64043c37b30e6b91f10cb5bab275806c32f6ab3"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:465b9e8844e3c3519a983d58b80be3f668e2a7a5db97f2784e7079fbc9f9822c"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5418b53c0d59b3824d05e029669efa023bbef0f3e92e75ec8428f3799487f361"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:504b6f59505f08ae014f724b6207ff6222662aab5cc9542577fb084ed0676ac7"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:c84d689db21a1c397d001aa08241044aa2069e7587b398c8cc63020390b1c1b8"}, + {file = "pillow-11.3.0.tar.gz", hash = "sha256:3828ee7586cd0b2091b6209e5ad53e20d0649bbe87164a459d0676e035e8f523"}, +] +markers = {main = "extra == \"mem-reader\" or extra == \"all\""} [package.extras] -docs = ["furo", "olefile", "sphinx (>=8.2)", "sphinx-copybutton", "sphinx-inline-tabs", "sphinxext-opengraph"] +docs = ["furo", "olefile", "sphinx (>=8.2)", "sphinx-autobuild", "sphinx-copybutton", "sphinx-inline-tabs", "sphinxext-opengraph"] fpx = ["olefile"] mic = ["olefile"] test-arrow = ["pyarrow"] -tests = ["check-manifest", "coverage (>=7.4.2)", "defusedxml", "markdown2", "olefile", "packaging", "pyroma", "pytest", "pytest-cov", "pytest-timeout", "trove-classifiers (>=2024.10.12)"] +tests = ["check-manifest", "coverage (>=7.4.2)", "defusedxml", "markdown2", "olefile", "packaging", "pyroma", "pytest", "pytest-cov", "pytest-timeout", "pytest-xdist", "trove-classifiers (>=2024.10.12)"] typing = ["typing-extensions ; python_version < \"3.10\""] xmp = ["defusedxml"] @@ -3115,19 +3463,19 @@ type = ["mypy (>=1.14.1)"] [[package]] name = "pluggy" -version = "1.5.0" +version = "1.6.0" description = "plugin and hook calling mechanisms for python" optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" groups = ["test"] files = [ - {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"}, - {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"}, + {file = "pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746"}, + {file = "pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3"}, ] [package.extras] dev = ["pre-commit", "tox"] -testing = ["pytest", "pytest-benchmark"] +testing = ["coverage", "pytest", "pytest-benchmark"] [[package]] name = "portalocker" @@ -3140,6 +3488,7 @@ files = [ {file = "portalocker-2.10.1-py3-none-any.whl", hash = "sha256:53a5984ebc86a025552264b459b46a2086e269b21823cb572f8f28ee759e45bf"}, {file = "portalocker-2.10.1.tar.gz", hash = "sha256:ef1bf844e878ab08aee7e40184156e1151f228f103aa5c6bd0724cc330960f8f"}, ] +markers = {main = "extra == \"all\""} [package.dependencies] pywin32 = {version = ">=226", markers = "platform_system == \"Windows\""} @@ -3151,14 +3500,14 @@ tests = ["pytest (>=5.4.1)", "pytest-cov (>=2.8.1)", "pytest-mypy (>=0.8.0)", "p [[package]] name = "posthog" -version = "5.3.0" +version = "6.1.1" description = "Integrate PostHog into any python application." optional = false python-versions = ">=3.9" groups = ["eval"] files = [ - {file = "posthog-5.3.0-py3-none-any.whl", hash = "sha256:fc69fa5455795b02fe9d7550624cd99ceb2007795722257a544a1e5fbd491fbd"}, - {file = "posthog-5.3.0.tar.gz", hash = "sha256:5cbcaacb98584b46776552a9e4e08565a1c88a9fa1171f9aa4d2b66610c5e2ef"}, + {file = "posthog-6.1.1-py3-none-any.whl", hash = "sha256:329fd3d06b4d54cec925f47235bd8e327c91403c2f9ec38f1deb849535934dba"}, + {file = "posthog-6.1.1.tar.gz", hash = "sha256:b453f54c4a2589da859fd575dd3bf86fcb40580727ec399535f268b1b9f318b8"}, ] [package.dependencies] @@ -3167,6 +3516,7 @@ distro = ">=1.5.0" python-dateutil = ">=2.2" requests = ">=2.7,<3.0" six = ">=1.5" +typing-extensions = ">=4.2.0" [package.extras] dev = ["django-stubs", "lxml", "mypy", "mypy-baseline", "packaging", "pre-commit", "pydantic", "ruff", "setuptools", "tomli", "tomli_w", "twine", "types-mock", "types-python-dateutil", "types-requests", "types-setuptools", "types-six", "wheel"] @@ -3194,46 +3544,23 @@ virtualenv = ">=20.10.0" [[package]] name = "protobuf" -version = "6.30.2" +version = "6.31.1" description = "" optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "protobuf-6.30.2-cp310-abi3-win32.whl", hash = "sha256:b12ef7df7b9329886e66404bef5e9ce6a26b54069d7f7436a0853ccdeb91c103"}, - {file = "protobuf-6.30.2-cp310-abi3-win_amd64.whl", hash = "sha256:7653c99774f73fe6b9301b87da52af0e69783a2e371e8b599b3e9cb4da4b12b9"}, - {file = "protobuf-6.30.2-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:0eb523c550a66a09a0c20f86dd554afbf4d32b02af34ae53d93268c1f73bc65b"}, - {file = "protobuf-6.30.2-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:50f32cc9fd9cb09c783ebc275611b4f19dfdfb68d1ee55d2f0c7fa040df96815"}, - {file = "protobuf-6.30.2-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:4f6c687ae8efae6cf6093389a596548214467778146b7245e886f35e1485315d"}, - {file = "protobuf-6.30.2-cp39-cp39-win32.whl", hash = "sha256:524afedc03b31b15586ca7f64d877a98b184f007180ce25183d1a5cb230ee72b"}, - {file = "protobuf-6.30.2-cp39-cp39-win_amd64.whl", hash = "sha256:acec579c39c88bd8fbbacab1b8052c793efe83a0a5bd99db4a31423a25c0a0e2"}, - {file = "protobuf-6.30.2-py3-none-any.whl", hash = "sha256:ae86b030e69a98e08c77beab574cbcb9fff6d031d57209f574a5aea1445f4b51"}, - {file = "protobuf-6.30.2.tar.gz", hash = "sha256:35c859ae076d8c56054c25b59e5e59638d86545ed6e2b6efac6be0b6ea3ba048"}, -] - -[[package]] -name = "psutil" -version = "7.0.0" -description = "Cross-platform lib for process and system monitoring in Python. NOTE: the syntax of this script MUST be kept compatible with Python 2.7." -optional = false -python-versions = ">=3.6" -groups = ["main"] -files = [ - {file = "psutil-7.0.0-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:101d71dc322e3cffd7cea0650b09b3d08b8e7c4109dd6809fe452dfd00e58b25"}, - {file = "psutil-7.0.0-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:39db632f6bb862eeccf56660871433e111b6ea58f2caea825571951d4b6aa3da"}, - {file = "psutil-7.0.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1fcee592b4c6f146991ca55919ea3d1f8926497a713ed7faaf8225e174581e91"}, - {file = "psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4b1388a4f6875d7e2aff5c4ca1cc16c545ed41dd8bb596cefea80111db353a34"}, - {file = "psutil-7.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a5f098451abc2828f7dc6b58d44b532b22f2088f4999a937557b603ce72b1993"}, - {file = "psutil-7.0.0-cp36-cp36m-win32.whl", hash = "sha256:84df4eb63e16849689f76b1ffcb36db7b8de703d1bc1fe41773db487621b6c17"}, - {file = "psutil-7.0.0-cp36-cp36m-win_amd64.whl", hash = "sha256:1e744154a6580bc968a0195fd25e80432d3afec619daf145b9e5ba16cc1d688e"}, - {file = "psutil-7.0.0-cp37-abi3-win32.whl", hash = "sha256:ba3fcef7523064a6c9da440fc4d6bd07da93ac726b5733c29027d7dc95b39d99"}, - {file = "psutil-7.0.0-cp37-abi3-win_amd64.whl", hash = "sha256:4cf3d4eb1aa9b348dec30105c55cd9b7d4629285735a102beb4441e38db90553"}, - {file = "psutil-7.0.0.tar.gz", hash = "sha256:7be9c3eba38beccb6495ea33afd982a44074b78f28c434a1f51cc07fd315c456"}, + {file = "protobuf-6.31.1-cp310-abi3-win32.whl", hash = "sha256:7fa17d5a29c2e04b7d90e5e32388b8bfd0e7107cd8e616feef7ed3fa6bdab5c9"}, + {file = "protobuf-6.31.1-cp310-abi3-win_amd64.whl", hash = "sha256:426f59d2964864a1a366254fa703b8632dcec0790d8862d30034d8245e1cd447"}, + {file = "protobuf-6.31.1-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:6f1227473dc43d44ed644425268eb7c2e488ae245d51c6866d19fe158e207402"}, + {file = "protobuf-6.31.1-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:a40fc12b84c154884d7d4c4ebd675d5b3b5283e155f324049ae396b95ddebc39"}, + {file = "protobuf-6.31.1-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:4ee898bf66f7a8b0bd21bce523814e6fbd8c6add948045ce958b73af7e8878c6"}, + {file = "protobuf-6.31.1-cp39-cp39-win32.whl", hash = "sha256:0414e3aa5a5f3ff423828e1e6a6e907d6c65c1d5b7e6e975793d5590bdeecc16"}, + {file = "protobuf-6.31.1-cp39-cp39-win_amd64.whl", hash = "sha256:8764cf4587791e7564051b35524b72844f845ad0bb011704c3736cce762d8fe9"}, + {file = "protobuf-6.31.1-py3-none-any.whl", hash = "sha256:720a6c7e6b77288b85063569baae8536671b39f15cc22037ec7045658d80489e"}, + {file = "protobuf-6.31.1.tar.gz", hash = "sha256:d8cac4c982f0b957a4dc73a80e2ea24fab08e679c0de9deb835f4a12d69aca9a"}, ] - -[package.extras] -dev = ["abi3audit", "black (==24.10.0)", "check-manifest", "coverage", "packaging", "pylint", "pyperf", "pypinfo", "pytest", "pytest-cov", "pytest-xdist", "requests", "rstcheck", "ruff", "setuptools", "sphinx", "sphinx_rtd_theme", "toml-sort", "twine", "virtualenv", "vulture", "wheel"] -test = ["pytest", "pytest-xdist", "setuptools"] +markers = {main = "extra == \"mem-reader\" or extra == \"all\""} [[package]] name = "pycparser" @@ -3262,6 +3589,7 @@ files = [ [package.dependencies] annotated-types = ">=0.6.0" +email-validator = {version = ">=2.0.0", optional = true, markers = "extra == \"email\""} pydantic-core = "2.33.2" typing-extensions = ">=4.12.2" typing-inspection = ">=0.4.0" @@ -3408,14 +3736,14 @@ semver = ["semver (>=3.0.2)"] [[package]] name = "pydantic-settings" -version = "2.9.1" +version = "2.10.1" description = "Settings management using Pydantic" optional = false python-versions = ">=3.9" groups = ["main"] files = [ - {file = "pydantic_settings-2.9.1-py3-none-any.whl", hash = "sha256:59b4f431b1defb26fe620c71a7d3968a710d719f5f4cdbbdb7926edeb770f6ef"}, - {file = "pydantic_settings-2.9.1.tar.gz", hash = "sha256:c509bf79d27563add44e8446233359004ed85066cd096d8b510f715e6ef5d268"}, + {file = "pydantic_settings-2.10.1-py3-none-any.whl", hash = "sha256:a60952460b99cf661dc25c29c0ef171721f98bfcb52ef8d9ea4c943d7c8cc796"}, + {file = "pydantic_settings-2.10.1.tar.gz", hash = "sha256:06f0062169818d0f5524420a360d632d5857b83cffd4d42fe29597807a1614ee"}, ] [package.dependencies] @@ -3432,14 +3760,14 @@ yaml = ["pyyaml (>=6.0.1)"] [[package]] name = "pygments" -version = "2.19.1" +version = "2.19.2" description = "Pygments is a syntax highlighting package written in Python." optional = false python-versions = ">=3.8" -groups = ["main"] +groups = ["main", "test"] files = [ - {file = "pygments-2.19.1-py3-none-any.whl", hash = "sha256:9ea1544ad55cecf4b8242fab6dd35a93bbce657034b0611ee383099054ab6d8c"}, - {file = "pygments-2.19.1.tar.gz", hash = "sha256:61c16d2a8576dc0649d9f39e089b5f02bcd27fba10d8fb4dcc28173f7a45151f"}, + {file = "pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b"}, + {file = "pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887"}, ] [package.extras] @@ -3460,14 +3788,25 @@ files = [ [package.extras] diagrams = ["jinja2", "railroad-diagrams"] +[[package]] +name = "pyperclip" +version = "1.9.0" +description = "A cross-platform clipboard module for Python. (Only handles plain text for now.)" +optional = false +python-versions = "*" +groups = ["main"] +files = [ + {file = "pyperclip-1.9.0.tar.gz", hash = "sha256:b7de0142ddc81bfc5c7507eea19da920b92252b548b96186caf94a5e2527d310"}, +] + [[package]] name = "pyreadline3" version = "3.5.4" description = "A python implementation of GNU readline." -optional = false +optional = true python-versions = ">=3.8" groups = ["main"] -markers = "sys_platform == \"win32\"" +markers = "(extra == \"mem-reader\" or extra == \"all\") and sys_platform == \"win32\"" files = [ {file = "pyreadline3-3.5.4-py3-none-any.whl", hash = "sha256:eaf8e6cc3c49bcccf145fc6067ba8643d1df34d604a1ec0eccbf7a18e6d3fae6"}, {file = "pyreadline3-3.5.4.tar.gz", hash = "sha256:8d57d53039a1c75adba8e50dd3d992b28143480816187ea5efbd5c78e6c885b7"}, @@ -3478,26 +3817,27 @@ dev = ["build", "flake8", "mypy", "pytest", "twine"] [[package]] name = "pytest" -version = "8.3.5" +version = "8.4.1" description = "pytest: simple powerful testing with Python" optional = false -python-versions = ">=3.8" +python-versions = ">=3.9" groups = ["test"] files = [ - {file = "pytest-8.3.5-py3-none-any.whl", hash = "sha256:c69214aa47deac29fad6c2a4f590b9c4a9fdb16a403176fe154b79c0b4d4d820"}, - {file = "pytest-8.3.5.tar.gz", hash = "sha256:f4efe70cc14e511565ac476b57c279e12a855b11f48f212af1080ef2263d3845"}, + {file = "pytest-8.4.1-py3-none-any.whl", hash = "sha256:539c70ba6fcead8e78eebbf1115e8b589e7565830d7d006a8723f19ac8a0afb7"}, + {file = "pytest-8.4.1.tar.gz", hash = "sha256:7c67fd69174877359ed9371ec3af8a3d2b04741818c51e5e99cc1742251fa93c"}, ] [package.dependencies] -colorama = {version = "*", markers = "sys_platform == \"win32\""} -exceptiongroup = {version = ">=1.0.0rc8", markers = "python_version < \"3.11\""} -iniconfig = "*" -packaging = "*" +colorama = {version = ">=0.4", markers = "sys_platform == \"win32\""} +exceptiongroup = {version = ">=1", markers = "python_version < \"3.11\""} +iniconfig = ">=1" +packaging = ">=20" pluggy = ">=1.5,<2" +pygments = ">=2.7.2" tomli = {version = ">=1", markers = "python_version < \"3.11\""} [package.extras] -dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "pygments (>=2.7.2)", "requests", "setuptools", "xmlschema"] +dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "requests", "setuptools", "xmlschema"] [[package]] name = "pytest-asyncio" @@ -3529,6 +3869,7 @@ files = [ {file = "python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3"}, {file = "python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427"}, ] +markers = {main = "extra == \"mem-reader\" or extra == \"all\""} [package.dependencies] six = ">=1.5" @@ -3564,9 +3905,10 @@ files = [ name = "python-pptx" version = "1.0.2" description = "Create, read, and update PowerPoint 2007+ (.pptx) files." -optional = false +optional = true python-versions = ">=3.8" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "python_pptx-1.0.2-py3-none-any.whl", hash = "sha256:160838e0b8565a8b1f67947675886e9fea18aa5e795db7ae531606d68e785cba"}, {file = "python_pptx-1.0.2.tar.gz", hash = "sha256:479a8af0eaf0f0d76b6f00b0887732874ad2e3188230315290cd1f9dd9cc7095"}, @@ -3589,33 +3931,38 @@ files = [ {file = "pytz-2025.2-py2.py3-none-any.whl", hash = "sha256:5ddf76296dd8c44c26eb8f4b6f35488f3ccbf6fbbd7adee0b7262d43f0ec2f00"}, {file = "pytz-2025.2.tar.gz", hash = "sha256:360b9e3dbb49a209c21ad61809c7fb453643e048b38924c765813546746e81c3"}, ] +markers = {main = "extra == \"tree-mem\" or extra == \"all\" or extra == \"mem-reader\""} [[package]] name = "pywin32" -version = "310" +version = "311" description = "Python for Window Extensions" optional = false python-versions = "*" groups = ["main", "eval"] -markers = "platform_system == \"Windows\"" -files = [ - {file = "pywin32-310-cp310-cp310-win32.whl", hash = "sha256:6dd97011efc8bf51d6793a82292419eba2c71cf8e7250cfac03bba284454abc1"}, - {file = "pywin32-310-cp310-cp310-win_amd64.whl", hash = "sha256:c3e78706e4229b915a0821941a84e7ef420bf2b77e08c9dae3c76fd03fd2ae3d"}, - {file = "pywin32-310-cp310-cp310-win_arm64.whl", hash = "sha256:33babed0cf0c92a6f94cc6cc13546ab24ee13e3e800e61ed87609ab91e4c8213"}, - {file = "pywin32-310-cp311-cp311-win32.whl", hash = "sha256:1e765f9564e83011a63321bb9d27ec456a0ed90d3732c4b2e312b855365ed8bd"}, - {file = "pywin32-310-cp311-cp311-win_amd64.whl", hash = "sha256:126298077a9d7c95c53823934f000599f66ec9296b09167810eb24875f32689c"}, - {file = "pywin32-310-cp311-cp311-win_arm64.whl", hash = "sha256:19ec5fc9b1d51c4350be7bb00760ffce46e6c95eaf2f0b2f1150657b1a43c582"}, - {file = "pywin32-310-cp312-cp312-win32.whl", hash = "sha256:8a75a5cc3893e83a108c05d82198880704c44bbaee4d06e442e471d3c9ea4f3d"}, - {file = "pywin32-310-cp312-cp312-win_amd64.whl", hash = "sha256:bf5c397c9a9a19a6f62f3fb821fbf36cac08f03770056711f765ec1503972060"}, - {file = "pywin32-310-cp312-cp312-win_arm64.whl", hash = "sha256:2349cc906eae872d0663d4d6290d13b90621eaf78964bb1578632ff20e152966"}, - {file = "pywin32-310-cp313-cp313-win32.whl", hash = "sha256:5d241a659c496ada3253cd01cfaa779b048e90ce4b2b38cd44168ad555ce74ab"}, - {file = "pywin32-310-cp313-cp313-win_amd64.whl", hash = "sha256:667827eb3a90208ddbdcc9e860c81bde63a135710e21e4cb3348968e4bd5249e"}, - {file = "pywin32-310-cp313-cp313-win_arm64.whl", hash = "sha256:e308f831de771482b7cf692a1f308f8fca701b2d8f9dde6cc440c7da17e47b33"}, - {file = "pywin32-310-cp38-cp38-win32.whl", hash = "sha256:0867beb8addefa2e3979d4084352e4ac6e991ca45373390775f7084cc0209b9c"}, - {file = "pywin32-310-cp38-cp38-win_amd64.whl", hash = "sha256:30f0a9b3138fb5e07eb4973b7077e1883f558e40c578c6925acc7a94c34eaa36"}, - {file = "pywin32-310-cp39-cp39-win32.whl", hash = "sha256:851c8d927af0d879221e616ae1f66145253537bbdd321a77e8ef701b443a9a1a"}, - {file = "pywin32-310-cp39-cp39-win_amd64.whl", hash = "sha256:96867217335559ac619f00ad70e513c0fcf84b8a3af9fc2bba3b59b97da70475"}, -] +files = [ + {file = "pywin32-311-cp310-cp310-win32.whl", hash = "sha256:d03ff496d2a0cd4a5893504789d4a15399133fe82517455e78bad62efbb7f0a3"}, + {file = "pywin32-311-cp310-cp310-win_amd64.whl", hash = "sha256:797c2772017851984b97180b0bebe4b620bb86328e8a884bb626156295a63b3b"}, + {file = "pywin32-311-cp310-cp310-win_arm64.whl", hash = "sha256:0502d1facf1fed4839a9a51ccbcc63d952cf318f78ffc00a7e78528ac27d7a2b"}, + {file = "pywin32-311-cp311-cp311-win32.whl", hash = "sha256:184eb5e436dea364dcd3d2316d577d625c0351bf237c4e9a5fabbcfa5a58b151"}, + {file = "pywin32-311-cp311-cp311-win_amd64.whl", hash = "sha256:3ce80b34b22b17ccbd937a6e78e7225d80c52f5ab9940fe0506a1a16f3dab503"}, + {file = "pywin32-311-cp311-cp311-win_arm64.whl", hash = "sha256:a733f1388e1a842abb67ffa8e7aad0e70ac519e09b0f6a784e65a136ec7cefd2"}, + {file = "pywin32-311-cp312-cp312-win32.whl", hash = "sha256:750ec6e621af2b948540032557b10a2d43b0cee2ae9758c54154d711cc852d31"}, + {file = "pywin32-311-cp312-cp312-win_amd64.whl", hash = "sha256:b8c095edad5c211ff31c05223658e71bf7116daa0ecf3ad85f3201ea3190d067"}, + {file = "pywin32-311-cp312-cp312-win_arm64.whl", hash = "sha256:e286f46a9a39c4a18b319c28f59b61de793654af2f395c102b4f819e584b5852"}, + {file = "pywin32-311-cp313-cp313-win32.whl", hash = "sha256:f95ba5a847cba10dd8c4d8fefa9f2a6cf283b8b88ed6178fa8a6c1ab16054d0d"}, + {file = "pywin32-311-cp313-cp313-win_amd64.whl", hash = "sha256:718a38f7e5b058e76aee1c56ddd06908116d35147e133427e59a3983f703a20d"}, + {file = "pywin32-311-cp313-cp313-win_arm64.whl", hash = "sha256:7b4075d959648406202d92a2310cb990fea19b535c7f4a78d3f5e10b926eeb8a"}, + {file = "pywin32-311-cp314-cp314-win32.whl", hash = "sha256:b7a2c10b93f8986666d0c803ee19b5990885872a7de910fc460f9b0c2fbf92ee"}, + {file = "pywin32-311-cp314-cp314-win_amd64.whl", hash = "sha256:3aca44c046bd2ed8c90de9cb8427f581c479e594e99b5c0bb19b29c10fd6cb87"}, + {file = "pywin32-311-cp314-cp314-win_arm64.whl", hash = "sha256:a508e2d9025764a8270f93111a970e1d0fbfc33f4153b388bb649b7eec4f9b42"}, + {file = "pywin32-311-cp38-cp38-win32.whl", hash = "sha256:6c6f2969607b5023b0d9ce2541f8d2cbb01c4f46bc87456017cf63b73f1e2d8c"}, + {file = "pywin32-311-cp38-cp38-win_amd64.whl", hash = "sha256:c8015b09fb9a5e188f83b7b04de91ddca4658cee2ae6f3bc483f0b21a77ef6cd"}, + {file = "pywin32-311-cp39-cp39-win32.whl", hash = "sha256:aba8f82d551a942cb20d4a83413ccbac30790b50efb89a75e4f586ac0bb8056b"}, + {file = "pywin32-311-cp39-cp39-win_amd64.whl", hash = "sha256:e0c4cfb0621281fe40387df582097fd796e80430597cb9944f0ae70447bacd91"}, + {file = "pywin32-311-cp39-cp39-win_arm64.whl", hash = "sha256:62ea666235135fee79bb154e695f3ff67370afefd71bd7fea7512fc70ef31e3d"}, +] +markers = {main = "platform_system == \"Windows\" and extra == \"all\" or sys_platform == \"win32\"", eval = "platform_system == \"Windows\""} [[package]] name = "pyyaml" @@ -3682,15 +4029,16 @@ files = [ [[package]] name = "qdrant-client" -version = "1.14.2" +version = "1.14.3" description = "Client library for the Qdrant vector search engine" optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "qdrant_client-1.14.2-py3-none-any.whl", hash = "sha256:7c283b1f0e71db9c21b85d898fb395791caca2a6d56ee751da96d797b001410c"}, - {file = "qdrant_client-1.14.2.tar.gz", hash = "sha256:da5cab4d367d099d1330b6f30d45aefc8bd76f8b8f9d8fa5d4f813501b93af0d"}, + {file = "qdrant_client-1.14.3-py3-none-any.whl", hash = "sha256:66faaeae00f9b5326946851fe4ca4ddb1ad226490712e2f05142266f68dfc04d"}, + {file = "qdrant_client-1.14.3.tar.gz", hash = "sha256:bb899e3e065b79c04f5e47053d59176150c0a5dabc09d7f476c8ce8e52f4d281"}, ] +markers = {main = "extra == \"all\""} [package.dependencies] grpcio = ">=1.41.0" @@ -3706,16 +4054,17 @@ pydantic = ">=1.10.8,<2.0.dev0 || >2.2.0" urllib3 = ">=1.26.14,<3" [package.extras] -fastembed = ["fastembed (==0.6.1)"] -fastembed-gpu = ["fastembed-gpu (==0.6.1)"] +fastembed = ["fastembed (>=0.7,<0.8)"] +fastembed-gpu = ["fastembed-gpu (>=0.7,<0.8)"] [[package]] name = "redis" version = "6.2.0" description = "Python client for Redis database and key-value store" -optional = false +optional = true python-versions = ">=3.9" groups = ["main"] +markers = "extra == \"mem-scheduler\" or extra == \"all\"" files = [ {file = "redis-6.2.0-py3-none-any.whl", hash = "sha256:c8ddf316ee0aab65f04a11229e94a64b2618451dab7a67cb2f77eb799d872d5e"}, {file = "redis-6.2.0.tar.gz", hash = "sha256:e821f129b75dde6cb99dd35e5c76e8c49512a5a0d8dfdc560b2fbd44b85ca977"}, @@ -3729,6 +4078,23 @@ hiredis = ["hiredis (>=3.2.0)"] jwt = ["pyjwt (>=2.9.0)"] ocsp = ["cryptography (>=36.0.1)", "pyopenssl (>=20.0.1)", "requests (>=2.31.0)"] +[[package]] +name = "referencing" +version = "0.36.2" +description = "JSON Referencing + Python" +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "referencing-0.36.2-py3-none-any.whl", hash = "sha256:e8699adbbf8b5c7de96d8ffa0eb5c158b3beafce084968e2ea8bb08c6794dcd0"}, + {file = "referencing-0.36.2.tar.gz", hash = "sha256:df2e89862cd09deabbdba16944cc3f10feb6b3e6f18e902f7cc25609a34775aa"}, +] + +[package.dependencies] +attrs = ">=22.2.0" +rpds-py = ">=0.7.0" +typing-extensions = {version = ">=4.4.0", markers = "python_version < \"3.13\""} + [[package]] name = "regex" version = "2024.11.6" @@ -3835,19 +4201,19 @@ files = [ [[package]] name = "requests" -version = "2.32.3" +version = "2.32.4" description = "Python HTTP for Humans." optional = false python-versions = ">=3.8" groups = ["main", "eval"] files = [ - {file = "requests-2.32.3-py3-none-any.whl", hash = "sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6"}, - {file = "requests-2.32.3.tar.gz", hash = "sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760"}, + {file = "requests-2.32.4-py3-none-any.whl", hash = "sha256:27babd3cda2a6d50b30443204ee89830707d396671944c998b5975b031ac2b2c"}, + {file = "requests-2.32.4.tar.gz", hash = "sha256:27d0316682c8a29834d3264820024b62a36942083d52caf2f14c0591336d3422"}, ] [package.dependencies] certifi = ">=2017.4.17" -charset-normalizer = ">=2,<4" +charset_normalizer = ">=2,<4" idna = ">=2.5,<4" urllib3 = ">=1.21.1,<3" @@ -3890,16 +4256,32 @@ typing-extensions = {version = ">=4.0.0,<5.0", markers = "python_version < \"3.1 [package.extras] jupyter = ["ipywidgets (>=7.5.1,<9)"] +[[package]] +name = "rich-rst" +version = "1.3.1" +description = "A beautiful reStructuredText renderer for rich" +optional = false +python-versions = ">=3.6" +groups = ["main"] +files = [ + {file = "rich_rst-1.3.1-py3-none-any.whl", hash = "sha256:498a74e3896507ab04492d326e794c3ef76e7cda078703aa592d1853d91098c1"}, + {file = "rich_rst-1.3.1.tar.gz", hash = "sha256:fad46e3ba42785ea8c1785e2ceaa56e0ffa32dbe5410dec432f37e4107c4f383"}, +] + +[package.dependencies] +docutils = "*" +rich = ">=12.0.0" + [[package]] name = "rich-toolkit" -version = "0.14.7" +version = "0.14.8" description = "Rich toolkit for building command-line applications" optional = false python-versions = ">=3.8" groups = ["main"] files = [ - {file = "rich_toolkit-0.14.7-py3-none-any.whl", hash = "sha256:def05cc6e0f1176d6263b6a26648f16a62c4563b277ca2f8538683acdba1e0da"}, - {file = "rich_toolkit-0.14.7.tar.gz", hash = "sha256:6cca5a68850cc5778915f528eb785662c27ba3b4b2624612cce8340fa9701c5e"}, + {file = "rich_toolkit-0.14.8-py3-none-any.whl", hash = "sha256:c54bda82b93145a79bbae04c3e15352e6711787c470728ff41fdfa0c2f0c11ae"}, + {file = "rich_toolkit-0.14.8.tar.gz", hash = "sha256:1f77b32e6c25d9e3644c1efbce00d8d90daf2457b3abdb4699e263c03b9ca6cf"}, ] [package.dependencies] @@ -3907,6 +4289,135 @@ click = ">=8.1.7" rich = ">=13.7.1" typing-extensions = ">=4.12.2" +[[package]] +name = "rignore" +version = "0.6.2" +description = "Python Bindings for the ignore crate" +optional = false +python-versions = ">=3.8" +groups = ["main"] +files = [ + {file = "rignore-0.6.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a87091687678b3537012f26707641386f11b92e78da6a407af39193fd74a1d96"}, + {file = "rignore-0.6.2-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:2eb003ac18a0269f727ec4ad5d8d5da1ac66a1c75a01f1fbea31b826c89346ee"}, + {file = "rignore-0.6.2-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5cf62660a521b0f0b350205964aa20af6dc2d11a9ec694809714a09e005079b3"}, + {file = "rignore-0.6.2-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:48ca296c0dec4a2eff7e38812434f01cdea49339fa09def9b31a8c85f35ffc0f"}, + {file = "rignore-0.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:92c7224bdf715b24cc2166af01b8f1d79ad1af1bca1aa91310e5aa5933b41872"}, + {file = "rignore-0.6.2-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:178503a244e0823e9be94f36d42b238c500f44580adc26c17f6b05abc0fbbbb6"}, + {file = "rignore-0.6.2-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:09265fab1b913e636d743602d325d49e4304a36476a9f14727188476fa99f221"}, + {file = "rignore-0.6.2-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:24fa6a3f85b97b81f122fd744dfbbff8c05b35b4bf968a77374f4e9123463e7d"}, + {file = "rignore-0.6.2-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:7306cec150c8942507a540fa8ff417962e8aaf327135929be8270a33b645f218"}, + {file = "rignore-0.6.2-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:84f24acef49f61289c4ac51fbdb418bc394e54267b86723bd5802d2bd388bf05"}, + {file = "rignore-0.6.2-cp310-cp310-win32.whl", hash = "sha256:2fe22933bd498ec6bd16a9bdfd9fe42004b5b40d36b9dd0ff3bb45000a0a0761"}, + {file = "rignore-0.6.2-cp310-cp310-win_amd64.whl", hash = "sha256:d91e1458c0c8142fc07ba1901d26c4413a4d0aa6f152eefd0dcab15e1a699c84"}, + {file = "rignore-0.6.2-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:20da9a861a95ef82beebfbc9201896bb04f66ba0d4a52d65b956037c70a60590"}, + {file = "rignore-0.6.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:bd03d343ec059258269aaa6b4c51aab878e2b2ff92bf62269010d2f7bd584ab4"}, + {file = "rignore-0.6.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:29e7382c053eb62b916699ce6e4ccff335ecbcf14ce4c9d8786c096c236dd42d"}, + {file = "rignore-0.6.2-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a16c32b1845d41e4c39350c3392ed389053efd40659026e3aad927339e6cfb2c"}, + {file = "rignore-0.6.2-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8341122d2453867e793cd350412734f7d34d9ab4c6ce23062ea38450db46d758"}, + {file = "rignore-0.6.2-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d286840b255f750b3abe5ecb265b9dab93f18a002da4f500d8b0088b01ab9140"}, + {file = "rignore-0.6.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:177c5ed48128a33d74c0c77bcf2d57e764ef1005f9ada63637b04075a9c75045"}, + {file = "rignore-0.6.2-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:cb46444e2ed3cf340979967c6f83ebe3782d7e8882accb5cf0ef63d26c339c2a"}, + {file = "rignore-0.6.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1300c1b0827bb9bc0b343a9e4a42b73dfd2c2ded010b30ba13807f746e3e6f51"}, + {file = "rignore-0.6.2-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:5dd7bce7f1feba0d828e34f63f5ab12b3f410590c0a55e3eec8658fa64cd7378"}, + {file = "rignore-0.6.2-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:b0d7ace3b10d87a742e5665501c55978a593b6e0d510f6b53acc6b595923218a"}, + {file = "rignore-0.6.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:aaf91280106c0c696148eeab83b1744caae7372d1f55570f82a6b7e480367b5e"}, + {file = "rignore-0.6.2-cp311-cp311-win32.whl", hash = "sha256:5dfee0390640d1b70c672e93e09a909088afd0f030098e46609e2ed6c72f735b"}, + {file = "rignore-0.6.2-cp311-cp311-win_amd64.whl", hash = "sha256:b56afc3ad3114e7f6e592c2aef75a40548073c086e96a2686423e0f038113027"}, + {file = "rignore-0.6.2-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:d71f5e48aa1e16a0d56d76ffac93831918c84539d59ee0949f903e8eef97c7ba"}, + {file = "rignore-0.6.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:9ed1bfad40929b0922c127d4b812428b9283a3bb515b143c39ddb8e123caf764"}, + {file = "rignore-0.6.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:dd1ef10e348903209183cfa9639c78fcf48f3b97ec76f26df1f66d9e237aafa8"}, + {file = "rignore-0.6.2-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e91146dc5c3f04d57e8cda8fb72b132ee9b58402ecfd1387108f7b5c498b9584"}, + {file = "rignore-0.6.2-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8e9230cd680325fa5a37bb482ce4e6f019856ee63c46b88272bb3f246e2b83f8"}, + {file = "rignore-0.6.2-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1e5defb13c328c7e7ccebc8f0179eb5d6306acef1339caa5f17785c1272e29da"}, + {file = "rignore-0.6.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c269f543e922010e08ff48eaa0ff7dbf13f9f005f5f0e7a9a17afdac2d8c0e8"}, + {file = "rignore-0.6.2-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:71042fdbd57255c82b2b4070b7fac8f6a78cbc9f27d3a74523d2966e8619d661"}, + {file = "rignore-0.6.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:83fc16b20b67d09f3e958b2b1c1fa51fedc9e177c398227b32799cb365fd6fe9"}, + {file = "rignore-0.6.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:3b2a8c5e22ba99bc995db4337b4c2f3422696ffb61d17fffc2bad3bb3d0ca3c4"}, + {file = "rignore-0.6.2-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:b5c1955b697c7f8f3ab156bae43197b7f85f993dc6865842b36fd7d7f32b1626"}, + {file = "rignore-0.6.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9280867b233076afbe467a8626f4cbf688549ef16d3a8661bd59991994b4c7ad"}, + {file = "rignore-0.6.2-cp312-cp312-win32.whl", hash = "sha256:68926e2467f595272214e568e93b187362d455839e5e0368934125bc9a2fab60"}, + {file = "rignore-0.6.2-cp312-cp312-win_amd64.whl", hash = "sha256:170d32f6a6bc628c0e8bc53daf95743537a90a90ccd58d28738928846ac0c99a"}, + {file = "rignore-0.6.2-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:b50d5221ca6f869d7622c228988609b6f82ce9e0368de23bbef67ea23b0000e2"}, + {file = "rignore-0.6.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:9f9d7e302e36a2fe9188c4e5553b66cf814a0ba416dbe2f824962eda0ff92e90"}, + {file = "rignore-0.6.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:38c5bdd8bb1900ff529fbefa1e2ca3eeb669a2fafc5a81be8213fd028182d2cf"}, + {file = "rignore-0.6.2-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:819963993c25d26474a807d615d07ca4d61ca5876dbb4c058cc0adb09bf3a363"}, + {file = "rignore-0.6.2-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6c7af68bf559f5b0ab8c575b0097db7fbf58558581d5e5f44dba27fcae388149"}, + {file = "rignore-0.6.2-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0fbbfdd5efed90757112c8b2587a57868882858e94311a57b08bbc24482eb966"}, + {file = "rignore-0.6.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d8cecfc7e406fdbbc0850dd8592e30f3fbb5f0ce485f7502ecb6ce432ad0af34"}, + {file = "rignore-0.6.2-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:3165f07a8e95bbda8203d30105754b68c30002d00cc970fbe78a588957b787b7"}, + {file = "rignore-0.6.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3b2eb202a83ac6ca8eedc6aab17c76fd593ffa26fd3e3a3b8055f54f0d6254cf"}, + {file = "rignore-0.6.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:3e639fe26d5457daaa7621bd67ad78035e4ca7af50a7c75dbd020b1f05661854"}, + {file = "rignore-0.6.2-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:fe3887178401c48452984ea540a7984cb0db8dc0bca03f8dd86e2c90fa4c8e97"}, + {file = "rignore-0.6.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:17b15e6485b11dbba809836cca000fbaa6dd37305bbd35ef8b2d100f35fdb889"}, + {file = "rignore-0.6.2-cp313-cp313-win32.whl", hash = "sha256:0b1e5e1606659a7d448d78a199b11eec4d8088379fea43536bcdf869fd629389"}, + {file = "rignore-0.6.2-cp313-cp313-win_amd64.whl", hash = "sha256:2f454ebd5ce3a4c5454b78ff8df26ed7b9e9e7fca9d691bbcd8e8b5a09c2d386"}, + {file = "rignore-0.6.2-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:235972ac52d0e38be4bd81c5d4e07459af99ae2713ff5c6f7ec7593c18c7ef67"}, + {file = "rignore-0.6.2-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:763a1cac91430bad7c2ccaf6b032966448bbb44457a1915e7ad4765209928437"}, + {file = "rignore-0.6.2-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c345a1ec8f508db7d6918961318382a26bca68d315f2e71c7a93be4182eaa82c"}, + {file = "rignore-0.6.2-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8d47b76d30e434052dbc54e408eb73341c7e702af78086e0f676f8afdcff9dc8"}, + {file = "rignore-0.6.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:206f1753fa0b2921fcba36eba8d280242e088b010b5263def8856c29d3eeef34"}, + {file = "rignore-0.6.2-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:8949da2148e1eb729a8ebc7725507a58a8bb0d0191eb7429c4cea2557945cfdd"}, + {file = "rignore-0.6.2-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:fd8dd54b0d0decace1d154d29fc09e7069b7c1d92c250fc3ca8ec1a148b26ab5"}, + {file = "rignore-0.6.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c7907763174c43b38525a490e2f9bc2b01534214e8af38f7737903e8fa195574"}, + {file = "rignore-0.6.2-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c27e1e93ece4296a593ff44d4200acf1b8212e13f0d2c3f4e1ac81e790015fd"}, + {file = "rignore-0.6.2-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:d2252d603550d529362c569b10401ab32536613517e7e1df0e4477fe65498245"}, + {file = "rignore-0.6.2-cp38-cp38-macosx_10_12_x86_64.whl", hash = "sha256:ba06c53e5aac7f0cfcfb60e18c84d2dacf3f0580651e67cf5b4908f0a3e3e92d"}, + {file = "rignore-0.6.2-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:b1fa16c7631a5c14679f54e0b0fcf124d5bdaeb550ae2130fe5bf9205f13d073"}, + {file = "rignore-0.6.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d6fffb5d844762087d67f2014a2417d11585c0596b75dcaba90611dfcbf6f605"}, + {file = "rignore-0.6.2-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f91c1842220f3450db2c4a56f4836082633d1940be23f9485850326cd1c3ec33"}, + {file = "rignore-0.6.2-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e034f8701afb53d3d3223a3ad10ecfc7c7d1f5b313e5355b5d3151e51c43ce87"}, + {file = "rignore-0.6.2-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:11500184174fb477b7547c12412ee1dff041864f7ee1e88fcd8a3c223646766b"}, + {file = "rignore-0.6.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:739729cb0ed7a42684989219aeb33f951cb0ed9f62c3345ab89592befe9c4e6e"}, + {file = "rignore-0.6.2-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6ed1fcaba15c0872862dc6ef3012198bdf4ab01d2d31346a8547e4522251371a"}, + {file = "rignore-0.6.2-cp38-cp38-musllinux_1_2_aarch64.whl", hash = "sha256:a556775a8fd6142d39707bb7fbec0f95361288c099b43700813227988c4d07f7"}, + {file = "rignore-0.6.2-cp38-cp38-musllinux_1_2_armv7l.whl", hash = "sha256:0e95c770035b8ec996cfc110b4f496f7dcebb1eac2908642f60be06de21ab57a"}, + {file = "rignore-0.6.2-cp38-cp38-musllinux_1_2_i686.whl", hash = "sha256:edb2f839d18c5e07a6227cf4d3e665ae6fb8114235039613014f2f7ea2c841ed"}, + {file = "rignore-0.6.2-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:6379149257d3251732e272bccc607238a1174f3912ba14e3068332b1b0d0cae9"}, + {file = "rignore-0.6.2-cp38-cp38-win32.whl", hash = "sha256:8267eeda42b5e907f2423b6f4298f64f27acc4adb768577a8c1130c5c80a167c"}, + {file = "rignore-0.6.2-cp38-cp38-win_amd64.whl", hash = "sha256:6fddda6c5280b05755e93c6879c763e2b646bd1b0deeeabbe9c6bc05d4666c05"}, + {file = "rignore-0.6.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:208b18aa28855bb48f03b3e2571a61089b48ccaf3239c8df023a01af5d89dcac"}, + {file = "rignore-0.6.2-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:52c4476332568016df3fb4aea8fe29aeb22739652a0f1259257c05ee8e4d453b"}, + {file = "rignore-0.6.2-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:51ad94352ce8c91a9d6472f56dc6980537b04700d380f7f9ef8d4b3090545b59"}, + {file = "rignore-0.6.2-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:db11fb019a7695d9f0b5c02b8ddd9c7d74a517edf692138e6c76ee7503d2dbac"}, + {file = "rignore-0.6.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:704529e3d1f107f7a1a20f234a2c4f7d126bc64cf2effbacd4d5dce4271ce11c"}, + {file = "rignore-0.6.2-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:bbb9884db4b76ba1782164200481c62ad5236b07e2c818496d5864f5ac2cdbaf"}, + {file = "rignore-0.6.2-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:fd7ad37ce8260925d3ce5cae4cd9ed96476468a6bb5f3277738555a8813efb0b"}, + {file = "rignore-0.6.2-cp39-cp39-musllinux_1_2_armv7l.whl", hash = "sha256:63061107a0b97c97399b5e73fbbd40314849c5f24d7cc2b78858cf9e29223a1c"}, + {file = "rignore-0.6.2-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:97c75cbfb2e9dcfb2a1724a3df98078d646edebf4ad6b2672221b0f27893afaf"}, + {file = "rignore-0.6.2-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:cab1fb76e42cfd17b37b4fc69a367f4a98d1724b01d6b569b90a66c18ca7fa07"}, + {file = "rignore-0.6.2-cp39-cp39-win32.whl", hash = "sha256:1be8c358e4a850a6fee8b05db55a8373c8e6ede9ce72e2c9345ad03065b5e1aa"}, + {file = "rignore-0.6.2-cp39-cp39-win_amd64.whl", hash = "sha256:89b21a78d2ec205e3b0bcb29348894f58a7d22ae44806666cc1c1903c488b548"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7d434f4c114ca1c8fa1ae52e4ee9f5db525c76d8d6516e35f27e3e7aca60117a"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:701f84035b2b9a42727a240118eedb9bdeebe5a887aa823587f5f9881cad1e7f"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:150e8d29ec2c9e3eb9c5fe92ec4e4d1f5c048ad3ae497f7aa62b63adde88ba6f"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f896d3a44cb395c89317d4920a1af459f63d2c467edd5d46461446647850454e"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:679ea98f5796579fadc32542f7a345c0bb22bc872d8d3f3901999c344037cf2f"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:285816d7f469dd0e240fd1897de41ec6bad6a30a768d81422c85613ef367addc"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:a73660be1dd5eba7f6be8ad4f6726d429d438e2f15edbd30e1e2ee2fdd674630"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-musllinux_1_2_armv7l.whl", hash = "sha256:3a3c532cedb25114022582465d66958dd8c793e96e17677b8660dcfe8239e1d6"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-musllinux_1_2_i686.whl", hash = "sha256:bd26826ea6077101f7f25c2968fce20d3c8a01a7a18665ced682ce89e64c68ed"}, + {file = "rignore-0.6.2-pp310-pypy310_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:2071f1b70b348a25195149c5e2c09486649b4127390c54349272668a7d895314"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ee1cb4eb60624a8b4cf3931b907de184a0ef963b2d76e7eb2a63fdd177fbf368"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:912319d741069fc371bb334f43bb541fa9dd825296956279d1b3544097945fc6"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:618c3b01398b293320e401ce3eb70f9f191262a93d6414258e73e5e059efee3c"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ca6d5016b0862848d2bd1b2491ba50b2ded0e578d0c7faea6bb475831d6a076b"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bbc82beee24d5fc82ca3872461da3d2ceb371acbf2b154da87db82995d18c613"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5a91270cd3506129fb22a60f88c177dc185f49c07cf65de57515be5851ae53b8"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:1e314f9e95ff9a6b303591c4d5307c7f9267cee28be88d83e2f76e0acf3c4288"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-musllinux_1_2_armv7l.whl", hash = "sha256:973794e2379b7b7bdb57d0862e5b49224d393bb6cc27455f423c768c8e84e2aa"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-musllinux_1_2_i686.whl", hash = "sha256:8c362ff5067fb999e850be1db98a54b35791cea7aa2a037f7f9540383c617156"}, + {file = "rignore-0.6.2-pp311-pypy311_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:5c4ccfa785284e1be08e58f77445c6a98b2ec6bce87031dd091ee03c4b9e31c6"}, + {file = "rignore-0.6.2-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c61beb81593bad4e28f105155b7080514e9b944bdde6a17167d5383405151732"}, + {file = "rignore-0.6.2-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:231a47c3cd527a86e3f32141ac1a3cb39699c228500d5ac79486802a572de9a6"}, + {file = "rignore-0.6.2-pp39-pypy39_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:50ce036fe2a9d6c7b6b756a18e8b62c9f2d5ab5aee24654cbdf55e9b38ebcbab"}, + {file = "rignore-0.6.2-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7770dc9b633bf5a005dd20ff479b50d4be643d6905fb553f510d9dc19f2e52ef"}, + {file = "rignore-0.6.2-pp39-pypy39_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:2695b9dbb44ba484f5e37cf4e7a42f357420bf24617b5325797475beeaba686d"}, + {file = "rignore-0.6.2-pp39-pypy39_pp73-musllinux_1_2_armv7l.whl", hash = "sha256:d38ac7971370dd47f3d41f15fc73ca91377e28deac9ce2e02eaa9f6533003576"}, + {file = "rignore-0.6.2-pp39-pypy39_pp73-musllinux_1_2_i686.whl", hash = "sha256:714f7c582913f98ca3bcebf644820706e61ced4f3feddc35708050c3db02c973"}, + {file = "rignore-0.6.2-pp39-pypy39_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:eb6381703a0b71f7ad44aa28463249a904f6a014f0598e5540b3d7d0fa5e23f5"}, + {file = "rignore-0.6.2.tar.gz", hash = "sha256:1fef5c83a18cbd2a45e2d568ad15c369e032170231fe7cd95e44e1c80fb65497"}, +] + [[package]] name = "rouge-score" version = "0.1.2" @@ -3924,32 +4435,186 @@ nltk = "*" numpy = "*" six = ">=1.14.0" +[[package]] +name = "rpds-py" +version = "0.26.0" +description = "Python bindings to Rust's persistent data structures (rpds)" +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "rpds_py-0.26.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:4c70c70f9169692b36307a95f3d8c0a9fcd79f7b4a383aad5eaa0e9718b79b37"}, + {file = "rpds_py-0.26.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:777c62479d12395bfb932944e61e915741e364c843afc3196b694db3d669fcd0"}, + {file = "rpds_py-0.26.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ec671691e72dff75817386aa02d81e708b5a7ec0dec6669ec05213ff6b77e1bd"}, + {file = "rpds_py-0.26.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:6a1cb5d6ce81379401bbb7f6dbe3d56de537fb8235979843f0d53bc2e9815a79"}, + {file = "rpds_py-0.26.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4f789e32fa1fb6a7bf890e0124e7b42d1e60d28ebff57fe806719abb75f0e9a3"}, + {file = "rpds_py-0.26.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9c55b0a669976cf258afd718de3d9ad1b7d1fe0a91cd1ab36f38b03d4d4aeaaf"}, + {file = "rpds_py-0.26.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c70d9ec912802ecfd6cd390dadb34a9578b04f9bcb8e863d0a7598ba5e9e7ccc"}, + {file = "rpds_py-0.26.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:3021933c2cb7def39d927b9862292e0f4c75a13d7de70eb0ab06efed4c508c19"}, + {file = "rpds_py-0.26.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:8a7898b6ca3b7d6659e55cdac825a2e58c638cbf335cde41f4619e290dd0ad11"}, + {file = "rpds_py-0.26.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:12bff2ad9447188377f1b2794772f91fe68bb4bbfa5a39d7941fbebdbf8c500f"}, + {file = "rpds_py-0.26.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:191aa858f7d4902e975d4cf2f2d9243816c91e9605070aeb09c0a800d187e323"}, + {file = "rpds_py-0.26.0-cp310-cp310-win32.whl", hash = "sha256:b37a04d9f52cb76b6b78f35109b513f6519efb481d8ca4c321f6a3b9580b3f45"}, + {file = "rpds_py-0.26.0-cp310-cp310-win_amd64.whl", hash = "sha256:38721d4c9edd3eb6670437d8d5e2070063f305bfa2d5aa4278c51cedcd508a84"}, + {file = "rpds_py-0.26.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:9e8cb77286025bdb21be2941d64ac6ca016130bfdcd228739e8ab137eb4406ed"}, + {file = "rpds_py-0.26.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:5e09330b21d98adc8ccb2dbb9fc6cb434e8908d4c119aeaa772cb1caab5440a0"}, + {file = "rpds_py-0.26.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2c9c1b92b774b2e68d11193dc39620d62fd8ab33f0a3c77ecdabe19c179cdbc1"}, + {file = "rpds_py-0.26.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:824e6d3503ab990d7090768e4dfd9e840837bae057f212ff9f4f05ec6d1975e7"}, + {file = "rpds_py-0.26.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8ad7fd2258228bf288f2331f0a6148ad0186b2e3643055ed0db30990e59817a6"}, + {file = "rpds_py-0.26.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0dc23bbb3e06ec1ea72d515fb572c1fea59695aefbffb106501138762e1e915e"}, + {file = "rpds_py-0.26.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d80bf832ac7b1920ee29a426cdca335f96a2b5caa839811803e999b41ba9030d"}, + {file = "rpds_py-0.26.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:0919f38f5542c0a87e7b4afcafab6fd2c15386632d249e9a087498571250abe3"}, + {file = "rpds_py-0.26.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d422b945683e409000c888e384546dbab9009bb92f7c0b456e217988cf316107"}, + {file = "rpds_py-0.26.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:77a7711fa562ba2da1aa757e11024ad6d93bad6ad7ede5afb9af144623e5f76a"}, + {file = "rpds_py-0.26.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:238e8c8610cb7c29460e37184f6799547f7e09e6a9bdbdab4e8edb90986a2318"}, + {file = "rpds_py-0.26.0-cp311-cp311-win32.whl", hash = "sha256:893b022bfbdf26d7bedb083efeea624e8550ca6eb98bf7fea30211ce95b9201a"}, + {file = "rpds_py-0.26.0-cp311-cp311-win_amd64.whl", hash = "sha256:87a5531de9f71aceb8af041d72fc4cab4943648d91875ed56d2e629bef6d4c03"}, + {file = "rpds_py-0.26.0-cp311-cp311-win_arm64.whl", hash = "sha256:de2713f48c1ad57f89ac25b3cb7daed2156d8e822cf0eca9b96a6f990718cc41"}, + {file = "rpds_py-0.26.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:894514d47e012e794f1350f076c427d2347ebf82f9b958d554d12819849a369d"}, + {file = "rpds_py-0.26.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc921b96fa95a097add244da36a1d9e4f3039160d1d30f1b35837bf108c21136"}, + {file = "rpds_py-0.26.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3e1157659470aa42a75448b6e943c895be8c70531c43cb78b9ba990778955582"}, + {file = "rpds_py-0.26.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:521ccf56f45bb3a791182dc6b88ae5f8fa079dd705ee42138c76deb1238e554e"}, + {file = "rpds_py-0.26.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9def736773fd56b305c0eef698be5192c77bfa30d55a0e5885f80126c4831a15"}, + {file = "rpds_py-0.26.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cdad4ea3b4513b475e027be79e5a0ceac8ee1c113a1a11e5edc3c30c29f964d8"}, + {file = "rpds_py-0.26.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:82b165b07f416bdccf5c84546a484cc8f15137ca38325403864bfdf2b5b72f6a"}, + {file = "rpds_py-0.26.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:d04cab0a54b9dba4d278fe955a1390da3cf71f57feb78ddc7cb67cbe0bd30323"}, + {file = "rpds_py-0.26.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:79061ba1a11b6a12743a2b0f72a46aa2758613d454aa6ba4f5a265cc48850158"}, + {file = "rpds_py-0.26.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:f405c93675d8d4c5ac87364bb38d06c988e11028a64b52a47158a355079661f3"}, + {file = "rpds_py-0.26.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:dafd4c44b74aa4bed4b250f1aed165b8ef5de743bcca3b88fc9619b6087093d2"}, + {file = "rpds_py-0.26.0-cp312-cp312-win32.whl", hash = "sha256:3da5852aad63fa0c6f836f3359647870e21ea96cf433eb393ffa45263a170d44"}, + {file = "rpds_py-0.26.0-cp312-cp312-win_amd64.whl", hash = "sha256:cf47cfdabc2194a669dcf7a8dbba62e37a04c5041d2125fae0233b720da6f05c"}, + {file = "rpds_py-0.26.0-cp312-cp312-win_arm64.whl", hash = "sha256:20ab1ae4fa534f73647aad289003f1104092890849e0266271351922ed5574f8"}, + {file = "rpds_py-0.26.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:696764a5be111b036256c0b18cd29783fab22154690fc698062fc1b0084b511d"}, + {file = "rpds_py-0.26.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:1e6c15d2080a63aaed876e228efe4f814bc7889c63b1e112ad46fdc8b368b9e1"}, + {file = "rpds_py-0.26.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:390e3170babf42462739a93321e657444f0862c6d722a291accc46f9d21ed04e"}, + {file = "rpds_py-0.26.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7da84c2c74c0f5bc97d853d9e17bb83e2dcafcff0dc48286916001cc114379a1"}, + {file = "rpds_py-0.26.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4c5fe114a6dd480a510b6d3661d09d67d1622c4bf20660a474507aaee7eeeee9"}, + {file = "rpds_py-0.26.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3100b3090269f3a7ea727b06a6080d4eb7439dca4c0e91a07c5d133bb1727ea7"}, + {file = "rpds_py-0.26.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2c03c9b0c64afd0320ae57de4c982801271c0c211aa2d37f3003ff5feb75bb04"}, + {file = "rpds_py-0.26.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5963b72ccd199ade6ee493723d18a3f21ba7d5b957017607f815788cef50eaf1"}, + {file = "rpds_py-0.26.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9da4e873860ad5bab3291438525cae80169daecbfafe5657f7f5fb4d6b3f96b9"}, + {file = "rpds_py-0.26.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:5afaddaa8e8c7f1f7b4c5c725c0070b6eed0228f705b90a1732a48e84350f4e9"}, + {file = "rpds_py-0.26.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4916dc96489616a6f9667e7526af8fa693c0fdb4f3acb0e5d9f4400eb06a47ba"}, + {file = "rpds_py-0.26.0-cp313-cp313-win32.whl", hash = "sha256:2a343f91b17097c546b93f7999976fd6c9d5900617aa848c81d794e062ab302b"}, + {file = "rpds_py-0.26.0-cp313-cp313-win_amd64.whl", hash = "sha256:0a0b60701f2300c81b2ac88a5fb893ccfa408e1c4a555a77f908a2596eb875a5"}, + {file = "rpds_py-0.26.0-cp313-cp313-win_arm64.whl", hash = "sha256:257d011919f133a4746958257f2c75238e3ff54255acd5e3e11f3ff41fd14256"}, + {file = "rpds_py-0.26.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:529c8156d7506fba5740e05da8795688f87119cce330c244519cf706a4a3d618"}, + {file = "rpds_py-0.26.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:f53ec51f9d24e9638a40cabb95078ade8c99251945dad8d57bf4aabe86ecee35"}, + {file = "rpds_py-0.26.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7ab504c4d654e4a29558eaa5bb8cea5fdc1703ea60a8099ffd9c758472cf913f"}, + {file = "rpds_py-0.26.0-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:fd0641abca296bc1a00183fe44f7fced8807ed49d501f188faa642d0e4975b83"}, + {file = "rpds_py-0.26.0-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:69b312fecc1d017b5327afa81d4da1480f51c68810963a7336d92203dbb3d4f1"}, + {file = "rpds_py-0.26.0-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c741107203954f6fc34d3066d213d0a0c40f7bb5aafd698fb39888af277c70d8"}, + {file = "rpds_py-0.26.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fc3e55a7db08dc9a6ed5fb7103019d2c1a38a349ac41901f9f66d7f95750942f"}, + {file = "rpds_py-0.26.0-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:9e851920caab2dbcae311fd28f4313c6953993893eb5c1bb367ec69d9a39e7ed"}, + {file = "rpds_py-0.26.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:dfbf280da5f876d0b00c81f26bedce274e72a678c28845453885a9b3c22ae632"}, + {file = "rpds_py-0.26.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:1cc81d14ddfa53d7f3906694d35d54d9d3f850ef8e4e99ee68bc0d1e5fed9a9c"}, + {file = "rpds_py-0.26.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:dca83c498b4650a91efcf7b88d669b170256bf8017a5db6f3e06c2bf031f57e0"}, + {file = "rpds_py-0.26.0-cp313-cp313t-win32.whl", hash = "sha256:4d11382bcaf12f80b51d790dee295c56a159633a8e81e6323b16e55d81ae37e9"}, + {file = "rpds_py-0.26.0-cp313-cp313t-win_amd64.whl", hash = "sha256:ff110acded3c22c033e637dd8896e411c7d3a11289b2edf041f86663dbc791e9"}, + {file = "rpds_py-0.26.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:da619979df60a940cd434084355c514c25cf8eb4cf9a508510682f6c851a4f7a"}, + {file = "rpds_py-0.26.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:ea89a2458a1a75f87caabefe789c87539ea4e43b40f18cff526052e35bbb4fdf"}, + {file = "rpds_py-0.26.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:feac1045b3327a45944e7dcbeb57530339f6b17baff154df51ef8b0da34c8c12"}, + {file = "rpds_py-0.26.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b818a592bd69bfe437ee8368603d4a2d928c34cffcdf77c2e761a759ffd17d20"}, + {file = "rpds_py-0.26.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1a8b0dd8648709b62d9372fc00a57466f5fdeefed666afe3fea5a6c9539a0331"}, + {file = "rpds_py-0.26.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:6d3498ad0df07d81112aa6ec6c95a7e7b1ae00929fb73e7ebee0f3faaeabad2f"}, + {file = "rpds_py-0.26.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:24a4146ccb15be237fdef10f331c568e1b0e505f8c8c9ed5d67759dac58ac246"}, + {file = "rpds_py-0.26.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a9a63785467b2d73635957d32a4f6e73d5e4df497a16a6392fa066b753e87387"}, + {file = "rpds_py-0.26.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:de4ed93a8c91debfd5a047be327b7cc8b0cc6afe32a716bbbc4aedca9e2a83af"}, + {file = "rpds_py-0.26.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:caf51943715b12af827696ec395bfa68f090a4c1a1d2509eb4e2cb69abbbdb33"}, + {file = "rpds_py-0.26.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:4a59e5bc386de021f56337f757301b337d7ab58baa40174fb150accd480bc953"}, + {file = "rpds_py-0.26.0-cp314-cp314-win32.whl", hash = "sha256:92c8db839367ef16a662478f0a2fe13e15f2227da3c1430a782ad0f6ee009ec9"}, + {file = "rpds_py-0.26.0-cp314-cp314-win_amd64.whl", hash = "sha256:b0afb8cdd034150d4d9f53926226ed27ad15b7f465e93d7468caaf5eafae0d37"}, + {file = "rpds_py-0.26.0-cp314-cp314-win_arm64.whl", hash = "sha256:ca3f059f4ba485d90c8dc75cb5ca897e15325e4e609812ce57f896607c1c0867"}, + {file = "rpds_py-0.26.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:5afea17ab3a126006dc2f293b14ffc7ef3c85336cf451564a0515ed7648033da"}, + {file = "rpds_py-0.26.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:69f0c0a3df7fd3a7eec50a00396104bb9a843ea6d45fcc31c2d5243446ffd7a7"}, + {file = "rpds_py-0.26.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:801a71f70f9813e82d2513c9a96532551fce1e278ec0c64610992c49c04c2dad"}, + {file = "rpds_py-0.26.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:df52098cde6d5e02fa75c1f6244f07971773adb4a26625edd5c18fee906fa84d"}, + {file = "rpds_py-0.26.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9bc596b30f86dc6f0929499c9e574601679d0341a0108c25b9b358a042f51bca"}, + {file = "rpds_py-0.26.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9dfbe56b299cf5875b68eb6f0ebaadc9cac520a1989cac0db0765abfb3709c19"}, + {file = "rpds_py-0.26.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ac64f4b2bdb4ea622175c9ab7cf09444e412e22c0e02e906978b3b488af5fde8"}, + {file = "rpds_py-0.26.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:181ef9b6bbf9845a264f9aa45c31836e9f3c1f13be565d0d010e964c661d1e2b"}, + {file = "rpds_py-0.26.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:49028aa684c144ea502a8e847d23aed5e4c2ef7cadfa7d5eaafcb40864844b7a"}, + {file = "rpds_py-0.26.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:e5d524d68a474a9688336045bbf76cb0def88549c1b2ad9dbfec1fb7cfbe9170"}, + {file = "rpds_py-0.26.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:c1851f429b822831bd2edcbe0cfd12ee9ea77868f8d3daf267b189371671c80e"}, + {file = "rpds_py-0.26.0-cp314-cp314t-win32.whl", hash = "sha256:7bdb17009696214c3b66bb3590c6d62e14ac5935e53e929bcdbc5a495987a84f"}, + {file = "rpds_py-0.26.0-cp314-cp314t-win_amd64.whl", hash = "sha256:f14440b9573a6f76b4ee4770c13f0b5921f71dde3b6fcb8dabbefd13b7fe05d7"}, + {file = "rpds_py-0.26.0-cp39-cp39-macosx_10_12_x86_64.whl", hash = "sha256:7a48af25d9b3c15684059d0d1fc0bc30e8eee5ca521030e2bffddcab5be40226"}, + {file = "rpds_py-0.26.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:0c71c2f6bf36e61ee5c47b2b9b5d47e4d1baad6426bfed9eea3e858fc6ee8806"}, + {file = "rpds_py-0.26.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1d815d48b1804ed7867b539236b6dd62997850ca1c91cad187f2ddb1b7bbef19"}, + {file = "rpds_py-0.26.0-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:84cfbd4d4d2cdeb2be61a057a258d26b22877266dd905809e94172dff01a42ae"}, + {file = "rpds_py-0.26.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:fbaa70553ca116c77717f513e08815aec458e6b69a028d4028d403b3bc84ff37"}, + {file = "rpds_py-0.26.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:39bfea47c375f379d8e87ab4bb9eb2c836e4f2069f0f65731d85e55d74666387"}, + {file = "rpds_py-0.26.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1533b7eb683fb5f38c1d68a3c78f5fdd8f1412fa6b9bf03b40f450785a0ab915"}, + {file = "rpds_py-0.26.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c5ab0ee51f560d179b057555b4f601b7df909ed31312d301b99f8b9fc6028284"}, + {file = "rpds_py-0.26.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:e5162afc9e0d1f9cae3b577d9c29ddbab3505ab39012cb794d94a005825bde21"}, + {file = "rpds_py-0.26.0-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:43f10b007033f359bc3fa9cd5e6c1e76723f056ffa9a6b5c117cc35720a80292"}, + {file = "rpds_py-0.26.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:e3730a48e5622e598293eee0762b09cff34dd3f271530f47b0894891281f051d"}, + {file = "rpds_py-0.26.0-cp39-cp39-win32.whl", hash = "sha256:4b1f66eb81eab2e0ff5775a3a312e5e2e16bf758f7b06be82fb0d04078c7ac51"}, + {file = "rpds_py-0.26.0-cp39-cp39-win_amd64.whl", hash = "sha256:519067e29f67b5c90e64fb1a6b6e9d2ec0ba28705c51956637bac23a2f4ddae1"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:3c0909c5234543ada2515c05dc08595b08d621ba919629e94427e8e03539c958"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:c1fb0cda2abcc0ac62f64e2ea4b4e64c57dfd6b885e693095460c61bde7bb18e"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:84d142d2d6cf9b31c12aa4878d82ed3b2324226270b89b676ac62ccd7df52d08"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a547e21c5610b7e9093d870be50682a6a6cf180d6da0f42c47c306073bfdbbf6"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:35e9a70a0f335371275cdcd08bc5b8051ac494dd58bff3bbfb421038220dc871"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0dfa6115c6def37905344d56fb54c03afc49104e2ca473d5dedec0f6606913b4"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:313cfcd6af1a55a286a3c9a25f64af6d0e46cf60bc5798f1db152d97a216ff6f"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:f7bf2496fa563c046d05e4d232d7b7fd61346e2402052064b773e5c378bf6f73"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:aa81873e2c8c5aa616ab8e017a481a96742fdf9313c40f14338ca7dbf50cb55f"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-musllinux_1_2_i686.whl", hash = "sha256:68ffcf982715f5b5b7686bdd349ff75d422e8f22551000c24b30eaa1b7f7ae84"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:6188de70e190847bb6db3dc3981cbadff87d27d6fe9b4f0e18726d55795cee9b"}, + {file = "rpds_py-0.26.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:1c962145c7473723df9722ba4c058de12eb5ebedcb4e27e7d902920aa3831ee8"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:f61a9326f80ca59214d1cceb0a09bb2ece5b2563d4e0cd37bfd5515c28510674"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:183f857a53bcf4b1b42ef0f57ca553ab56bdd170e49d8091e96c51c3d69ca696"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:941c1cfdf4799d623cf3aa1d326a6b4fdb7a5799ee2687f3516738216d2262fb"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:72a8d9564a717ee291f554eeb4bfeafe2309d5ec0aa6c475170bdab0f9ee8e88"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:511d15193cbe013619dd05414c35a7dedf2088fcee93c6bbb7c77859765bd4e8"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:aea1f9741b603a8d8fedb0ed5502c2bc0accbc51f43e2ad1337fe7259c2b77a5"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4019a9d473c708cf2f16415688ef0b4639e07abaa569d72f74745bbeffafa2c7"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:093d63b4b0f52d98ebae33b8c50900d3d67e0666094b1be7a12fffd7f65de74b"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:2abe21d8ba64cded53a2a677e149ceb76dcf44284202d737178afe7ba540c1eb"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-musllinux_1_2_i686.whl", hash = "sha256:4feb7511c29f8442cbbc28149a92093d32e815a28aa2c50d333826ad2a20fdf0"}, + {file = "rpds_py-0.26.0-pp311-pypy311_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:e99685fc95d386da368013e7fb4269dd39c30d99f812a8372d62f244f662709c"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-macosx_10_12_x86_64.whl", hash = "sha256:a90a13408a7a856b87be8a9f008fff53c5080eea4e4180f6c2e546e4a972fb5d"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-macosx_11_0_arm64.whl", hash = "sha256:3ac51b65e8dc76cf4949419c54c5528adb24fc721df722fd452e5fbc236f5c40"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:59b2093224a18c6508d95cfdeba8db9cbfd6f3494e94793b58972933fcee4c6d"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4f01a5d6444a3258b00dc07b6ea4733e26f8072b788bef750baa37b370266137"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b6e2c12160c72aeda9d1283e612f68804621f448145a210f1bf1d79151c47090"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cb28c1f569f8d33b2b5dcd05d0e6ef7005d8639c54c2f0be824f05aedf715255"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1766b5724c3f779317d5321664a343c07773c8c5fd1532e4039e6cc7d1a815be"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:b6d9e5a2ed9c4988c8f9b28b3bc0e3e5b1aaa10c28d210a594ff3a8c02742daf"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:b5f7a446ddaf6ca0fad9a5535b56fbfc29998bf0e0b450d174bbec0d600e1d72"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-musllinux_1_2_i686.whl", hash = "sha256:eed5ac260dd545fbc20da5f4f15e7efe36a55e0e7cf706e4ec005b491a9546a0"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:582462833ba7cee52e968b0341b85e392ae53d44c0f9af6a5927c80e539a8b67"}, + {file = "rpds_py-0.26.0-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:69a607203441e07e9a8a529cff1d5b73f6a160f22db1097211e6212a68567d11"}, + {file = "rpds_py-0.26.0.tar.gz", hash = "sha256:20dae58a859b0906f0685642e591056f1e787f3a8b39c8e8749a45dc7d26bdb0"}, +] + [[package]] name = "ruff" -version = "0.11.8" +version = "0.11.13" description = "An extremely fast Python linter and code formatter, written in Rust." optional = false python-versions = ">=3.7" -groups = ["dev"] +groups = ["test"] files = [ - {file = "ruff-0.11.8-py3-none-linux_armv6l.whl", hash = "sha256:896a37516c594805e34020c4a7546c8f8a234b679a7716a3f08197f38913e1a3"}, - {file = "ruff-0.11.8-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:ab86d22d3d721a40dd3ecbb5e86ab03b2e053bc93c700dc68d1c3346b36ce835"}, - {file = "ruff-0.11.8-py3-none-macosx_11_0_arm64.whl", hash = "sha256:258f3585057508d317610e8a412788cf726efeefa2fec4dba4001d9e6f90d46c"}, - {file = "ruff-0.11.8-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:727d01702f7c30baed3fc3a34901a640001a2828c793525043c29f7614994a8c"}, - {file = "ruff-0.11.8-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3dca977cc4fc8f66e89900fa415ffe4dbc2e969da9d7a54bfca81a128c5ac219"}, - {file = "ruff-0.11.8-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c657fa987d60b104d2be8b052d66da0a2a88f9bd1d66b2254333e84ea2720c7f"}, - {file = "ruff-0.11.8-py3-none-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:f2e74b021d0de5eceb8bd32919f6ff8a9b40ee62ed97becd44993ae5b9949474"}, - {file = "ruff-0.11.8-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f9b5ef39820abc0f2c62111f7045009e46b275f5b99d5e59dda113c39b7f4f38"}, - {file = "ruff-0.11.8-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c1dba3135ca503727aa4648152c0fa67c3b1385d3dc81c75cd8a229c4b2a1458"}, - {file = "ruff-0.11.8-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7f024d32e62faad0f76b2d6afd141b8c171515e4fb91ce9fd6464335c81244e5"}, - {file = "ruff-0.11.8-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:d365618d3ad747432e1ae50d61775b78c055fee5936d77fb4d92c6f559741948"}, - {file = "ruff-0.11.8-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:4d9aaa91035bdf612c8ee7266153bcf16005c7c7e2f5878406911c92a31633cb"}, - {file = "ruff-0.11.8-py3-none-musllinux_1_2_i686.whl", hash = "sha256:0eba551324733efc76116d9f3a0d52946bc2751f0cd30661564117d6fd60897c"}, - {file = "ruff-0.11.8-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:161eb4cff5cfefdb6c9b8b3671d09f7def2f960cee33481dd898caf2bcd02304"}, - {file = "ruff-0.11.8-py3-none-win32.whl", hash = "sha256:5b18caa297a786465cc511d7f8be19226acf9c0a1127e06e736cd4e1878c3ea2"}, - {file = "ruff-0.11.8-py3-none-win_amd64.whl", hash = "sha256:6e70d11043bef637c5617297bdedec9632af15d53ac1e1ba29c448da9341b0c4"}, - {file = "ruff-0.11.8-py3-none-win_arm64.whl", hash = "sha256:304432e4c4a792e3da85b7699feb3426a0908ab98bf29df22a31b0cdd098fac2"}, - {file = "ruff-0.11.8.tar.gz", hash = "sha256:6d742d10626f9004b781f4558154bb226620a7242080e11caeffab1a40e99df8"}, + {file = "ruff-0.11.13-py3-none-linux_armv6l.whl", hash = "sha256:4bdfbf1240533f40042ec00c9e09a3aade6f8c10b6414cf11b519488d2635d46"}, + {file = "ruff-0.11.13-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:aef9c9ed1b5ca28bb15c7eac83b8670cf3b20b478195bd49c8d756ba0a36cf48"}, + {file = "ruff-0.11.13-py3-none-macosx_11_0_arm64.whl", hash = "sha256:53b15a9dfdce029c842e9a5aebc3855e9ab7771395979ff85b7c1dedb53ddc2b"}, + {file = "ruff-0.11.13-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ab153241400789138d13f362c43f7edecc0edfffce2afa6a68434000ecd8f69a"}, + {file = "ruff-0.11.13-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:6c51f93029d54a910d3d24f7dd0bb909e31b6cd989a5e4ac513f4eb41629f0dc"}, + {file = "ruff-0.11.13-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1808b3ed53e1a777c2ef733aca9051dc9bf7c99b26ece15cb59a0320fbdbd629"}, + {file = "ruff-0.11.13-py3-none-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:d28ce58b5ecf0f43c1b71edffabe6ed7f245d5336b17805803312ec9bc665933"}, + {file = "ruff-0.11.13-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:55e4bc3a77842da33c16d55b32c6cac1ec5fb0fbec9c8c513bdce76c4f922165"}, + {file = "ruff-0.11.13-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:633bf2c6f35678c56ec73189ba6fa19ff1c5e4807a78bf60ef487b9dd272cc71"}, + {file = "ruff-0.11.13-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4ffbc82d70424b275b089166310448051afdc6e914fdab90e08df66c43bb5ca9"}, + {file = "ruff-0.11.13-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:4a9ddd3ec62a9a89578c85842b836e4ac832d4a2e0bfaad3b02243f930ceafcc"}, + {file = "ruff-0.11.13-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:d237a496e0778d719efb05058c64d28b757c77824e04ffe8796c7436e26712b7"}, + {file = "ruff-0.11.13-py3-none-musllinux_1_2_i686.whl", hash = "sha256:26816a218ca6ef02142343fd24c70f7cd8c5aa6c203bca284407adf675984432"}, + {file = "ruff-0.11.13-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:51c3f95abd9331dc5b87c47ac7f376db5616041173826dfd556cfe3d4977f492"}, + {file = "ruff-0.11.13-py3-none-win32.whl", hash = "sha256:96c27935418e4e8e77a26bb05962817f28b8ef3843a6c6cc49d8783b5507f250"}, + {file = "ruff-0.11.13-py3-none-win_amd64.whl", hash = "sha256:29c3189895a8a6a657b7af4e97d330c8a3afd2c9c8f46c81e2fc5a31866517e3"}, + {file = "ruff-0.11.13-py3-none-win_arm64.whl", hash = "sha256:b4385285e9179d608ff1d2fb9922062663c658605819a6876d8beef0c30b7f3b"}, + {file = "ruff-0.11.13.tar.gz", hash = "sha256:26fa247dc68d1d4e72c179e08889a25ac0c7ba4d78aecfc835d49cbfd60bf514"}, ] [[package]] @@ -3994,9 +4659,10 @@ torch = ["safetensors[numpy]", "torch (>=1.10)"] name = "schedule" version = "1.2.2" description = "Job scheduling for humans." -optional = false +optional = true python-versions = ">=3.7" groups = ["main"] +markers = "extra == \"tree-mem\" or extra == \"all\"" files = [ {file = "schedule-1.2.2-py3-none-any.whl", hash = "sha256:5bef4a2a0183abf44046ae0d164cadcac21b1db011bdd8102e4a0c1e91e06a7d"}, {file = "schedule-1.2.2.tar.gz", hash = "sha256:15fe9c75fe5fd9b9627f3f19cc0ef1420508f9f9a46f45cd0769ef75ede5f0b7"}, @@ -4180,9 +4846,10 @@ test = ["Cython", "array-api-strict (>=2.3.1)", "asv", "gmpy2", "hypothesis (>=6 name = "sentence-transformers" version = "4.1.0" description = "Embeddings, Retrieval, and Reranking" -optional = false +optional = true python-versions = ">=3.9" groups = ["main"] +markers = "extra == \"all\"" files = [ {file = "sentence_transformers-4.1.0-py3-none-any.whl", hash = "sha256:382a7f6be1244a100ce40495fb7523dbe8d71b3c10b299f81e6b735092b3b8ca"}, {file = "sentence_transformers-4.1.0.tar.gz", hash = "sha256:f125ffd1c727533e0eca5d4567de72f84728de8f7482834de442fd90c2c3d50b"}, @@ -4205,6 +4872,63 @@ onnx-gpu = ["optimum[onnxruntime-gpu] (>=1.23.1)"] openvino = ["optimum-intel[openvino] (>=1.20.0)"] train = ["accelerate (>=0.20.3)", "datasets"] +[[package]] +name = "sentry-sdk" +version = "2.33.0" +description = "Python client for Sentry (https://sentry.io)" +optional = false +python-versions = ">=3.6" +groups = ["main"] +files = [ + {file = "sentry_sdk-2.33.0-py2.py3-none-any.whl", hash = "sha256:a762d3f19a1c240e16c98796f2a5023f6e58872997d5ae2147ac3ed378b23ec2"}, + {file = "sentry_sdk-2.33.0.tar.gz", hash = "sha256:cdceed05e186846fdf80ceea261fe0a11ebc93aab2f228ed73d076a07804152e"}, +] + +[package.dependencies] +certifi = "*" +urllib3 = ">=1.26.11" + +[package.extras] +aiohttp = ["aiohttp (>=3.5)"] +anthropic = ["anthropic (>=0.16)"] +arq = ["arq (>=0.23)"] +asyncpg = ["asyncpg (>=0.23)"] +beam = ["apache-beam (>=2.12)"] +bottle = ["bottle (>=0.12.13)"] +celery = ["celery (>=3)"] +celery-redbeat = ["celery-redbeat (>=2)"] +chalice = ["chalice (>=1.16.0)"] +clickhouse-driver = ["clickhouse-driver (>=0.2.0)"] +django = ["django (>=1.8)"] +falcon = ["falcon (>=1.4)"] +fastapi = ["fastapi (>=0.79.0)"] +flask = ["blinker (>=1.1)", "flask (>=0.11)", "markupsafe"] +grpcio = ["grpcio (>=1.21.1)", "protobuf (>=3.8.0)"] +http2 = ["httpcore[http2] (==1.*)"] +httpx = ["httpx (>=0.16.0)"] +huey = ["huey (>=2)"] +huggingface-hub = ["huggingface_hub (>=0.22)"] +langchain = ["langchain (>=0.0.210)"] +launchdarkly = ["launchdarkly-server-sdk (>=9.8.0)"] +litestar = ["litestar (>=2.0.0)"] +loguru = ["loguru (>=0.5)"] +openai = ["openai (>=1.0.0)", "tiktoken (>=0.3.0)"] +openfeature = ["openfeature-sdk (>=0.7.1)"] +opentelemetry = ["opentelemetry-distro (>=0.35b0)"] +opentelemetry-experimental = ["opentelemetry-distro"] +pure-eval = ["asttokens", "executing", "pure_eval"] +pymongo = ["pymongo (>=3.1)"] +pyspark = ["pyspark (>=2.4.4)"] +quart = ["blinker (>=1.1)", "quart (>=0.16.1)"] +rq = ["rq (>=0.6)"] +sanic = ["sanic (>=0.8)"] +sqlalchemy = ["sqlalchemy (>=1.2)"] +starlette = ["starlette (>=0.19.1)"] +starlite = ["starlite (>=1.48)"] +statsig = ["statsig (>=0.55.3)"] +tornado = ["tornado (>=6)"] +unleash = ["UnleashClient (>=6.0.1)"] + [[package]] name = "setuptools" version = "80.9.0" @@ -4212,11 +4936,11 @@ description = "Easily download, build, install, upgrade, and uninstall Python pa optional = false python-versions = ">=3.9" groups = ["main", "eval"] -markers = "platform_system == \"Linux\" and platform_machine == \"x86_64\" or python_version >= \"3.12\"" files = [ {file = "setuptools-80.9.0-py3-none-any.whl", hash = "sha256:062d34222ad13e0cc312a4c02d73f059e86a4acbfbdea8f8f76b28c99f306922"}, {file = "setuptools-80.9.0.tar.gz", hash = "sha256:f36b47402ecde768dbfafc46e8e4207b4360c654f1f3bb84475f0a28628fb19c"}, ] +markers = {main = "extra == \"all\" and platform_system == \"Linux\" and platform_machine == \"x86_64\" or python_version >= \"3.12\" and extra == \"all\"", eval = "platform_system == \"Linux\" and platform_machine == \"x86_64\" or python_version >= \"3.12\""} [package.extras] check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1) ; sys_platform != \"cygwin\"", "ruff (>=0.8.0) ; sys_platform != \"cygwin\""] @@ -4250,6 +4974,7 @@ files = [ {file = "six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274"}, {file = "six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81"}, ] +markers = {main = "extra == \"mem-reader\" or extra == \"all\""} [[package]] name = "sniffio" @@ -4267,9 +4992,10 @@ files = [ name = "soupsieve" version = "2.7" description = "A modern CSS selector implementation for Beautiful Soup." -optional = false +optional = true python-versions = ">=3.8" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ {file = "soupsieve-2.7-py3-none-any.whl", hash = "sha256:6e60cc5c1ffaf1cebcc12e8188320b72071e922c2e897f737cadce79ad5d30c4"}, {file = "soupsieve-2.7.tar.gz", hash = "sha256:ad282f9b6926286d2ead4750552c8a6142bc4c783fd66b0293547c8fe6ae126a"}, @@ -4371,6 +5097,27 @@ postgresql-psycopgbinary = ["psycopg[binary] (>=3.0.7)"] pymysql = ["pymysql"] sqlcipher = ["sqlcipher3_binary"] +[[package]] +name = "sse-starlette" +version = "2.4.1" +description = "SSE plugin for Starlette" +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "sse_starlette-2.4.1-py3-none-any.whl", hash = "sha256:08b77ea898ab1a13a428b2b6f73cfe6d0e607a7b4e15b9bb23e4a37b087fd39a"}, + {file = "sse_starlette-2.4.1.tar.gz", hash = "sha256:7c8a800a1ca343e9165fc06bbda45c78e4c6166320707ae30b416c42da070926"}, +] + +[package.dependencies] +anyio = ">=4.7.0" + +[package.extras] +daphne = ["daphne (>=4.2.0)"] +examples = ["aiosqlite (>=0.21.0)", "fastapi (>=0.115.12)", "sqlalchemy[asyncio,examples] (>=2.0.41)", "starlette (>=0.41.3)", "uvicorn (>=0.34.0)"] +granian = ["granian (>=2.3.1)"] +uvicorn = ["uvicorn (>=0.34.0)"] + [[package]] name = "starlette" version = "0.46.2" @@ -4400,6 +5147,7 @@ files = [ {file = "sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5"}, {file = "sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517"}, ] +markers = {main = "extra == \"mem-reader\" or extra == \"all\""} [package.dependencies] mpmath = ">=1.1.0,<1.4" @@ -4485,27 +5233,27 @@ blobfile = ["blobfile (>=2)"] [[package]] name = "tokenizers" -version = "0.21.1" +version = "0.21.2" description = "" optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "tokenizers-0.21.1-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:e78e413e9e668ad790a29456e677d9d3aa50a9ad311a40905d6861ba7692cf41"}, - {file = "tokenizers-0.21.1-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:cd51cd0a91ecc801633829fcd1fda9cf8682ed3477c6243b9a095539de4aecf3"}, - {file = "tokenizers-0.21.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:28da6b72d4fb14ee200a1bd386ff74ade8992d7f725f2bde2c495a9a98cf4d9f"}, - {file = "tokenizers-0.21.1-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:34d8cfde551c9916cb92014e040806122295a6800914bab5865deb85623931cf"}, - {file = "tokenizers-0.21.1-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:aaa852d23e125b73d283c98f007e06d4595732104b65402f46e8ef24b588d9f8"}, - {file = "tokenizers-0.21.1-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a21a15d5c8e603331b8a59548bbe113564136dc0f5ad8306dd5033459a226da0"}, - {file = "tokenizers-0.21.1-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2fdbd4c067c60a0ac7eca14b6bd18a5bebace54eb757c706b47ea93204f7a37c"}, - {file = "tokenizers-0.21.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2dd9a0061e403546f7377df940e866c3e678d7d4e9643d0461ea442b4f89e61a"}, - {file = "tokenizers-0.21.1-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:db9484aeb2e200c43b915a1a0150ea885e35f357a5a8fabf7373af333dcc8dbf"}, - {file = "tokenizers-0.21.1-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:ed248ab5279e601a30a4d67bdb897ecbe955a50f1e7bb62bd99f07dd11c2f5b6"}, - {file = "tokenizers-0.21.1-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:9ac78b12e541d4ce67b4dfd970e44c060a2147b9b2a21f509566d556a509c67d"}, - {file = "tokenizers-0.21.1-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e5a69c1a4496b81a5ee5d2c1f3f7fbdf95e90a0196101b0ee89ed9956b8a168f"}, - {file = "tokenizers-0.21.1-cp39-abi3-win32.whl", hash = "sha256:1039a3a5734944e09de1d48761ade94e00d0fa760c0e0551151d4dd851ba63e3"}, - {file = "tokenizers-0.21.1-cp39-abi3-win_amd64.whl", hash = "sha256:0f0dcbcc9f6e13e675a66d7a5f2f225a736745ce484c1a4e07476a89ccdad382"}, - {file = "tokenizers-0.21.1.tar.gz", hash = "sha256:a1bb04dc5b448985f86ecd4b05407f5a8d97cb2c0532199b2a302a604a0165ab"}, + {file = "tokenizers-0.21.2-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:342b5dfb75009f2255ab8dec0041287260fed5ce00c323eb6bab639066fef8ec"}, + {file = "tokenizers-0.21.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:126df3205d6f3a93fea80c7a8a266a78c1bd8dd2fe043386bafdd7736a23e45f"}, + {file = "tokenizers-0.21.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4a32cd81be21168bd0d6a0f0962d60177c447a1aa1b1e48fa6ec9fc728ee0b12"}, + {file = "tokenizers-0.21.2-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8bd8999538c405133c2ab999b83b17c08b7fc1b48c1ada2469964605a709ef91"}, + {file = "tokenizers-0.21.2-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:5e9944e61239b083a41cf8fc42802f855e1dca0f499196df37a8ce219abac6eb"}, + {file = "tokenizers-0.21.2-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:514cd43045c5d546f01142ff9c79a96ea69e4b5cda09e3027708cb2e6d5762ab"}, + {file = "tokenizers-0.21.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b1b9405822527ec1e0f7d8d2fdb287a5730c3a6518189c968254a8441b21faae"}, + {file = "tokenizers-0.21.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fed9a4d51c395103ad24f8e7eb976811c57fbec2af9f133df471afcd922e5020"}, + {file = "tokenizers-0.21.2-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:2c41862df3d873665ec78b6be36fcc30a26e3d4902e9dd8608ed61d49a48bc19"}, + {file = "tokenizers-0.21.2-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:ed21dc7e624e4220e21758b2e62893be7101453525e3d23264081c9ef9a6d00d"}, + {file = "tokenizers-0.21.2-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:0e73770507e65a0e0e2a1affd6b03c36e3bc4377bd10c9ccf51a82c77c0fe365"}, + {file = "tokenizers-0.21.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:106746e8aa9014a12109e58d540ad5465b4c183768ea96c03cbc24c44d329958"}, + {file = "tokenizers-0.21.2-cp39-abi3-win32.whl", hash = "sha256:cabda5a6d15d620b6dfe711e1af52205266d05b379ea85a8a301b3593c60e962"}, + {file = "tokenizers-0.21.2-cp39-abi3-win_amd64.whl", hash = "sha256:58747bb898acdb1007f37a7bbe614346e98dc28708ffb66a3fd50ce169ac6c98"}, + {file = "tokenizers-0.21.2.tar.gz", hash = "sha256:fdc7cffde3e2113ba0e6cc7318c40e3438a4d74bbc62bf04bcc63bdfb082ac77"}, ] [package.dependencies] @@ -4592,6 +5340,7 @@ files = [ {file = "torch-2.7.1-cp39-cp39-win_amd64.whl", hash = "sha256:df41989d9300e6e3c19ec9f56f856187a6ef060c3662fe54f4b6baf1fc90bd19"}, {file = "torch-2.7.1-cp39-none-macosx_11_0_arm64.whl", hash = "sha256:a737b5edd1c44a5c1ece2e9f3d00df9d1b3fb9541138bee56d83d38293fb6c9d"}, ] +markers = {main = "extra == \"all\""} [package.dependencies] filelock = "*" @@ -4645,14 +5394,14 @@ telegram = ["requests"] [[package]] name = "transformers" -version = "4.51.3" +version = "4.53.2" description = "State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow" optional = false python-versions = ">=3.9.0" groups = ["main", "eval"] files = [ - {file = "transformers-4.51.3-py3-none-any.whl", hash = "sha256:fd3279633ceb2b777013234bbf0b4f5c2d23c4626b05497691f00cfda55e8a83"}, - {file = "transformers-4.51.3.tar.gz", hash = "sha256:e292fcab3990c6defe6328f0f7d2004283ca81a7a07b2de9a46d67fd81ea1409"}, + {file = "transformers-4.53.2-py3-none-any.whl", hash = "sha256:db8f4819bb34f000029c73c3c557e7d06fc1b8e612ec142eecdae3947a9c78bf"}, + {file = "transformers-4.53.2.tar.gz", hash = "sha256:6c3ed95edfb1cba71c4245758f1b4878c93bf8cde77d076307dacb2cbbd72be2"}, ] [package.dependencies] @@ -4669,30 +5418,30 @@ tqdm = ">=4.27" [package.extras] accelerate = ["accelerate (>=0.26.0)"] -agents = ["Pillow (>=10.0.1,<=15.0)", "accelerate (>=0.26.0)", "datasets (!=2.5.0)", "diffusers", "opencv-python", "sentencepiece (>=0.1.91,!=0.1.92)", "torch (>=2.0)"] -all = ["Pillow (>=10.0.1,<=15.0)", "accelerate (>=0.26.0)", "av", "codecarbon (>=2.8.1)", "flax (>=0.4.1,<=0.7.0)", "jax (>=0.4.1,<=0.4.13)", "jaxlib (>=0.4.1,<=0.4.13)", "kenlm", "keras-nlp (>=0.3.1,<0.14.0)", "kernels (>=0.3.2,<0.4)", "librosa", "num2words", "onnxconverter-common", "optax (>=0.0.8,<=0.1.4)", "optuna", "phonemizer", "protobuf", "pyctcdecode (>=0.4.0)", "ray[tune] (>=2.7.0)", "scipy (<1.13.0)", "sentencepiece (>=0.1.91,!=0.1.92)", "sigopt", "tensorflow (>2.9,<2.16)", "tensorflow-text (<2.16)", "tf2onnx", "timm (<=1.0.11)", "tokenizers (>=0.21,<0.22)", "torch (>=2.0)", "torchaudio", "torchvision"] +all = ["Pillow (>=10.0.1,<=15.0)", "Pillow (>=10.0.1,<=15.0)", "accelerate (>=0.26.0)", "accelerate (>=0.26.0)", "av", "codecarbon (>=2.8.1)", "flax (>=0.4.1,<=0.7.0)", "jax (>=0.4.1,<=0.4.13)", "jaxlib (>=0.4.1,<=0.4.13)", "kenlm", "keras-nlp (>=0.3.1,<0.14.0)", "kernels (>=0.6.1,<0.7)", "librosa", "num2words", "onnxconverter-common", "optax (>=0.0.8,<=0.1.4)", "optuna", "phonemizer", "protobuf", "pyctcdecode (>=0.4.0)", "ray[tune] (>=2.7.0)", "scipy (<1.13.0)", "sentencepiece (>=0.1.91,!=0.1.92)", "sigopt", "tensorflow (>2.9,<2.16)", "tensorflow-text (<2.16)", "tf2onnx", "timm (<=1.0.11)", "tokenizers (>=0.21,<0.22)", "torch (>=2.1)", "torchaudio", "torchvision"] audio = ["kenlm", "librosa", "phonemizer", "pyctcdecode (>=0.4.0)"] benchmark = ["optimum-benchmark (>=0.3.0)"] codecarbon = ["codecarbon (>=2.8.1)"] deepspeed = ["accelerate (>=0.26.0)", "deepspeed (>=0.9.3)"] -deepspeed-testing = ["GitPython (<3.1.19)", "accelerate (>=0.26.0)", "beautifulsoup4", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "deepspeed (>=0.9.3)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "nltk (<=3.8.1)", "optuna", "parameterized", "protobuf", "psutil", "pydantic", "pytest (>=7.2.0,<8.0.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "sentencepiece (>=0.1.91,!=0.1.92)", "tensorboard", "timeout-decorator"] -dev = ["GitPython (<3.1.19)", "Pillow (>=10.0.1,<=15.0)", "accelerate (>=0.26.0)", "av", "beautifulsoup4", "codecarbon (>=2.8.1)", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "flax (>=0.4.1,<=0.7.0)", "fugashi (>=1.0)", "ipadic (>=1.0.0,<2.0)", "isort (>=5.5.4)", "jax (>=0.4.1,<=0.4.13)", "jaxlib (>=0.4.1,<=0.4.13)", "kenlm", "keras-nlp (>=0.3.1,<0.14.0)", "kernels (>=0.3.2,<0.4)", "libcst", "librosa", "nltk (<=3.8.1)", "num2words", "onnxconverter-common", "optax (>=0.0.8,<=0.1.4)", "optuna", "parameterized", "phonemizer", "protobuf", "psutil", "pyctcdecode (>=0.4.0)", "pydantic", "pytest (>=7.2.0,<8.0.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "ray[tune] (>=2.7.0)", "rhoknp (>=1.1.0,<1.3.1)", "rich", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "scikit-learn", "scipy (<1.13.0)", "sentencepiece (>=0.1.91,!=0.1.92)", "sigopt", "sudachidict-core (>=20220729)", "sudachipy (>=0.6.6)", "tensorboard", "tensorflow (>2.9,<2.16)", "tensorflow-text (<2.16)", "tf2onnx", "timeout-decorator", "timm (<=1.0.11)", "tokenizers (>=0.21,<0.22)", "torch (>=2.0)", "torchaudio", "torchvision", "unidic (>=1.0.2)", "unidic-lite (>=1.0.7)", "urllib3 (<2.0.0)"] -dev-tensorflow = ["GitPython (<3.1.19)", "Pillow (>=10.0.1,<=15.0)", "beautifulsoup4", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "isort (>=5.5.4)", "kenlm", "keras-nlp (>=0.3.1,<0.14.0)", "libcst", "librosa", "nltk (<=3.8.1)", "onnxconverter-common", "onnxruntime (>=1.4.0)", "onnxruntime-tools (>=1.4.2)", "parameterized", "phonemizer", "protobuf", "psutil", "pyctcdecode (>=0.4.0)", "pydantic", "pytest (>=7.2.0,<8.0.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "rich", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "scikit-learn", "sentencepiece (>=0.1.91,!=0.1.92)", "tensorboard", "tensorflow (>2.9,<2.16)", "tensorflow-text (<2.16)", "tf2onnx", "timeout-decorator", "tokenizers (>=0.21,<0.22)", "urllib3 (<2.0.0)"] -dev-torch = ["GitPython (<3.1.19)", "Pillow (>=10.0.1,<=15.0)", "accelerate (>=0.26.0)", "beautifulsoup4", "codecarbon (>=2.8.1)", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "fugashi (>=1.0)", "ipadic (>=1.0.0,<2.0)", "isort (>=5.5.4)", "kenlm", "kernels (>=0.3.2,<0.4)", "libcst", "librosa", "nltk (<=3.8.1)", "num2words", "onnxruntime (>=1.4.0)", "onnxruntime-tools (>=1.4.2)", "optuna", "parameterized", "phonemizer", "protobuf", "psutil", "pyctcdecode (>=0.4.0)", "pydantic", "pytest (>=7.2.0,<8.0.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "ray[tune] (>=2.7.0)", "rhoknp (>=1.1.0,<1.3.1)", "rich", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "scikit-learn", "sentencepiece (>=0.1.91,!=0.1.92)", "sigopt", "sudachidict-core (>=20220729)", "sudachipy (>=0.6.6)", "tensorboard", "timeout-decorator", "timm (<=1.0.11)", "tokenizers (>=0.21,<0.22)", "torch (>=2.0)", "torchaudio", "torchvision", "unidic (>=1.0.2)", "unidic-lite (>=1.0.7)", "urllib3 (<2.0.0)"] +deepspeed-testing = ["GitPython (<3.1.19)", "accelerate (>=0.26.0)", "beautifulsoup4", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "datasets (!=2.5.0)", "deepspeed (>=0.9.3)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "nltk (<=3.8.1)", "optuna", "parameterized", "protobuf", "psutil", "pydantic", "pytest (>=7.2.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "sentencepiece (>=0.1.91,!=0.1.92)", "sentencepiece (>=0.1.91,!=0.1.92)", "tensorboard", "timeout-decorator"] +dev = ["GitPython (<3.1.19)", "GitPython (<3.1.19)", "Pillow (>=10.0.1,<=15.0)", "Pillow (>=10.0.1,<=15.0)", "accelerate (>=0.26.0)", "accelerate (>=0.26.0)", "av", "beautifulsoup4", "codecarbon (>=2.8.1)", "cookiecutter (==1.7.3)", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "datasets (!=2.5.0)", "datasets (!=2.5.0)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "flax (>=0.4.1,<=0.7.0)", "fugashi (>=1.0)", "ipadic (>=1.0.0,<2.0)", "jax (>=0.4.1,<=0.4.13)", "jaxlib (>=0.4.1,<=0.4.13)", "kenlm", "keras-nlp (>=0.3.1,<0.14.0)", "kernels (>=0.6.1,<0.7)", "libcst", "librosa", "nltk (<=3.8.1)", "num2words", "onnxconverter-common", "optax (>=0.0.8,<=0.1.4)", "optuna", "pandas (<2.3.0)", "parameterized", "phonemizer", "protobuf", "psutil", "pyctcdecode (>=0.4.0)", "pydantic", "pytest (>=7.2.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "ray[tune] (>=2.7.0)", "rhoknp (>=1.1.0,<1.3.1)", "rich", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "scikit-learn", "scipy (<1.13.0)", "sentencepiece (>=0.1.91,!=0.1.92)", "sentencepiece (>=0.1.91,!=0.1.92)", "sigopt", "sudachidict_core (>=20220729)", "sudachipy (>=0.6.6)", "tensorboard", "tensorflow (>2.9,<2.16)", "tensorflow-text (<2.16)", "tf2onnx", "timeout-decorator", "timm (<=1.0.11)", "tokenizers (>=0.21,<0.22)", "torch (>=2.1)", "torchaudio", "torchvision", "unidic (>=1.0.2)", "unidic_lite (>=1.0.7)", "urllib3 (<2.0.0)"] +dev-tensorflow = ["GitPython (<3.1.19)", "GitPython (<3.1.19)", "Pillow (>=10.0.1,<=15.0)", "beautifulsoup4", "cookiecutter (==1.7.3)", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "datasets (!=2.5.0)", "datasets (!=2.5.0)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "kenlm", "keras-nlp (>=0.3.1,<0.14.0)", "libcst", "librosa", "nltk (<=3.8.1)", "onnxconverter-common", "onnxconverter-common", "onnxruntime (>=1.4.0)", "onnxruntime-tools (>=1.4.2)", "pandas (<2.3.0)", "parameterized", "phonemizer", "protobuf", "psutil", "pyctcdecode (>=0.4.0)", "pydantic", "pytest (>=7.2.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "rich", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "scikit-learn", "sentencepiece (>=0.1.91,!=0.1.92)", "sentencepiece (>=0.1.91,!=0.1.92)", "tensorboard", "tensorflow (>2.9,<2.16)", "tensorflow-text (<2.16)", "tf2onnx", "tf2onnx", "timeout-decorator", "tokenizers (>=0.21,<0.22)", "urllib3 (<2.0.0)"] +dev-torch = ["GitPython (<3.1.19)", "GitPython (<3.1.19)", "Pillow (>=10.0.1,<=15.0)", "Pillow (>=10.0.1,<=15.0)", "accelerate (>=0.26.0)", "beautifulsoup4", "codecarbon (>=2.8.1)", "cookiecutter (==1.7.3)", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "datasets (!=2.5.0)", "datasets (!=2.5.0)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "fugashi (>=1.0)", "ipadic (>=1.0.0,<2.0)", "kenlm", "kernels (>=0.6.1,<0.7)", "libcst", "librosa", "nltk (<=3.8.1)", "num2words", "onnxruntime (>=1.4.0)", "onnxruntime-tools (>=1.4.2)", "optuna", "pandas (<2.3.0)", "parameterized", "phonemizer", "protobuf", "psutil", "pyctcdecode (>=0.4.0)", "pydantic", "pytest (>=7.2.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "ray[tune] (>=2.7.0)", "rhoknp (>=1.1.0,<1.3.1)", "rich", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "scikit-learn", "sentencepiece (>=0.1.91,!=0.1.92)", "sentencepiece (>=0.1.91,!=0.1.92)", "sigopt", "sudachidict_core (>=20220729)", "sudachipy (>=0.6.6)", "tensorboard", "timeout-decorator", "timm (<=1.0.11)", "tokenizers (>=0.21,<0.22)", "torch (>=2.1)", "torchaudio", "torchvision", "unidic (>=1.0.2)", "unidic_lite (>=1.0.7)", "urllib3 (<2.0.0)"] flax = ["flax (>=0.4.1,<=0.7.0)", "jax (>=0.4.1,<=0.4.13)", "jaxlib (>=0.4.1,<=0.4.13)", "optax (>=0.0.8,<=0.1.4)", "scipy (<1.13.0)"] flax-speech = ["kenlm", "librosa", "phonemizer", "pyctcdecode (>=0.4.0)"] ftfy = ["ftfy"] -hf-xet = ["hf-xet"] -hub-kernels = ["kernels (>=0.3.2,<0.4)"] -integrations = ["kernels (>=0.3.2,<0.4)", "optuna", "ray[tune] (>=2.7.0)", "sigopt"] -ja = ["fugashi (>=1.0)", "ipadic (>=1.0.0,<2.0)", "rhoknp (>=1.1.0,<1.3.1)", "sudachidict-core (>=20220729)", "sudachipy (>=0.6.6)", "unidic (>=1.0.2)", "unidic-lite (>=1.0.7)"] +hf-xet = ["hf_xet"] +hub-kernels = ["kernels (>=0.6.1,<0.7)"] +integrations = ["kernels (>=0.6.1,<0.7)", "optuna", "ray[tune] (>=2.7.0)", "sigopt"] +ja = ["fugashi (>=1.0)", "ipadic (>=1.0.0,<2.0)", "rhoknp (>=1.1.0,<1.3.1)", "sudachidict_core (>=20220729)", "sudachipy (>=0.6.6)", "unidic (>=1.0.2)", "unidic_lite (>=1.0.7)"] modelcreation = ["cookiecutter (==1.7.3)"] natten = ["natten (>=0.14.6,<0.15.0)"] num2words = ["num2words"] onnx = ["onnxconverter-common", "onnxruntime (>=1.4.0)", "onnxruntime-tools (>=1.4.2)", "tf2onnx"] onnxruntime = ["onnxruntime (>=1.4.0)", "onnxruntime-tools (>=1.4.2)"] +open-telemetry = ["opentelemetry-api", "opentelemetry-exporter-otlp", "opentelemetry-sdk"] optuna = ["optuna"] -quality = ["GitPython (<3.1.19)", "datasets (!=2.5.0)", "isort (>=5.5.4)", "libcst", "rich", "ruff (==0.11.2)", "urllib3 (<2.0.0)"] +quality = ["GitPython (<3.1.19)", "datasets (!=2.5.0)", "libcst", "pandas (<2.3.0)", "rich", "ruff (==0.11.2)", "urllib3 (<2.0.0)"] ray = ["ray[tune] (>=2.7.0)"] retrieval = ["datasets (!=2.5.0)", "faiss-cpu"] ruff = ["ruff (==0.11.2)"] @@ -4702,17 +5451,17 @@ serving = ["fastapi", "pydantic", "starlette", "uvicorn"] sigopt = ["sigopt"] sklearn = ["scikit-learn"] speech = ["kenlm", "librosa", "phonemizer", "pyctcdecode (>=0.4.0)", "torchaudio"] -testing = ["GitPython (<3.1.19)", "beautifulsoup4", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "nltk (<=3.8.1)", "parameterized", "psutil", "pydantic", "pytest (>=7.2.0,<8.0.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "sentencepiece (>=0.1.91,!=0.1.92)", "tensorboard", "timeout-decorator"] +testing = ["GitPython (<3.1.19)", "beautifulsoup4", "cookiecutter (==1.7.3)", "datasets (!=2.5.0)", "datasets (!=2.5.0)", "dill (<0.3.5)", "evaluate (>=0.2.0)", "faiss-cpu", "nltk (<=3.8.1)", "parameterized", "psutil", "pydantic", "pytest (>=7.2.0)", "pytest-asyncio", "pytest-order", "pytest-rerunfailures", "pytest-rich", "pytest-timeout", "pytest-xdist", "rjieba", "rouge-score (!=0.0.7,!=0.0.8,!=0.1,!=0.1.1)", "ruff (==0.11.2)", "sacrebleu (>=1.4.12,<2.0.0)", "sacremoses", "sentencepiece (>=0.1.91,!=0.1.92)", "tensorboard", "timeout-decorator"] tf = ["keras-nlp (>=0.3.1,<0.14.0)", "onnxconverter-common", "tensorflow (>2.9,<2.16)", "tensorflow-text (<2.16)", "tf2onnx"] tf-cpu = ["keras (>2.9,<2.16)", "keras-nlp (>=0.3.1,<0.14.0)", "onnxconverter-common", "tensorflow-cpu (>2.9,<2.16)", "tensorflow-probability (<0.24)", "tensorflow-text (<2.16)", "tf2onnx"] tf-speech = ["kenlm", "librosa", "phonemizer", "pyctcdecode (>=0.4.0)"] tiktoken = ["blobfile", "tiktoken"] timm = ["timm (<=1.0.11)"] tokenizers = ["tokenizers (>=0.21,<0.22)"] -torch = ["accelerate (>=0.26.0)", "torch (>=2.0)"] +torch = ["accelerate (>=0.26.0)", "torch (>=2.1)"] torch-speech = ["kenlm", "librosa", "phonemizer", "pyctcdecode (>=0.4.0)", "torchaudio"] torch-vision = ["Pillow (>=10.0.1,<=15.0)", "torchvision"] -torchhub = ["filelock", "huggingface-hub (>=0.30.0,<1.0)", "importlib-metadata", "numpy (>=1.17)", "packaging (>=20.0)", "protobuf", "regex (!=2019.12.17)", "requests", "sentencepiece (>=0.1.91,!=0.1.92)", "tokenizers (>=0.21,<0.22)", "torch (>=2.0)", "tqdm (>=4.27)"] +torchhub = ["filelock", "huggingface-hub (>=0.30.0,<1.0)", "importlib_metadata", "numpy (>=1.17)", "packaging (>=20.0)", "protobuf", "regex (!=2019.12.17)", "requests", "sentencepiece (>=0.1.91,!=0.1.92)", "tokenizers (>=0.21,<0.22)", "torch (>=2.1)", "tqdm (>=4.27)"] video = ["av"] vision = ["Pillow (>=10.0.1,<=15.0)"] @@ -4723,7 +5472,6 @@ description = "A language and compiler for custom Deep Learning operations" optional = false python-versions = "*" groups = ["main", "eval"] -markers = "platform_machine == \"x86_64\" and platform_system == \"Linux\"" files = [ {file = "triton-3.3.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b74db445b1c562844d3cfad6e9679c72e93fdfb1a90a24052b03bb5c49d1242e"}, {file = "triton-3.3.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b31e3aa26f8cb3cc5bf4e187bf737cbacf17311e1112b781d4a059353dfd731b"}, @@ -4732,6 +5480,7 @@ files = [ {file = "triton-3.3.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a3198adb9d78b77818a5388bff89fa72ff36f9da0bc689db2f0a651a67ce6a42"}, {file = "triton-3.3.1-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f6139aeb04a146b0b8e0fbbd89ad1e65861c57cfed881f21d62d3cb94a36bab7"}, ] +markers = {main = "platform_machine == \"x86_64\" and extra == \"all\" and platform_system == \"Linux\"", eval = "platform_machine == \"x86_64\" and platform_system == \"Linux\""} [package.dependencies] setuptools = ">=40.8.0" @@ -4778,26 +5527,27 @@ typing-extensions = ">=3.7.4.3" [[package]] name = "typing-extensions" -version = "4.13.2" -description = "Backported and Experimental Type Hints for Python 3.8+" +version = "4.14.1" +description = "Backported and Experimental Type Hints for Python 3.9+" optional = false -python-versions = ">=3.8" -groups = ["main", "eval"] +python-versions = ">=3.9" +groups = ["main", "eval", "test"] files = [ - {file = "typing_extensions-4.13.2-py3-none-any.whl", hash = "sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c"}, - {file = "typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef"}, + {file = "typing_extensions-4.14.1-py3-none-any.whl", hash = "sha256:d1e1e3b58374dc93031d6eda2420a48ea44a36c2b4766a4fdeb3710755731d76"}, + {file = "typing_extensions-4.14.1.tar.gz", hash = "sha256:38b39f4aeeab64884ce9f74c94263ef78f3c22467c8724005483154c26648d36"}, ] +markers = {test = "python_version == \"3.10\""} [[package]] name = "typing-inspection" -version = "0.4.0" +version = "0.4.1" description = "Runtime typing introspection tools" optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "typing_inspection-0.4.0-py3-none-any.whl", hash = "sha256:50e72559fcd2a6367a19f7a7e610e6afcb9fac940c650290eed893d61386832f"}, - {file = "typing_inspection-0.4.0.tar.gz", hash = "sha256:9765c87de36671694a67904bf2c96e395be9c6439bb6c87b5142569dcdd65122"}, + {file = "typing_inspection-0.4.1-py3-none-any.whl", hash = "sha256:389055682238f53b04f7badcb49b989835495a96700ced5dab2d8feae4b26f51"}, + {file = "typing_inspection-0.4.1.tar.gz", hash = "sha256:6ae134cc0203c33377d43188d4064e9b357dba58cff3185f22924610e70a9d28"}, ] [package.dependencies] @@ -4814,6 +5564,7 @@ files = [ {file = "tzdata-2025.2-py2.py3-none-any.whl", hash = "sha256:1a403fada01ff9221ca8044d701868fa132215d84beb92242d9acd2147f667a8"}, {file = "tzdata-2025.2.tar.gz", hash = "sha256:b60a638fcc0daffadf82fe0f57e53d06bdec2f36c4df66280ae79bce6bd6f2b9"}, ] +markers = {main = "extra == \"mem-reader\" or extra == \"all\""} [[package]] name = "ujson" @@ -4905,14 +5656,14 @@ files = [ [[package]] name = "urllib3" -version = "2.4.0" +version = "2.5.0" description = "HTTP library with thread-safe connection pooling, file post, and more." optional = false python-versions = ">=3.9" groups = ["main", "eval"] files = [ - {file = "urllib3-2.4.0-py3-none-any.whl", hash = "sha256:4e16665048960a0900c702d4a66415956a584919c03361cac9f1df5c5dd7e813"}, - {file = "urllib3-2.4.0.tar.gz", hash = "sha256:414bc6535b787febd7567804cc015fee39daab8ad86268f1310a9250697de466"}, + {file = "urllib3-2.5.0-py3-none-any.whl", hash = "sha256:e6b01673c0fa6a13e374b50871808eb3bf7046c4b125b216f6bf1cc604cff0dc"}, + {file = "urllib3-2.5.0.tar.gz", hash = "sha256:3fc47733c7e419d4bc3f6b3dc2b4f890bb743906a30d56ba4a5bfa4bbff92760"}, ] [package.extras] @@ -4923,14 +5674,14 @@ zstd = ["zstandard (>=0.18.0)"] [[package]] name = "uvicorn" -version = "0.34.3" +version = "0.35.0" description = "The lightning-fast ASGI server." optional = false python-versions = ">=3.9" groups = ["main"] files = [ - {file = "uvicorn-0.34.3-py3-none-any.whl", hash = "sha256:16246631db62bdfbf069b0645177d6e8a77ba950cfedbfd093acef9444e4d885"}, - {file = "uvicorn-0.34.3.tar.gz", hash = "sha256:35919a9a979d7a59334b6b10e05d77c1d0d574c50e0fc98b8b1a0f165708b55a"}, + {file = "uvicorn-0.35.0-py3-none-any.whl", hash = "sha256:197535216b25ff9b785e29a0b79199f55222193d47f820816e7da751e9bc8d4a"}, + {file = "uvicorn-0.35.0.tar.gz", hash = "sha256:bc662f087f7cf2ce11a1d7fd70b90c9f98ef2e2831556dd078d131b96cc94a01"}, ] [package.dependencies] @@ -5022,6 +5773,27 @@ platformdirs = ">=3.9.1,<5" docs = ["furo (>=2023.7.26)", "proselint (>=0.13)", "sphinx (>=7.1.2,!=7.3)", "sphinx-argparse (>=0.4)", "sphinxcontrib-towncrier (>=0.2.1a0)", "towncrier (>=23.6)"] test = ["covdefaults (>=2.3)", "coverage (>=7.2.7)", "coverage-enable-subprocess (>=1)", "flaky (>=3.7)", "packaging (>=23.1)", "pytest (>=7.4)", "pytest-env (>=0.8.2)", "pytest-freezer (>=0.4.8) ; platform_python_implementation == \"PyPy\" or platform_python_implementation == \"GraalVM\" or platform_python_implementation == \"CPython\" and sys_platform == \"win32\" and python_version >= \"3.13\"", "pytest-mock (>=3.11.1)", "pytest-randomly (>=3.12)", "pytest-timeout (>=2.1)", "setuptools (>=68)", "time-machine (>=2.10) ; platform_python_implementation == \"CPython\""] +[[package]] +name = "volcengine-python-sdk" +version = "4.0.6" +description = "Volcengine SDK for Python" +optional = true +python-versions = "*" +groups = ["main"] +markers = "extra == \"all\"" +files = [ + {file = "volcengine-python-sdk-4.0.6.tar.gz", hash = "sha256:6367a892f10759c96133a31508aa32fd761b09f877d2efce00bf74b4eabb832f"}, +] + +[package.dependencies] +certifi = ">=2017.4.17" +python-dateutil = ">=2.1" +six = ">=1.10" +urllib3 = ">=1.23" + +[package.extras] +ark = ["anyio (>=3.5.0,<5)", "cached-property ; python_version < \"3.8\"", "cryptography (>=42.0.0)", "httpx (>=0.23.0,<1)", "pydantic (>=1.9.0,<3)"] + [[package]] name = "watchfiles" version = "1.1.0" @@ -5222,14 +5994,15 @@ files = [ [[package]] name = "xlrd" -version = "2.0.1" +version = "2.0.2" description = "Library for developers to extract data from Microsoft Excel (tm) .xls spreadsheet files" -optional = false -python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*" +optional = true +python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ - {file = "xlrd-2.0.1-py2.py3-none-any.whl", hash = "sha256:6a33ee89877bd9abc1158129f6e94be74e2679636b8a205b43b85206c3f0bbdd"}, - {file = "xlrd-2.0.1.tar.gz", hash = "sha256:f72f148f54442c6b056bf931dbc34f986fd0c3b0b6b5a58d013c9aef274d0c88"}, + {file = "xlrd-2.0.2-py2.py3-none-any.whl", hash = "sha256:ea762c3d29f4cca48d82df517b6d89fbce4db3107f9d78713e48cd321d5c9aa9"}, + {file = "xlrd-2.0.2.tar.gz", hash = "sha256:08b5e25de58f21ce71dc7db3b3b8106c1fa776f3024c54e45b45b374e89234c9"}, ] [package.extras] @@ -5239,14 +6012,15 @@ test = ["pytest", "pytest-cov"] [[package]] name = "xlsxwriter" -version = "3.2.3" +version = "3.2.5" description = "A Python module for creating Excel XLSX files." -optional = false -python-versions = ">=3.6" +optional = true +python-versions = ">=3.8" groups = ["main"] +markers = "extra == \"mem-reader\" or extra == \"all\"" files = [ - {file = "XlsxWriter-3.2.3-py3-none-any.whl", hash = "sha256:593f8296e8a91790c6d0378ab08b064f34a642b3feb787cf6738236bd0a4860d"}, - {file = "xlsxwriter-3.2.3.tar.gz", hash = "sha256:ad6fd41bdcf1b885876b1f6b7087560aecc9ae5a9cc2ba97dcac7ab2e210d3d5"}, + {file = "xlsxwriter-3.2.5-py3-none-any.whl", hash = "sha256:4f4824234e1eaf9d95df9a8fe974585ff91d0f5e3d3f12ace5b71e443c1c6abd"}, + {file = "xlsxwriter-3.2.5.tar.gz", hash = "sha256:7e88469d607cdc920151c0ab3ce9cf1a83992d4b7bc730c5ffdd1a12115a7dbe"}, ] [[package]] @@ -5384,14 +6158,14 @@ files = [ [[package]] name = "zep-cloud" -version = "2.15.0" +version = "2.19.0" description = "" optional = false python-versions = "<4.0,>=3.9.0" groups = ["eval"] files = [ - {file = "zep_cloud-2.15.0-py3-none-any.whl", hash = "sha256:14bd590e9d05ece9a7bf04e5f324ac255e8f6d42d9c45d38cd84e204d8e77793"}, - {file = "zep_cloud-2.15.0.tar.gz", hash = "sha256:bb9cf591fa2cd65178d2c0b54b3f8af7edf09f538343c3b4587184740886df54"}, + {file = "zep_cloud-2.19.0-py3-none-any.whl", hash = "sha256:5372fa74502976e558ce89b7034a47c601a0fba5b1c9d3a2adb1452fd73b5e9a"}, + {file = "zep_cloud-2.19.0.tar.gz", hash = "sha256:9683d8347acf21723434bfc6568602b5627928419cb5a9e3695fa6b2839d58c7"}, ] [package.dependencies] @@ -5512,7 +6286,13 @@ cffi = {version = ">=1.11", markers = "platform_python_implementation == \"PyPy\ [package.extras] cffi = ["cffi (>=1.11)"] +[extras] +all = ["chonkie", "markitdown", "neo4j", "pika", "qdrant-client", "redis", "schedule", "sentence-transformers", "torch", "volcengine-python-sdk"] +mem-reader = ["chonkie", "markitdown"] +mem-scheduler = ["pika", "redis"] +tree-mem = ["neo4j", "schedule"] + [metadata] lock-version = "2.1" -python-versions = "^3.10" -content-hash = "b56786e7bcdae03bbe0f031e2fe0bee284fef76534a3d3bfb24d6f70a44f4f0d" +python-versions = ">=3.10,<4.0" +content-hash = "d3a7dda79ef4954155ed988794a560458db3c410d280a88cabafb469f6bf3613" diff --git a/pyproject.toml b/pyproject.toml index e67acea0d..63bc78020 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,46 +1,146 @@ -# Poetry related +[project] +############################################################################## +# Here define the project metadata and dependencies for the MemoryOS package. +############################################################################## -[tool.poetry] name = "MemoryOS" -version = "0.2.0" +version = "0.2.1" description = "Intelligence Begins with Memory" -license = "Apache-2.0" -authors = ["MemTensor "] +license = {text = "Apache-2.0"} readme = "README.md" +requires-python = ">=3.10" +authors = [ + {name = "MemTensor", email = "MemTensor@memtensor.cn"} +] +keywords = [ + "memory", + "llm", + "language model", + "memoryOS", + "agent", + "kv cache", + "lora", +] +classifiers = [ + "Intended Audience :: Developers", + "License :: OSI Approved :: Apache Software License", + "Natural Language :: English", + "Natural Language :: Chinese (Simplified)", + "Operating System :: OS Independent", + "Programming Language :: Python :: 3 :: Only", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Topic :: Software Development :: Libraries", + "Topic :: Software Development :: Libraries :: Python Modules", +] +dependencies = [ + "openai (>=1.77.0,<2.0.0)", + "ollama (>=0.4.8,<0.5.0)", + "transformers (>=4.51.3,<5.0.0)", + "tenacity (>=9.1.2,<10.0.0)", # Error handling and retrying library + "fastapi[all] (>=0.115.12,<0.116.0)", # Web framework for building APIs + "sqlalchemy (>=2.0.41,<3.0.0)", # SQL toolkit + "scikit-learn (>=1.7.0,<2.0.0)", # Machine learning + "fastmcp (>=2.10.5,<3.0.0)", +] + +[project.urls] +homepage = "https://memos.openmem.net/" repository = "https://github.com/MemTensor/MemOS" -keywords = ["memory", "llm", "language model", "memoryOS", "agent"] -packages = [{include = "memos", from = "src"}] +download = "https://pypi.org/project/MemoryOS/#files" +changelog = "https://github.com/MemTensor/MemOS/releases" +releasenotes = "https://github.com/MemTensor/MemOS/releases" +documentation = "https://memos-docs.openmem.net/home/overview/" +issues = "https://github.com/MemTensor/MemOS/issues" + +[project.scripts] +memos = "memos.cli:main" + +[project.optional-dependencies] +# These are optional dependencies for various features of MemoryOS. +# Developers install: `poetry install --extras `. e.g., `poetry install --extras general-mem` +# Users install: `pip install MemoryOS[]`. e.g., `pip install MemoryOS[general-mem]` + +# TreeTextualMemory +tree-mem = [ + "neo4j (>=5.28.1,<6.0.0)", # Graph database + "schedule (>=1.2.2,<2.0.0)", # Task scheduling +] -[tool.poetry.dependencies] -python = "^3.10" -openai = "^1.77.0" -ollama = "^0.4.8" -qdrant-client = "^1.14.2" -transformers = "^4.51.3" -markitdown = {extras = ["docx", "pdf", "pptx", "xls", "xlsx"], version = "^0.1.1"} -chonkie = "^1.0.7" -tenacity = "^9.1.2" -neo4j = "^5.28.1" -accelerate = "^1.7.0" -fastapi = {extras = ["all"], version = "^0.115.12"} -sentence-transformers = "^4.1.0" -sqlalchemy = "^2.0.41" -redis = "^6.2.0" -schedule = "^1.2.2" +# MemScheduler +mem-scheduler = [ + "redis (>=6.2.0,<7.0.0)", # Key-value store + "pika (>=1.3.2,<2.0.0)", # RabbitMQ client +] + +# MemReader +mem-reader = [ + "chonkie (>=1.0.7,<2.0.0)", # Sentence chunking library + "markitdown[docx,pdf,pptx,xls,xlsx] (>=0.1.1,<0.2.0)", # Markdown parser for various file formats +] + +# All optional dependencies +# Allow users to install with `pip install MemoryOS[all]` +all = [ + # Exist in the above optional groups + "neo4j (>=5.28.1,<6.0.0)", + "schedule (>=1.2.2,<2.0.0)", + "redis (>=6.2.0,<7.0.0)", + "pika (>=1.3.2,<2.0.0)", + "chonkie (>=1.0.7,<2.0.0)", + "markitdown[docx,pdf,pptx,xls,xlsx] (>=0.1.1,<0.2.0)", + + # NOT exist in the above optional groups + # Because they are either huge-size dependencies or infrequently used dependencies. + # We kindof don't want users to install them. + "torch (>=2.7.1,<3.0.0)", + "sentence-transformers (>=4.1.0,<5.0.0)", + "qdrant-client (>=1.14.2,<2.0.0)", + "volcengine-python-sdk (>=4.0.4,<5.0.0)", + + # Uncategorized dependencies +] + + +[build-system] +############################################################################## +# Python package build system requirements. +############################################################################## + +requires = ["poetry-core"] +build-backend = "poetry.core.masonry.api" + + +[tool.poetry] +############################################################################## +# Here mainly define dependencies for development, testing, and evaluation. +# These dependencies will NOT be included in the MemoryOS package itself. +# They will be installed when you run `poetry install --with dev,test,eval`. +# +# More about version specifiers (e.g. "^0.1.0" or ">=0.1.0,<0.2.0"): +# https://python-poetry.org/docs/dependency-specification#caret-requirements +############################################################################## + +packages = [{include = "memos", from = "src"}] +requires-poetry = ">=2.0" +dependencies = { "python" = ">=3.10,<4.0" } [tool.poetry.group.dev] -optional = false +optional = true [tool.poetry.group.dev.dependencies] pre-commit = "^4.2.0" -ruff = "^0.11.8" + [tool.poetry.group.test] -optional = false +optional = true [tool.poetry.group.test.dependencies] pytest = "^8.3.5" pytest-asyncio = "^0.23.5" +ruff = "^0.11.8" [tool.poetry.group.eval] optional = true @@ -63,16 +163,11 @@ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" priority = "supplemental" -# Building related - -[build-system] -requires = ["poetry-core"] -build-backend = "poetry.core.masonry.api" - - -# Testing related - [tool.pytest.ini_options] +############################################################################## +# PyTest settings for running tests/ +############################################################################## + asyncio_mode = "auto" pythonpath = "src" filterwarnings = [ @@ -80,15 +175,15 @@ filterwarnings = [ ] -# Linting related - [tool.ruff] +############################################################################## +# Ruff is a fast Python linter and formatter. +############################################################################## + fix = true line-length = 100 target-version = "py310" - -[tool.ruff.lint] -extend-select = [ +lint.extend-select = [ "B", # flake8-bugbear "C4", # flake8-comprehensions "ERA", # flake8-eradicate/eradicate @@ -98,15 +193,13 @@ extend-select = [ "PGH", # pygrep "RUF", # ruff checks "SIM", # flake8-simplify - "TCH", # flake8-type-checking + "TC", # flake8-type-checking "TID", # flake8-tidy-imports "UP", # pyupgrade ] -ignore = [ +lint.ignore = [ "RUF001", # ambiguous-unicode-character-string "PGH003", # blanket-type-ignore ] - -[tool.ruff.lint.isort] -lines-between-types = 1 -lines-after-imports = 2 +lint.isort.lines-between-types = 1 +lint.isort.lines-after-imports = 2 diff --git a/scripts/check_dependencies.py b/scripts/check_dependencies.py new file mode 100644 index 000000000..f25d42555 --- /dev/null +++ b/scripts/check_dependencies.py @@ -0,0 +1,80 @@ +import ast +import importlib +import sys + +from pathlib import Path + + +EXCLUDE_MODULES = {"memos"} # Exclude from import checks (e.g., our own package) +PYTHON_PACKAGE_DIR = Path("src/memos") + + +def extract_top_level_modules(tree: ast.Module) -> set[str]: + """ + Extract all top-level imported modules (excluding relative imports). + """ + modules = set() + for node in tree.body: + if isinstance(node, ast.Import): + # Collect absolute imports only + for alias in node.names: + modules.add(alias.name.split(".")[0]) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + modules.add(node.module.split(".")[0]) + return modules + + +def check_importable(modules: set[str], filename: str) -> list[str]: + """ + Attempt to import each module in the current environment. + Return a list of modules that fail to import. + """ + failed = [] + for mod in sorted(modules): + if mod in EXCLUDE_MODULES: + # Skip excluded modules such as your own package + continue + try: + importlib.import_module(mod) + except ModuleNotFoundError: + failed.append(mod) + except Exception as e: + print( + f"⚠️ Warning: Importing module '{mod}' from {filename} raised unexpected error: {e}" + ) + return failed + + +def main(): + py_files = list(PYTHON_PACKAGE_DIR.rglob("*.py")) + + has_error = False + + for py_file in py_files: + try: + source = py_file.read_text(encoding="utf-8") + tree = ast.parse(source, filename=str(py_file)) + except SyntaxError as e: + print(f"❌ Syntax error in {py_file}: {e}") + has_error = True + continue + + modules = extract_top_level_modules(tree) + failed_imports = check_importable(modules, str(py_file)) + + for mod in failed_imports: + print(f"❌ {py_file}: Top-level import of unavailable module '{mod}'") + + if failed_imports: + has_error = True + + if has_error: + print( + "\n💥 Top-level imports failed. These modules may not be main dependencies." + " Try moving the imports to a function or class scope, and decorate it with @require_python_package." + ) + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/scripts/export_openapi.py b/scripts/export_openapi.py deleted file mode 100644 index 818b16466..000000000 --- a/scripts/export_openapi.py +++ /dev/null @@ -1,16 +0,0 @@ -import argparse -import json - -from memos.api.start_api import app - - -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Export OpenAPI schema to JSON file.") - parser.add_argument( - "--output", type=str, default="docs/openapi.json", help="Output path for OpenAPI schema." - ) - args = parser.parse_args() - with open(args.output, "w") as f: - json.dump(app.openapi(), f, indent=2) - f.write("\n") - print("Export completed successfully") diff --git a/src/memos/__init__.py b/src/memos/__init__.py index 74657b679..e8195c9d6 100644 --- a/src/memos/__init__.py +++ b/src/memos/__init__.py @@ -1,4 +1,4 @@ -__version__ = "0.2.0" +__version__ = "0.2.1" from memos.configs.mem_cube import GeneralMemCubeConfig from memos.configs.mem_os import MOSConfig diff --git a/src/memos/api/config.py b/src/memos/api/config.py new file mode 100644 index 000000000..75bec5569 --- /dev/null +++ b/src/memos/api/config.py @@ -0,0 +1,471 @@ +import os + +from typing import Any + +from dotenv import load_dotenv + +from memos.configs.mem_cube import GeneralMemCubeConfig +from memos.configs.mem_os import MOSConfig +from memos.mem_cube.general import GeneralMemCube + + +# Load environment variables +load_dotenv() + + +class APIConfig: + """Centralized configuration management for MemOS APIs.""" + + @staticmethod + def get_openai_config() -> dict[str, Any]: + """Get OpenAI configuration.""" + return { + "model_name_or_path": os.getenv("MOS_OPENAI_MODEL", "gpt-4o-mini"), + "temperature": float(os.getenv("MOS_CHAT_TEMPERATURE", "0.8")), + "max_tokens": int(os.getenv("MOS_MAX_TOKENS", "1024")), + "top_p": float(os.getenv("MOS_TOP_P", "0.9")), + "top_k": int(os.getenv("MOS_TOP_K", "50")), + "remove_think_prefix": True, + "api_key": os.getenv("OPENAI_API_KEY", "your-api-key-here"), + "api_base": os.getenv("OPENAI_API_BASE", "https://api.openai.com/v1"), + } + + @staticmethod + def qwen_config() -> dict[str, Any]: + """Get Qwen configuration.""" + return { + "model_name_or_path": os.getenv("MOS_CHAT_MODEL", "Qwen/Qwen3-1.7B"), + "temperature": float(os.getenv("MOS_CHAT_TEMPERATURE", "0.8")), + "max_tokens": int(os.getenv("MOS_MAX_TOKENS", "4096")), + "remove_think_prefix": True, + } + + @staticmethod + def vllm_config() -> dict[str, Any]: + """Get Qwen configuration.""" + return { + "model_name_or_path": os.getenv("MOS_CHAT_MODEL", "Qwen/Qwen3-1.7B"), + "temperature": float(os.getenv("MOS_CHAT_TEMPERATURE", "0.8")), + "max_tokens": int(os.getenv("MOS_MAX_TOKENS", "4096")), + "remove_think_prefix": True, + "api_key": os.getenv("VLLM_API_KEY", ""), + "api_base": os.getenv("VLLM_API_BASE", "http://localhost:8088/v1"), + "model_schema": os.getenv("MOS_MODEL_SCHEMA", "memos.configs.llm.VLLMLLMConfig"), + } + + @staticmethod + def get_activation_config() -> dict[str, Any]: + """Get Ollama configuration.""" + return { + "backend": "kv_cache", + "config": { + "memory_filename": "activation_memory.pickle", + "extractor_llm": { + "backend": "huggingface_singleton", + "config": { + "model_name_or_path": os.getenv("MOS_CHAT_MODEL", "Qwen/Qwen3-1.7B"), + "temperature": 0.8, + "max_tokens": 1024, + "top_p": 0.9, + "top_k": 50, + "add_generation_prompt": True, + "remove_think_prefix": False, + }, + }, + }, + } + + @staticmethod + def get_activation_vllm_config() -> dict[str, Any]: + """Get Ollama configuration.""" + return { + "backend": "vllm_kv_cache", + "config": { + "memory_filename": "activation_memory.pickle", + "extractor_llm": { + "backend": "vllm", + "config": APIConfig.vllm_config(), + }, + }, + } + + @staticmethod + def get_embedder_config() -> dict[str, Any]: + """Get embedder configuration.""" + embedder_backend = os.getenv("MOS_EMBEDDER_BACKEND", "ollama") + + if embedder_backend == "universal_api": + return { + "backend": "universal_api", + "config": { + "provider": os.getenv("MOS_EMBEDDER_PROVIDER", "openai"), + "api_key": os.getenv("OPENAI_API_KEY", "sk-xxxx"), + "model_name_or_path": os.getenv("MOS_EMBEDDER_MODEL", "text-embedding-3-large"), + "base_url": os.getenv("OPENAI_API_BASE", "http://openai.com"), + }, + } + else: # ollama + return { + "backend": "ollama", + "config": { + "model_name_or_path": os.getenv( + "MOS_EMBEDDER_MODEL", "nomic-embed-text:latest" + ), + "api_base": os.getenv("OLLAMA_API_BASE", "http://localhost:11434"), + }, + } + + @staticmethod + def get_neo4j_community_config(user_id: str | None = None) -> dict[str, Any]: + """Get Neo4j community configuration.""" + return { + "uri": os.getenv("NEO4J_URI", "bolt://localhost:7687"), + "user": os.getenv("NEO4J_USER", "neo4j"), + "db_name": os.getenv("NEO4J_DB_NAME", "shared-tree-textual-memory"), + "password": os.getenv("NEO4J_PASSWORD", "12345678"), + "user_name": f"memos{user_id.replace('-', '')}", + "auto_create": True, + "use_multi_db": False, + "embedding_dimension": 3072, + "vec_config": { + # Pass nested config to initialize external vector DB + # If you use qdrant, please use Server instead of local mode. + "backend": "qdrant", + "config": { + "collection_name": "neo4j_vec_db", + "vector_dimension": 3072, + "distance_metric": "cosine", + "host": "localhost", + "port": 6333, + }, + }, + } + + @staticmethod + def get_neo4j_config(user_id: str | None = None) -> dict[str, Any]: + """Get Neo4j configuration.""" + if os.getenv("MOS_NEO4J_SHARED_DB", "false").lower() == "true": + return APIConfig.get_neo4j_shared_config(user_id) + else: + return APIConfig.get_noshared_neo4j_config(user_id) + + @staticmethod + def get_noshared_neo4j_config(user_id) -> dict[str, Any]: + """Get Neo4j configuration.""" + return { + "uri": os.getenv("NEO4J_URI", "bolt://localhost:7687"), + "user": os.getenv("NEO4J_USER", "neo4j"), + "db_name": f"memos{user_id.replace('-', '')}", + "password": os.getenv("NEO4J_PASSWORD", "12345678"), + "auto_create": True, + "use_multi_db": True, + "embedding_dimension": 3072, + } + + @staticmethod + def get_neo4j_shared_config(user_id: str | None = None) -> dict[str, Any]: + """Get Neo4j configuration.""" + return { + "uri": os.getenv("NEO4J_URI", "bolt://localhost:7687"), + "user": os.getenv("NEO4J_USER", "neo4j"), + "db_name": os.getenv("NEO4J_DB_NAME", "shared-tree-textual-memory"), + "password": os.getenv("NEO4J_PASSWORD", "12345678"), + "user_name": f"memos{user_id.replace('-', '')}", + "auto_create": True, + "use_multi_db": False, + "embedding_dimension": 3072, + } + + @staticmethod + def get_scheduler_config() -> dict[str, Any]: + """Get scheduler configuration.""" + return { + "backend": "general_scheduler", + "config": { + "top_k": int(os.getenv("MOS_SCHEDULER_TOP_K", "10")), + "top_n": int(os.getenv("MOS_SCHEDULER_TOP_N", "5")), + "act_mem_update_interval": int( + os.getenv("MOS_SCHEDULER_ACT_MEM_UPDATE_INTERVAL", "300") + ), + "context_window_size": int(os.getenv("MOS_SCHEDULER_CONTEXT_WINDOW_SIZE", "5")), + "thread_pool_max_workers": int( + os.getenv("MOS_SCHEDULER_THREAD_POOL_MAX_WORKERS", "10") + ), + "consume_interval_seconds": int( + os.getenv("MOS_SCHEDULER_CONSUME_INTERVAL_SECONDS", "3") + ), + "enable_parallel_dispatch": os.getenv( + "MOS_SCHEDULER_ENABLE_PARALLEL_DISPATCH", "true" + ).lower() + == "true", + "enable_act_memory_update": True, + }, + } + + @staticmethod + def is_scheduler_enabled() -> bool: + """Check if scheduler is enabled via environment variable.""" + return os.getenv("MOS_ENABLE_SCHEDULER", "false").lower() == "true" + + @staticmethod + def is_default_cube_config_enabled() -> bool: + """Check if default cube config is enabled via environment variable.""" + return os.getenv("MOS_ENABLE_DEFAULT_CUBE_CONFIG", "false").lower() == "true" + + @staticmethod + def get_product_default_config() -> dict[str, Any]: + """Get default configuration for Product API.""" + openai_config = APIConfig.get_openai_config() + qwen_config = APIConfig.qwen_config() + vllm_config = APIConfig.vllm_config() + backend_model = { + "openai": openai_config, + "huggingface": qwen_config, + "vllm": vllm_config, + } + backend = os.getenv("MOS_CHAT_MODEL_PROVIDER", "openai") + config = { + "user_id": os.getenv("MOS_USER_ID", "root"), + "chat_model": {"backend": backend, "config": backend_model[backend]}, + "mem_reader": { + "backend": "simple_struct", + "config": { + "llm": { + "backend": "openai", + "config": openai_config, + }, + "embedder": APIConfig.get_embedder_config(), + "chunker": { + "backend": "sentence", + "config": { + "tokenizer_or_token_counter": "gpt2", + "chunk_size": 512, + "chunk_overlap": 128, + "min_sentences_per_chunk": 1, + }, + }, + }, + }, + "enable_textual_memory": True, + "enable_activation_memory": os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() + == "true", + "top_k": int(os.getenv("MOS_TOP_K", "50")), + "max_turns_window": int(os.getenv("MOS_MAX_TURNS_WINDOW", "20")), + } + + # Add scheduler configuration if enabled + if APIConfig.is_scheduler_enabled(): + config["mem_scheduler"] = APIConfig.get_scheduler_config() + config["enable_mem_scheduler"] = True + else: + config["enable_mem_scheduler"] = False + + return config + + @staticmethod + def get_start_default_config() -> dict[str, Any]: + """Get default configuration for Start API.""" + config = { + "user_id": os.getenv("MOS_USER_ID", "default_user"), + "session_id": os.getenv("MOS_SESSION_ID", "default_session"), + "enable_textual_memory": True, + "enable_activation_memory": os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() + == "true", + "top_k": int(os.getenv("MOS_TOP_K", "5")), + "chat_model": { + "backend": os.getenv("MOS_CHAT_MODEL_PROVIDER", "openai"), + "config": { + "model_name_or_path": os.getenv("MOS_CHAT_MODEL", "gpt-4o-mini"), + "api_key": os.getenv("OPENAI_API_KEY", "sk-xxxxxx"), + "temperature": float(os.getenv("MOS_CHAT_TEMPERATURE", 0.7)), + "api_base": os.getenv("OPENAI_API_BASE", "http://xxxxxx:3000/v1"), + "max_tokens": int(os.getenv("MOS_MAX_TOKENS", 1024)), + "top_p": float(os.getenv("MOS_TOP_P", 0.9)), + "top_k": int(os.getenv("MOS_TOP_K", 50)), + "remove_think_prefix": True, + }, + }, + } + + # Add scheduler configuration if enabled + if APIConfig.is_scheduler_enabled(): + config["mem_scheduler"] = APIConfig.get_scheduler_config() + config["enable_mem_scheduler"] = True + else: + config["enable_mem_scheduler"] = False + + return config + + @staticmethod + def create_user_config(user_name: str, user_id: str) -> tuple[MOSConfig, GeneralMemCube]: + """Create configuration for a specific user.""" + openai_config = APIConfig.get_openai_config() + + qwen_config = APIConfig.qwen_config() + vllm_config = APIConfig.vllm_config() + backend = os.getenv("MOS_CHAT_MODEL_PROVIDER", "openai") + backend_model = { + "openai": openai_config, + "huggingface": qwen_config, + "vllm": vllm_config, + } + # Create MOSConfig + config_dict = { + "user_id": user_id, + "chat_model": { + "backend": backend, + "config": backend_model[backend], + }, + "mem_reader": { + "backend": "simple_struct", + "config": { + "llm": { + "backend": "openai", + "config": openai_config, + }, + "embedder": APIConfig.get_embedder_config(), + "chunker": { + "backend": "sentence", + "config": { + "tokenizer_or_token_counter": "gpt2", + "chunk_size": 512, + "chunk_overlap": 128, + "min_sentences_per_chunk": 1, + }, + }, + }, + }, + "enable_textual_memory": True, + "enable_activation_memory": os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() + == "true", + "top_k": 30, + "max_turns_window": 20, + } + + # Add scheduler configuration if enabled + if APIConfig.is_scheduler_enabled(): + config_dict["mem_scheduler"] = APIConfig.get_scheduler_config() + config_dict["enable_mem_scheduler"] = True + else: + config_dict["enable_mem_scheduler"] = False + + default_config = MOSConfig(**config_dict) + + if os.getenv("NEO4J_BACKEND", "neo4j_community").lower() == "neo4j_community": + neo4j_community_config = APIConfig.get_neo4j_community_config(user_id) + # Create MemCube config + default_cube_config = GeneralMemCubeConfig.model_validate( + { + "user_id": user_id, + "cube_id": f"{user_name}_default_cube", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": {"backend": "openai", "config": openai_config}, + "dispatcher_llm": {"backend": "openai", "config": openai_config}, + "graph_db": { + "backend": "neo4j-community", + "config": neo4j_community_config, + }, + "embedder": APIConfig.get_embedder_config(), + }, + }, + "act_mem": {} + if os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() == "false" + else APIConfig.get_activation_vllm_config(), + "para_mem": {}, + } + ) + else: + neo4j_config = APIConfig.get_neo4j_config(user_id) + # Create MemCube config + default_cube_config = GeneralMemCubeConfig.model_validate( + { + "user_id": user_id, + "cube_id": f"{user_name}_default_cube", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": {"backend": "openai", "config": openai_config}, + "dispatcher_llm": {"backend": "openai", "config": openai_config}, + "graph_db": { + "backend": "neo4j", + "config": neo4j_config, + }, + "embedder": APIConfig.get_embedder_config(), + }, + }, + "act_mem": {} + if os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() == "false" + else APIConfig.get_activation_vllm_config(), + "para_mem": {}, + } + ) + + default_mem_cube = GeneralMemCube(default_cube_config) + return default_config, default_mem_cube + + @staticmethod + def get_default_cube_config() -> GeneralMemCubeConfig | None: + """Get default cube configuration for product initialization. + + Returns: + GeneralMemCubeConfig | None: Default cube configuration if enabled, None otherwise. + """ + if not APIConfig.is_default_cube_config_enabled(): + return None + + openai_config = APIConfig.get_openai_config() + + if os.getenv("NEO4J_BACKEND", "neo4j_community").lower() == "neo4j_community": + neo4j_community_config = APIConfig.get_neo4j_community_config(user_id="default") + return GeneralMemCubeConfig.model_validate( + { + "user_id": "default", + "cube_id": "default_cube", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": {"backend": "openai", "config": openai_config}, + "dispatcher_llm": {"backend": "openai", "config": openai_config}, + "graph_db": { + "backend": "neo4j-community", + "config": neo4j_community_config, + }, + "embedder": APIConfig.get_embedder_config(), + "reorganize": os.getenv("MOS_ENABLE_REORGANIZE", "false").lower() + == "true", + }, + }, + "act_mem": {} + if os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() == "false" + else APIConfig.get_activation_vllm_config(), + "para_mem": {}, + } + ) + else: + neo4j_config = APIConfig.get_neo4j_config(user_id="default") + return GeneralMemCubeConfig.model_validate( + { + "user_id": "default", + "cube_id": "default_cube", + "text_mem": { + "backend": "tree_text", + "config": { + "extractor_llm": {"backend": "openai", "config": openai_config}, + "dispatcher_llm": {"backend": "openai", "config": openai_config}, + "graph_db": { + "backend": "neo4j", + "config": neo4j_config, + }, + "embedder": APIConfig.get_embedder_config(), + "reorganize": os.getenv("MOS_ENABLE_REORGANIZE", "false").lower() + == "true", + }, + }, + "act_mem": {} + if os.getenv("ENABLE_ACTIVATION_MEMORY", "false").lower() == "false" + else APIConfig.get_activation_vllm_config(), + "para_mem": {}, + } + ) diff --git a/src/memos/api/exceptions.py b/src/memos/api/exceptions.py new file mode 100644 index 000000000..2fd22ad52 --- /dev/null +++ b/src/memos/api/exceptions.py @@ -0,0 +1,28 @@ +import logging + +from fastapi.requests import Request +from fastapi.responses import JSONResponse + + +logger = logging.getLogger(__name__) + + +class APIExceptionHandler: + """Centralized exception handling for MemOS APIs.""" + + @staticmethod + async def value_error_handler(request: Request, exc: ValueError): + """Handle ValueError exceptions globally.""" + return JSONResponse( + status_code=400, + content={"code": 400, "message": str(exc), "data": None}, + ) + + @staticmethod + async def global_exception_handler(request: Request, exc: Exception): + """Handle all unhandled exceptions globally.""" + logger.exception("Unhandled error:") + return JSONResponse( + status_code=500, + content={"code": 500, "message": str(exc), "data": None}, + ) diff --git a/src/memos/api/mcp_serve.py b/src/memos/api/mcp_serve.py new file mode 100644 index 000000000..9eb1e59d0 --- /dev/null +++ b/src/memos/api/mcp_serve.py @@ -0,0 +1,502 @@ +import asyncio +import os + +from typing import Any + +from dotenv import load_dotenv +from fastmcp import FastMCP + +# Assuming these are your imports +from memos.mem_os.main import MOS +from memos.mem_os.utils.default_config import get_default +from memos.mem_user.user_manager import UserRole + + +load_dotenv() + + +def load_default_config(user_id="default_user"): + config, cube = get_default( + openai_api_key=os.getenv("OPENAI_API_KEY"), + openai_api_base=os.getenv("OPENAI_API_BASE"), + text_mem_type=os.getenv("MOS_TEXT_MEM_TYPE"), + user_id=user_id, + neo4j_uri=os.getenv("NEO4J_URI"), + neo4j_user=os.getenv("NEO4J_USER"), + neo4j_password=os.getenv("NEO4J_PASSWORD"), + ) + return config, cube + + +class MOSMCPStdioServer: + def __init__(self): + self.mcp = FastMCP("MOS Memory System") + config, cube = load_default_config() + self.mos_core = MOS(config=config) + self._setup_tools() + + def _setup_tools(self): + """Setup MCP tools""" + + @self.mcp.tool() + async def chat(query: str, user_id: str | None = None) -> str: + """ + Chat with MOS system using memory-enhanced responses. + + This method provides intelligent responses by searching through user's memory cubes + and incorporating relevant context. It supports both standard chat mode and enhanced + Chain of Thought (CoT) mode for complex queries when PRO_MODE is enabled. + + Args: + query (str): The user's query or question to be answered + user_id (str, optional): User ID for the chat session. If not provided, uses the default user + + Returns: + str: AI-generated response incorporating relevant memories and context + """ + try: + response = self.mos_core.chat(query, user_id) + return response + except Exception as e: + return f"Chat error: {e!s}" + + @self.mcp.tool() + async def create_user( + user_id: str, role: str = "USER", user_name: str | None = None + ) -> str: + """ + Create a new user in the MOS system. + + This method creates a new user account with specified role and name. + Users can have different access levels and can own or access memory cubes. + + Args: + user_id (str): Unique identifier for the user + role (str): User role - "USER" for regular users, "ADMIN" for administrators + user_name (str, optional): Display name for the user. If not provided, uses user_id + + Returns: + str: Success message with the created user ID + """ + try: + user_role = UserRole.ADMIN if role.upper() == "ADMIN" else UserRole.USER + created_user_id = self.mos_core.create_user(user_id, user_role, user_name) + return f"User created successfully: {created_user_id}" + except Exception as e: + return f"Error creating user: {e!s}" + + @self.mcp.tool() + async def create_cube( + cube_name: str, owner_id: str, cube_path: str | None = None, cube_id: str | None = None + ) -> str: + """ + Create a new memory cube for a user. + + Memory cubes are containers that store different types of memories (textual, activation, parametric). + Each cube can be owned by a user and shared with other users. + + Args: + cube_name (str): Human-readable name for the memory cube + owner_id (str): User ID of the cube owner who has full control + cube_path (str, optional): File system path where cube data will be stored + cube_id (str, optional): Custom unique identifier for the cube. If not provided, one will be generated + + Returns: + str: Success message with the created cube ID + """ + try: + created_cube_id = self.mos_core.create_cube_for_user( + cube_name, owner_id, cube_path, cube_id + ) + return f"Cube created successfully: {created_cube_id}" + except Exception as e: + return f"Error creating cube: {e!s}" + + @self.mcp.tool() + async def register_cube( + cube_name_or_path: str, cube_id: str | None = None, user_id: str | None = None + ) -> str: + """ + Register an existing memory cube with the MOS system. + + This method loads and registers a memory cube from a file path or creates a new one + if the path doesn't exist. The cube becomes available for memory operations. + + Args: + cube_name_or_path (str): File path to the memory cube or name for a new cube + cube_id (str, optional): Custom identifier for the cube. If not provided, one will be generated + user_id (str, optional): User ID to associate with the cube. If not provided, uses default user + + Returns: + str: Success message with the registered cube ID + """ + try: + if not os.path.exists(cube_name_or_path): + mos_config, cube_name_or_path = load_default_config(user_id=user_id) + self.mos_core.register_mem_cube( + cube_name_or_path, mem_cube_id=cube_id, user_id=user_id + ) + return f"Cube registered successfully: {cube_id or cube_name_or_path}" + except Exception as e: + return f"Error registering cube: {e!s}" + + @self.mcp.tool() + async def unregister_cube(cube_id: str, user_id: str | None = None) -> str: + """ + Unregister a memory cube from the MOS system. + + This method removes a memory cube from the active session, making it unavailable + for memory operations. The cube data remains intact on disk. + + Args: + cube_id (str): Unique identifier of the cube to unregister + user_id (str, optional): User ID for access validation. If not provided, uses default user + + Returns: + str: Success message confirming the cube was unregistered + """ + try: + self.mos_core.unregister_mem_cube(cube_id, user_id) + return f"Cube unregistered successfully: {cube_id}" + except Exception as e: + return f"Error unregistering cube: {e!s}" + + @self.mcp.tool() + async def search_memories( + query: str, user_id: str | None = None, cube_ids: list[str] | None = None + ) -> dict[str, Any]: + """ + Search for memories across user's accessible memory cubes. + + This method performs semantic search through textual memories stored in the specified + cubes, returning relevant memories based on the query. Results are ranked by relevance. + + Args: + query (str): Search query to find relevant memories + user_id (str, optional): User ID whose cubes to search. If not provided, uses default user + cube_ids (list[str], optional): Specific cube IDs to search. If not provided, searches all user's cubes + + Returns: + dict: Search results containing text_mem, act_mem, and para_mem categories with relevant memories + """ + try: + result = self.mos_core.search(query, user_id, cube_ids) + return result + except Exception as e: + return {"error": str(e)} + + @self.mcp.tool() + async def add_memory( + memory_content: str | None = None, + doc_path: str | None = None, + messages: list[dict[str, str]] | None = None, + cube_id: str | None = None, + user_id: str | None = None, + ) -> str: + """ + Add memories to a memory cube. + + This method can add memories from different sources: direct text content, document files, + or conversation messages. The memories are processed and stored in the specified cube. + + Args: + memory_content (str, optional): Direct text content to add as memory + doc_path (str, optional): Path to a document file to process and add as memories + messages (list[dict[str, str]], optional): List of conversation messages to add as memories + cube_id (str, optional): Target cube ID. If not provided, uses user's default cube + user_id (str, optional): User ID for access validation. If not provided, uses default user + + Returns: + str: Success message confirming memories were added + """ + try: + self.mos_core.add( + messages=messages, + memory_content=memory_content, + doc_path=doc_path, + mem_cube_id=cube_id, + user_id=user_id, + ) + return "Memory added successfully" + except Exception as e: + return f"Error adding memory: {e!s}" + + @self.mcp.tool() + async def get_memory( + cube_id: str, memory_id: str, user_id: str | None = None + ) -> dict[str, Any]: + """ + Retrieve a specific memory from a memory cube. + + This method fetches a single memory item by its unique identifier from the specified cube. + + Args: + cube_id (str): Unique identifier of the cube containing the memory + memory_id (str): Unique identifier of the specific memory to retrieve + user_id (str, optional): User ID for access validation. If not provided, uses default user + + Returns: + dict: Memory content with metadata including memory text, creation time, and source + """ + try: + memory = self.mos_core.get(cube_id, memory_id, user_id) + return {"memory": str(memory)} + except Exception as e: + return {"error": str(e)} + + @self.mcp.tool() + async def update_memory( + cube_id: str, memory_id: str, memory_content: str, user_id: str | None = None + ) -> str: + """ + Update an existing memory in a memory cube. + + This method modifies the content of a specific memory while preserving its metadata. + Note: Update functionality may not be supported by all memory backends (e.g., tree_text). + + Args: + cube_id (str): Unique identifier of the cube containing the memory + memory_id (str): Unique identifier of the memory to update + memory_content (str): New content to replace the existing memory + user_id (str, optional): User ID for access validation. If not provided, uses default user + + Returns: + str: Success message confirming the memory was updated + """ + try: + from memos.memories.textual.item import TextualMemoryItem, TextualMemoryMetadata + + metadata = TextualMemoryMetadata( + user_id=user_id or self.mos_core.user_id, + session_id=self.mos_core.session_id, + source="mcp_update", + ) + memory_item = TextualMemoryItem(memory=memory_content, metadata=metadata) + + self.mos_core.update(cube_id, memory_id, memory_item, user_id) + return f"Memory updated successfully: {memory_id}" + except Exception as e: + return f"Error updating memory: {e!s}" + + @self.mcp.tool() + async def delete_memory(cube_id: str, memory_id: str, user_id: str | None = None) -> str: + """ + Delete a specific memory from a memory cube. + + This method permanently removes a memory item from the specified cube. + The operation cannot be undone. + + Args: + cube_id (str): Unique identifier of the cube containing the memory + memory_id (str): Unique identifier of the memory to delete + user_id (str, optional): User ID for access validation. If not provided, uses default user + + Returns: + str: Success message confirming the memory was deleted + """ + try: + self.mos_core.delete(cube_id, memory_id, user_id) + return f"Memory deleted successfully: {memory_id}" + except Exception as e: + return f"Error deleting memory: {e!s}" + + @self.mcp.tool() + async def delete_all_memories(cube_id: str, user_id: str | None = None) -> str: + """ + Delete all memories from a memory cube. + + This method permanently removes all memory items from the specified cube. + The operation cannot be undone and will clear all textual memories. + + Args: + cube_id (str): Unique identifier of the cube to clear + user_id (str, optional): User ID for access validation. If not provided, uses default user + + Returns: + str: Success message confirming all memories were deleted + """ + try: + self.mos_core.delete_all(cube_id, user_id) + return f"All memories deleted successfully from cube: {cube_id}" + except Exception as e: + return f"Error deleting all memories: {e!s}" + + @self.mcp.tool() + async def clear_chat_history(user_id: str | None = None) -> str: + """ + Clear the chat history for a user. + + This method resets the conversation history, removing all previous messages + while keeping the memory cubes and stored memories intact. + + Args: + user_id (str, optional): User ID whose chat history to clear. If not provided, uses default user + + Returns: + str: Success message confirming chat history was cleared + """ + try: + self.mos_core.clear_messages(user_id) + target_user = user_id or self.mos_core.user_id + return f"Chat history cleared for user: {target_user}" + except Exception as e: + return f"Error clearing chat history: {e!s}" + + @self.mcp.tool() + async def dump_cube( + dump_dir: str, user_id: str | None = None, cube_id: str | None = None + ) -> str: + """ + Export a memory cube to a directory. + + This method creates a backup or export of a memory cube, including all memories + and metadata, to the specified directory for backup or migration purposes. + + Args: + dump_dir (str): Directory path where the cube data will be exported + user_id (str, optional): User ID for access validation. If not provided, uses default user + cube_id (str, optional): Cube ID to export. If not provided, uses user's default cube + + Returns: + str: Success message with the export directory path + """ + try: + self.mos_core.dump(dump_dir, user_id, cube_id) + return f"Cube dumped successfully to: {dump_dir}" + except Exception as e: + return f"Error dumping cube: {e!s}" + + @self.mcp.tool() + async def share_cube(cube_id: str, target_user_id: str) -> str: + """ + Share a memory cube with another user. + + This method grants access to a memory cube to another user, allowing them + to read and search through the memories stored in that cube. + + Args: + cube_id (str): Unique identifier of the cube to share + target_user_id (str): User ID of the person to share the cube with + + Returns: + str: Success message confirming the cube was shared or error message if failed + """ + try: + success = self.mos_core.share_cube_with_user(cube_id, target_user_id) + if success: + return f"Cube {cube_id} shared successfully with user {target_user_id}" + else: + return f"Failed to share cube {cube_id} with user {target_user_id}" + except Exception as e: + return f"Error sharing cube: {e!s}" + + @self.mcp.tool() + async def get_user_info(user_id: str | None = None) -> dict[str, Any]: + """ + Get detailed information about a user and their accessible memory cubes. + + This method returns comprehensive user information including profile details, + role, creation time, and a list of all memory cubes the user can access. + + Args: + user_id (str, optional): User ID to get information for. If not provided, uses current user + + Returns: + dict: User information including user_id, user_name, role, created_at, and accessible_cubes + """ + try: + if user_id and user_id != self.mos_core.user_id: + # Temporarily switch user + original_user = self.mos_core.user_id + self.mos_core.user_id = user_id + user_info = self.mos_core.get_user_info() + self.mos_core.user_id = original_user + return user_info + else: + return self.mos_core.get_user_info() + except Exception as e: + return {"error": str(e)} + + @self.mcp.tool() + async def control_memory_scheduler(action: str) -> str: + """ + Control the memory scheduler service. + + The memory scheduler is responsible for processing and organizing memories + in the background. This method allows starting or stopping the scheduler service. + + Args: + action (str): Action to perform - "start" to enable the scheduler, "stop" to disable it + + Returns: + str: Success message confirming the scheduler action or error message if failed + """ + try: + if action.lower() == "start": + success = self.mos_core.mem_scheduler_on() + return ( + "Memory scheduler started" + if success + else "Failed to start memory scheduler" + ) + elif action.lower() == "stop": + success = self.mos_core.mem_scheduler_off() + return ( + "Memory scheduler stopped" if success else "Failed to stop memory scheduler" + ) + else: + return "Invalid action. Use 'start' or 'stop'" + except Exception as e: + return f"Error controlling memory scheduler: {e!s}" + + def run(self, transport: str = "stdio", **kwargs): + """Run MCP server with specified transport""" + if transport == "stdio": + # Run stdio mode (default for local usage) + self.mcp.run(transport="stdio") + elif transport == "http": + # Run HTTP mode + host = kwargs.get("host", "localhost") + port = kwargs.get("port", 8000) + asyncio.run(self.mcp.run_http_async(host=host, port=port)) + elif transport == "sse": + # Run SSE mode (deprecated but still supported) + host = kwargs.get("host", "localhost") + port = kwargs.get("port", 8000) + self.mcp.run(transport="sse", host=host, port=port) + else: + raise ValueError(f"Unsupported transport: {transport}") + + +# Usage example +if __name__ == "__main__": + import argparse + + from dotenv import load_dotenv + + load_dotenv() + + # Parse command line arguments + parser = argparse.ArgumentParser(description="MOS MCP Server") + parser.add_argument( + "--transport", + choices=["stdio", "http", "sse"], + default="stdio", + help="Transport method (default: stdio)", + ) + parser.add_argument("--host", default="localhost", help="Host for HTTP/SSE transport") + parser.add_argument("--port", type=int, default=8000, help="Port for HTTP/SSE transport") + + args = parser.parse_args() + + # Set environment variables + os.environ["OPENAI_API_BASE"] = os.getenv("OPENAI_API_BASE") + os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY") + os.environ["MOS_TEXT_MEM_TYPE"] = "tree_text" # "tree_text" need set neo4j + os.environ["NEO4J_URI"] = os.getenv("NEO4J_URI") + os.environ["NEO4J_USER"] = os.getenv("NEO4J_USER") + os.environ["NEO4J_PASSWORD"] = os.getenv("NEO4J_PASSWORD") + + # Create and run MCP server + server = MOSMCPStdioServer() + server.run(transport=args.transport, host=args.host, port=args.port) diff --git a/src/memos/api/product_api.py b/src/memos/api/product_api.py new file mode 100644 index 000000000..d6a41af72 --- /dev/null +++ b/src/memos/api/product_api.py @@ -0,0 +1,35 @@ +import logging + +from fastapi import FastAPI + +from memos.api.exceptions import APIExceptionHandler +from memos.api.routers.product_router import router as product_router + + +# Configure logging +logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s") +logger = logging.getLogger(__name__) + +app = FastAPI( + title="MemOS Product REST APIs", + description="A REST API for managing multiple users with MemOS Product.", + version="1.0.0", +) + +# Include routers +app.include_router(product_router) + +# Exception handlers +app.exception_handler(ValueError)(APIExceptionHandler.value_error_handler) +app.exception_handler(Exception)(APIExceptionHandler.global_exception_handler) + + +if __name__ == "__main__": + import argparse + + import uvicorn + + parser = argparse.ArgumentParser() + parser.add_argument("--port", type=int, default=8001) + args = parser.parse_args() + uvicorn.run(app, host="0.0.0.0", port=args.port) diff --git a/src/memos/api/product_models.py b/src/memos/api/product_models.py new file mode 100644 index 000000000..0e9d5ff59 --- /dev/null +++ b/src/memos/api/product_models.py @@ -0,0 +1,159 @@ +import uuid + +from typing import Generic, Literal, TypeAlias, TypeVar + +from pydantic import BaseModel, Field +from typing_extensions import TypedDict + + +T = TypeVar("T") + + +# ─── Message Types ────────────────────────────────────────────────────────────── + +# Chat message roles +MessageRole: TypeAlias = Literal["user", "assistant", "system"] + + +# Message structure +class MessageDict(TypedDict): + """Typed dictionary for chat message dictionaries.""" + + role: MessageRole + content: str + + +class BaseRequest(BaseModel): + """Base model for all requests.""" + + +class BaseResponse(BaseModel, Generic[T]): + """Base model for all responses.""" + + code: int = Field(200, description="Response status code") + message: str = Field(..., description="Response message") + data: T | None = Field(None, description="Response data") + + +# Product API Models +class UserRegisterRequest(BaseRequest): + """Request model for user registration.""" + + user_id: str = Field( + default_factory=lambda: str(uuid.uuid4()), description="User ID for registration" + ) + user_name: str | None = Field(None, description="User name for registration") + interests: str | None = Field(None, description="User interests") + + +class GetMemoryRequest(BaseRequest): + """Request model for getting memories.""" + + user_id: str = Field(..., description="User ID") + memory_type: Literal["text_mem", "act_mem", "param_mem", "para_mem"] = Field( + ..., description="Memory type" + ) + mem_cube_ids: list[str] | None = Field(None, description="Cube IDs") + search_query: str | None = Field(None, description="Search query") + + +# Start API Models +class Message(BaseModel): + role: str = Field(..., description="Role of the message (user or assistant).") + content: str = Field(..., description="Message content.") + + +class MemoryCreate(BaseRequest): + user_id: str = Field(..., description="User ID") + messages: list[Message] | None = Field(None, description="List of messages to store.") + memory_content: str | None = Field(None, description="Content to store as memory") + doc_path: str | None = Field(None, description="Path to document to store") + mem_cube_id: str | None = Field(None, description="ID of the memory cube") + + +class MemCubeRegister(BaseRequest): + mem_cube_name_or_path: str = Field(..., description="Name or path of the MemCube to register.") + mem_cube_id: str | None = Field(None, description="ID for the MemCube") + + +class ChatRequest(BaseRequest): + """Request model for chat operations.""" + + user_id: str = Field(..., description="User ID") + query: str = Field(..., description="Chat query message") + mem_cube_id: str | None = Field(None, description="Cube ID to use for chat") + history: list[MessageDict] | None = Field(None, description="Chat history") + + +class UserCreate(BaseRequest): + user_name: str | None = Field(None, description="Name of the user") + role: str = Field("user", description="Role of the user") + user_id: str = Field(..., description="User ID") + + +class CubeShare(BaseRequest): + target_user_id: str = Field(..., description="Target user ID to share with") + + +# Response Models +class SimpleResponse(BaseResponse[None]): + """Simple response model for operations without data return.""" + + +class UserRegisterResponse(BaseResponse[dict]): + """Response model for user registration.""" + + +class MemoryResponse(BaseResponse[list]): + """Response model for memory operations.""" + + +class SuggestionResponse(BaseResponse[list]): + """Response model for suggestion operations.""" + + data: dict[str, list[str]] | None = Field(None, description="Response data") + + +class ConfigResponse(BaseResponse[None]): + """Response model for configuration endpoint.""" + + +class SearchResponse(BaseResponse[dict]): + """Response model for search operations.""" + + +class ChatResponse(BaseResponse[str]): + """Response model for chat operations.""" + + +class UserResponse(BaseResponse[dict]): + """Response model for user operations.""" + + +class UserListResponse(BaseResponse[list]): + """Response model for user list operations.""" + + +class MemoryCreateRequest(BaseRequest): + """Request model for creating memories.""" + + user_id: str = Field(..., description="User ID") + messages: list[MessageDict] | None = Field(None, description="List of messages to store.") + memory_content: str | None = Field(None, description="Memory content to store") + doc_path: str | None = Field(None, description="Path to document to store") + mem_cube_id: str | None = Field(None, description="Cube ID") + + +class SearchRequest(BaseRequest): + """Request model for searching memories.""" + + user_id: str = Field(..., description="User ID") + query: str = Field(..., description="Search query") + mem_cube_id: str | None = Field(None, description="Cube ID to search in") + + +class SuggestionRequest(BaseRequest): + """Request model for getting suggestion queries.""" + + user_id: str = Field(..., description="User ID") + language: Literal["zh", "en"] = Field("zh", description="Language for suggestions") diff --git a/src/memos/api/routers/__init__.py b/src/memos/api/routers/__init__.py new file mode 100644 index 000000000..40ed96f43 --- /dev/null +++ b/src/memos/api/routers/__init__.py @@ -0,0 +1 @@ +# API routers module diff --git a/src/memos/api/routers/product_router.py b/src/memos/api/routers/product_router.py new file mode 100644 index 000000000..92acb38a4 --- /dev/null +++ b/src/memos/api/routers/product_router.py @@ -0,0 +1,358 @@ +import json +import logging +import traceback + +from fastapi import APIRouter, HTTPException +from fastapi.responses import StreamingResponse + +from memos.api.config import APIConfig +from memos.api.product_models import ( + BaseResponse, + ChatRequest, + GetMemoryRequest, + MemoryCreateRequest, + MemoryResponse, + SearchRequest, + SearchResponse, + SimpleResponse, + SuggestionRequest, + SuggestionResponse, + UserRegisterRequest, + UserRegisterResponse, +) +from memos.configs.mem_os import MOSConfig +from memos.mem_os.product import MOSProduct + + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/product", tags=["Product API"]) + +# Initialize MOSProduct instance with lazy initialization +MOS_PRODUCT_INSTANCE = None + + +def get_mos_product_instance(): + """Get or create MOSProduct instance.""" + global MOS_PRODUCT_INSTANCE + if MOS_PRODUCT_INSTANCE is None: + default_config = APIConfig.get_product_default_config() + print(default_config) + from memos.configs.mem_os import MOSConfig + + mos_config = MOSConfig(**default_config) + + # Get default cube config from APIConfig (may be None if disabled) + default_cube_config = APIConfig.get_default_cube_config() + print("*********default_cube_config*********", default_cube_config) + MOS_PRODUCT_INSTANCE = MOSProduct( + default_config=mos_config, default_cube_config=default_cube_config + ) + logger.info("MOSProduct instance created successfully with inheritance architecture") + return MOS_PRODUCT_INSTANCE + + +get_mos_product_instance() + + +@router.post("/configure", summary="Configure MOSProduct", response_model=SimpleResponse) +async def set_config(config): + """Set MOSProduct configuration.""" + global MOS_PRODUCT_INSTANCE + MOS_PRODUCT_INSTANCE = MOSProduct(default_config=config) + return SimpleResponse(message="Configuration set successfully") + + +@router.post("/users/register", summary="Register a new user", response_model=UserRegisterResponse) +async def register_user(user_req: UserRegisterRequest): + """Register a new user with configuration and default cube.""" + try: + # Get configuration for the user + user_config, default_mem_cube = APIConfig.create_user_config( + user_name=user_req.user_id, user_id=user_req.user_id + ) + logger.info(f"user_config: {user_config.model_dump(mode='json')}") + logger.info(f"default_mem_cube: {default_mem_cube.config.model_dump(mode='json')}") + mos_product = get_mos_product_instance() + + # Register user with default config and mem cube + result = mos_product.user_register( + user_id=user_req.user_id, + user_name=user_req.user_name, + interests=user_req.interests, + config=user_config, + default_mem_cube=default_mem_cube, + ) + + if result["status"] == "success": + return UserRegisterResponse( + message="User registered successfully", + data={"user_id": result["user_id"], "mem_cube_id": result["default_cube_id"]}, + ) + else: + raise HTTPException(status_code=400, detail=result["message"]) + + except Exception as err: + logger.error(f"Failed to register user: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.get( + "/suggestions/{user_id}", summary="Get suggestion queries", response_model=SuggestionResponse +) +async def get_suggestion_queries(user_id: str): + """Get suggestion queries for a specific user.""" + try: + mos_product = get_mos_product_instance() + suggestions = mos_product.get_suggestion_query(user_id) + return SuggestionResponse( + message="Suggestions retrieved successfully", data={"query": suggestions} + ) + except ValueError as err: + raise HTTPException(status_code=404, detail=str(traceback.format_exc())) from err + except Exception as err: + logger.error(f"Failed to get suggestions: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.post( + "/suggestions", + summary="Get suggestion queries with language", + response_model=SuggestionResponse, +) +async def get_suggestion_queries_post(suggestion_req: SuggestionRequest): + """Get suggestion queries for a specific user with language preference.""" + try: + mos_product = get_mos_product_instance() + suggestions = mos_product.get_suggestion_query( + user_id=suggestion_req.user_id, language=suggestion_req.language + ) + return SuggestionResponse( + message="Suggestions retrieved successfully", data={"query": suggestions} + ) + except ValueError as err: + raise HTTPException(status_code=404, detail=str(traceback.format_exc())) from err + except Exception as err: + logger.error(f"Failed to get suggestions: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.post("/get_all", summary="Get all memories for user", response_model=MemoryResponse) +async def get_all_memories(memory_req: GetMemoryRequest): + """Get all memories for a specific user.""" + try: + mos_product = get_mos_product_instance() + if memory_req.search_query: + result = mos_product.get_subgraph( + user_id=memory_req.user_id, + query=memory_req.search_query, + mem_cube_ids=memory_req.mem_cube_ids, + ) + return MemoryResponse(message="Memories retrieved successfully", data=result) + else: + result = mos_product.get_all( + user_id=memory_req.user_id, + memory_type=memory_req.memory_type, + mem_cube_ids=memory_req.mem_cube_ids, + ) + return MemoryResponse(message="Memories retrieved successfully", data=result) + + except ValueError as err: + raise HTTPException(status_code=404, detail=str(traceback.format_exc())) from err + except Exception as err: + logger.error(f"Failed to get memories: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.post("/add", summary="add a new memory", response_model=SimpleResponse) +async def create_memory(memory_req: MemoryCreateRequest): + """Create a new memory for a specific user.""" + try: + mos_product = get_mos_product_instance() + mos_product.add( + user_id=memory_req.user_id, + memory_content=memory_req.memory_content, + messages=memory_req.messages, + doc_path=memory_req.doc_path, + mem_cube_id=memory_req.mem_cube_id, + ) + return SimpleResponse(message="Memory created successfully") + + except ValueError as err: + raise HTTPException(status_code=404, detail=str(traceback.format_exc())) from err + except Exception as err: + logger.error(f"Failed to create memory: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.post("/search", summary="Search memories", response_model=SearchResponse) +async def search_memories(search_req: SearchRequest): + """Search memories for a specific user.""" + try: + mos_product = get_mos_product_instance() + result = mos_product.search( + query=search_req.query, + user_id=search_req.user_id, + install_cube_ids=[search_req.mem_cube_id] if search_req.mem_cube_id else None, + ) + return SearchResponse(message="Search completed successfully", data=result) + + except ValueError as err: + raise HTTPException(status_code=404, detail=str(traceback.format_exc())) from err + except Exception as err: + logger.error(f"Failed to search memories: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.post("/chat", summary="Chat with MemOS") +async def chat(chat_req: ChatRequest): + """Chat with MemOS for a specific user. Returns SSE stream.""" + try: + mos_product = get_mos_product_instance() + + async def generate_chat_response(): + """Generate chat response as SSE stream.""" + try: + import asyncio + + for chunk in mos_product.chat_with_references( + query=chat_req.query, + user_id=chat_req.user_id, + cube_id=chat_req.mem_cube_id, + history=chat_req.history, + ): + yield chunk + await asyncio.sleep(0.00001) # 50ms delay between chunks + except Exception as e: + logger.error(f"Error in chat stream: {e}") + error_data = f"data: {json.dumps({'type': 'error', 'content': str(traceback.format_exc())})}\n\n" + yield error_data + + return StreamingResponse( + generate_chat_response(), + media_type="text/plain", + headers={ + "Cache-Control": "no-cache", + "Connection": "keep-alive", + "Content-Type": "text/event-stream", + }, + ) + + except ValueError as err: + raise HTTPException(status_code=404, detail=str(traceback.format_exc())) from err + except Exception as err: + logger.error(f"Failed to start chat: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.get("/users", summary="List all users", response_model=BaseResponse[list]) +async def list_users(): + """List all registered users.""" + try: + mos_product = get_mos_product_instance() + users = mos_product.list_users() + return BaseResponse(message="Users retrieved successfully", data=users) + except Exception as err: + logger.error(f"Failed to list users: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.get("/users/{user_id}", summary="Get user info", response_model=BaseResponse[dict]) +async def get_user_info(user_id: str): + """Get user information including accessible cubes.""" + try: + mos_product = get_mos_product_instance() + user_info = mos_product.get_user_info(user_id) + return BaseResponse(message="User info retrieved successfully", data=user_info) + except ValueError as err: + raise HTTPException(status_code=404, detail=str(traceback.format_exc())) from err + except Exception as err: + logger.error(f"Failed to get user info: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.get( + "/configure/{user_id}", summary="Get MOSProduct configuration", response_model=SimpleResponse +) +async def get_config(user_id: str): + """Get MOSProduct configuration.""" + global MOS_PRODUCT_INSTANCE + config = MOS_PRODUCT_INSTANCE.default_config + return SimpleResponse(message="Configuration retrieved successfully", data=config) + + +@router.get( + "/users/{user_id}/config", summary="Get user configuration", response_model=BaseResponse[dict] +) +async def get_user_config(user_id: str): + """Get user-specific configuration.""" + try: + mos_product = get_mos_product_instance() + config = mos_product.get_user_config(user_id) + if config: + return BaseResponse( + message="User configuration retrieved successfully", + data=config.model_dump(mode="json"), + ) + else: + raise HTTPException( + status_code=404, detail=f"Configuration not found for user {user_id}" + ) + except ValueError as err: + raise HTTPException(status_code=404, detail=str(traceback.format_exc())) from err + except Exception as err: + logger.error(f"Failed to get user config: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.put( + "/users/{user_id}/config", summary="Update user configuration", response_model=SimpleResponse +) +async def update_user_config(user_id: str, config_data: dict): + """Update user-specific configuration.""" + try: + mos_product = get_mos_product_instance() + + # Create MOSConfig from the provided data + config = MOSConfig(**config_data) + + # Update the configuration + success = mos_product.update_user_config(user_id, config) + if success: + return SimpleResponse(message="User configuration updated successfully") + else: + raise HTTPException(status_code=500, detail="Failed to update user configuration") + + except ValueError as err: + raise HTTPException(status_code=400, detail=str(traceback.format_exc())) from err + except Exception as err: + logger.error(f"Failed to update user config: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.get( + "/instances/status", summary="Get user configuration status", response_model=BaseResponse[dict] +) +async def get_instance_status(): + """Get information about active user configurations in memory.""" + try: + mos_product = get_mos_product_instance() + status_info = mos_product.get_user_instance_info() + return BaseResponse( + message="User configuration status retrieved successfully", data=status_info + ) + except Exception as err: + logger.error(f"Failed to get user configuration status: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err + + +@router.get("/instances/count", summary="Get active user count", response_model=BaseResponse[int]) +async def get_active_user_count(): + """Get the number of active user configurations in memory.""" + try: + mos_product = get_mos_product_instance() + count = mos_product.get_active_user_count() + return BaseResponse(message="Active user count retrieved successfully", data=count) + except Exception as err: + logger.error(f"Failed to get active user count: {traceback.format_exc()}") + raise HTTPException(status_code=500, detail=str(traceback.format_exc())) from err diff --git a/src/memos/chunkers/sentence_chunker.py b/src/memos/chunkers/sentence_chunker.py index d2073ab20..4de0cf32b 100644 --- a/src/memos/chunkers/sentence_chunker.py +++ b/src/memos/chunkers/sentence_chunker.py @@ -1,6 +1,5 @@ -from chonkie import SentenceChunker as ChonkieSentenceChunker - from memos.configs.chunker import SentenceChunkerConfig +from memos.dependency import require_python_package from memos.log import get_logger from .base import BaseChunker, Chunk @@ -12,7 +11,14 @@ class SentenceChunker(BaseChunker): """Sentence-based text chunker.""" + @require_python_package( + import_name="chonkie", + install_command="pip install chonkie", + install_link="https://docs.chonkie.ai/python-sdk/getting-started/installation", + ) def __init__(self, config: SentenceChunkerConfig): + from chonkie import SentenceChunker as ChonkieSentenceChunker + self.config = config self.chunker = ChonkieSentenceChunker( tokenizer_or_token_counter=config.tokenizer_or_token_counter, diff --git a/src/memos/cli.py b/src/memos/cli.py new file mode 100644 index 000000000..fb3475ff3 --- /dev/null +++ b/src/memos/cli.py @@ -0,0 +1,113 @@ +""" +MemOS CLI Tool +This script provides command-line interface for MemOS operations. +""" + +import argparse +import json +import os +import zipfile + +from io import BytesIO + + +def export_openapi(output: str) -> bool: + """Export OpenAPI schema to JSON file.""" + from memos.api.start_api import app + + # Create directory if it doesn't exist + if os.path.dirname(output): + os.makedirs(os.path.dirname(output), exist_ok=True) + + with open(output, "w") as f: + json.dump(app.openapi(), f, indent=2) + f.write("\n") + + print(f"✅ OpenAPI schema exported to: {output}") + return True + + +def download_examples(dest: str) -> bool: + import requests + + """Download examples from the MemOS repository.""" + zip_url = "https://github.com/MemTensor/MemOS/archive/refs/heads/main.zip" + print(f"📥 Downloading examples from {zip_url}...") + + try: + response = requests.get(zip_url) + response.raise_for_status() + + with zipfile.ZipFile(BytesIO(response.content)) as z: + extracted_files = [] + for file in z.namelist(): + if "MemOS-main/examples/" in file and not file.endswith("/"): + # Remove the prefix and extract to dest + relative_path = file.replace("MemOS-main/examples/", "") + extract_path = os.path.join(dest, relative_path) + + # Create directory if it doesn't exist + os.makedirs(os.path.dirname(extract_path), exist_ok=True) + + # Extract the file + with z.open(file) as source, open(extract_path, "wb") as target: + target.write(source.read()) + extracted_files.append(extract_path) + + print(f"✅ Examples downloaded to: {dest}") + print(f"📁 {len(extracted_files)} files extracted") + + except requests.RequestException as e: + print(f"❌ Error downloading examples: {e}") + return False + except Exception as e: + print(f"❌ Error extracting examples: {e}") + return False + + return True + + +def main(): + """Main CLI entry point.""" + parser = argparse.ArgumentParser( + prog="memos", + description="MemOS Command Line Interface", + ) + + # Create subparsers for different commands + subparsers = parser.add_subparsers(dest="command", help="Available commands") + + # Download examples command + examples_parser = subparsers.add_parser("download_examples", help="Download example files") + examples_parser.add_argument( + "--dest", + type=str, + default="./examples", + help="Destination directory for examples (default: ./examples)", + ) + + # Export API command + api_parser = subparsers.add_parser("export_openapi", help="Export OpenAPI schema to JSON file") + api_parser.add_argument( + "--output", + type=str, + default="openapi.json", + help="Output path for OpenAPI schema (default: openapi.json)", + ) + + # Parse arguments + args = parser.parse_args() + + # Handle commands + if args.command == "download_examples": + success = download_examples(args.dest) + exit(0 if success else 1) + elif args.command == "export_openapi": + success = export_openapi(args.output) + exit(0 if success else 1) + else: + parser.print_help() + + +if __name__ == "__main__": + main() diff --git a/src/memos/configs/embedder.py b/src/memos/configs/embedder.py index f143e5725..70095a194 100644 --- a/src/memos/configs/embedder.py +++ b/src/memos/configs/embedder.py @@ -18,6 +18,18 @@ class OllamaEmbedderConfig(BaseEmbedderConfig): api_base: str = Field(default="http://localhost:11434", description="Base URL for Ollama API") +class ArkEmbedderConfig(BaseEmbedderConfig): + api_key: str = Field(..., description="Ark API key") + api_base: str = Field( + default="https://ark.cn-beijing.volces.com/api/v3/", description="Base URL for Ark API" + ) + chunk_size: int = Field(default=1, description="Chunk size for Ark API") + multi_modal: bool = Field( + default=False, + description="Whether to use multi-modal embedding (text + image) with Ark", + ) + + class SenTranEmbedderConfig(BaseEmbedderConfig): """Configuration class for Sentence Transformer embeddings.""" @@ -27,6 +39,19 @@ class SenTranEmbedderConfig(BaseEmbedderConfig): ) +class UniversalAPIEmbedderConfig(BaseEmbedderConfig): + """ + Configuration class for universal API embedding providers, e.g., + OpenAI, etc. + """ + + provider: str = Field(..., description="Provider name, e.g., 'openai'") + api_key: str = Field(..., description="API key for the embedding provider") + base_url: str | None = Field( + default=None, description="Optional base URL for custom or proxied endpoint" + ) + + class EmbedderConfigFactory(BaseConfig): """Factory class for creating embedder configurations.""" @@ -36,6 +61,8 @@ class EmbedderConfigFactory(BaseConfig): backend_to_class: ClassVar[dict[str, Any]] = { "ollama": OllamaEmbedderConfig, "sentence_transformer": SenTranEmbedderConfig, + "ark": ArkEmbedderConfig, + "universal_api": UniversalAPIEmbedderConfig, } @field_validator("backend") diff --git a/src/memos/configs/graph_db.py b/src/memos/configs/graph_db.py index 786743726..cca93fede 100644 --- a/src/memos/configs/graph_db.py +++ b/src/memos/configs/graph_db.py @@ -3,6 +3,7 @@ from pydantic import BaseModel, Field, field_validator, model_validator from memos.configs.base import BaseConfig +from memos.configs.vec_db import VectorDBConfigFactory class BaseGraphDBConfig(BaseConfig): @@ -14,14 +15,93 @@ class BaseGraphDBConfig(BaseConfig): class Neo4jGraphDBConfig(BaseGraphDBConfig): - """Neo4j-specific configuration.""" + """ + Neo4j-specific configuration. + + This config supports: + 1) Physical isolation (multi-db) — each user gets a dedicated Neo4j database. + 2) Logical isolation (single-db) — all users share one or more databases, but each node is tagged with `user_name`. + + How to use: + - If `use_multi_db=True`, then `db_name` should usually be the same as `user_name`. + Each user gets a separate database for physical isolation. + Example: db_name = "alice", user_name = None or "alice". + + - If `use_multi_db=False`, then `db_name` is your shared database (e.g., "neo4j" or "shared_db"). + You must provide `user_name` to logically isolate each user's data. + All nodes and queries must respect this tag. + + Example configs: + --- + # Physical isolation: + db_name = "alice" + use_multi_db = True + user_name = None + + # Logical isolation: + db_name = "shared_db_student_group" + use_multi_db = False + user_name = "alice" + """ db_name: str = Field(..., description="The name of the target Neo4j database") auto_create: bool = Field( - default=False, description="Whether to create the DB if it doesn't exist" + default=False, + description="If True, automatically create the target db_name in multi-db mode if it does not exist.", ) + + use_multi_db: bool = Field( + default=True, + description=( + "If True: use Neo4j's multi-database feature for physical isolation; " + "each user typically gets a separate database. " + "If False: use a single shared database with logical isolation by user_name." + ), + ) + + user_name: str | None = Field( + default=None, + description=( + "Logical user or tenant ID for data isolation. " + "Required if use_multi_db is False. " + "All nodes must be tagged with this and all queries must filter by this." + ), + ) + embedding_dimension: int = Field(default=768, description="Dimension of vector embedding") + @model_validator(mode="after") + def validate_config(self): + """Validate logical constraints to avoid misconfiguration.""" + if not self.use_multi_db and not self.user_name: + raise ValueError( + "In single-database mode (use_multi_db=False), `user_name` must be provided for logical isolation." + ) + return self + + +class Neo4jCommunityGraphDBConfig(Neo4jGraphDBConfig): + """ + Community edition config for Neo4j. + + Notes: + - Must set `use_multi_db = False` + - Must provide `user_name` for logical isolation + - Embedding vector DB config is required + """ + + vec_config: VectorDBConfigFactory = Field( + ..., description="Vector DB config for embedding search" + ) + + @model_validator(mode="after") + def validate_community(self): + if self.use_multi_db: + raise ValueError("Neo4j Community Edition does not support use_multi_db=True.") + if not self.user_name: + raise ValueError("Neo4j Community config requires user_name for logical isolation.") + return self + class GraphDBConfigFactory(BaseModel): backend: str = Field(..., description="Backend for graph database") @@ -29,6 +109,7 @@ class GraphDBConfigFactory(BaseModel): backend_to_class: ClassVar[dict[str, Any]] = { "neo4j": Neo4jGraphDBConfig, + "neo4j-community": Neo4jCommunityGraphDBConfig, } @field_validator("backend") diff --git a/src/memos/configs/llm.py b/src/memos/configs/llm.py index e0b09b73f..d69a0a0fc 100644 --- a/src/memos/configs/llm.py +++ b/src/memos/configs/llm.py @@ -27,6 +27,40 @@ class OpenAILLMConfig(BaseLLMConfig): extra_body: Any = Field(default=None, description="extra body") +class QwenLLMConfig(BaseLLMConfig): + api_key: str = Field(..., description="API key for DashScope (Qwen)") + api_base: str = Field( + default="https://dashscope-intl.aliyuncs.com/compatible-mode/v1", + description="Base URL for Qwen OpenAI-compatible API", + ) + extra_body: Any = Field(default=None, description="extra body") + model_name_or_path: str = Field(..., description="Model name for Qwen, e.g., 'qwen-plus'") + + +class DeepSeekLLMConfig(BaseLLMConfig): + api_key: str = Field(..., description="API key for DeepSeek") + api_base: str = Field( + default="https://api.deepseek.com", + description="Base URL for DeepSeek OpenAI-compatible API", + ) + extra_body: Any = Field(default=None, description="Extra options for API") + model_name_or_path: str = Field( + ..., description="Model name: 'deepseek-chat' or 'deepseek-reasoner'" + ) + + +class AzureLLMConfig(BaseLLMConfig): + base_url: str = Field( + default="https://api.openai.azure.com/", + description="Base URL for Azure OpenAI API", + ) + api_version: str = Field( + default="2024-03-01-preview", + description="API version for Azure OpenAI", + ) + api_key: str = Field(..., description="API key for Azure OpenAI") + + class OllamaLLMConfig(BaseLLMConfig): api_base: str = Field( default="http://localhost:11434", @@ -45,6 +79,14 @@ class HFLLMConfig(BaseLLMConfig): ) +class VLLMLLMConfig(BaseLLMConfig): + api_key: str = Field(default="", description="API key for vLLM (optional for local server)") + api_base: str = Field( + default="http://localhost:8088/v1", + description="Base URL for vLLM API", + ) + + class LLMConfigFactory(BaseConfig): """Factory class for creating LLM configurations.""" @@ -54,7 +96,12 @@ class LLMConfigFactory(BaseConfig): backend_to_class: ClassVar[dict[str, Any]] = { "openai": OpenAILLMConfig, "ollama": OllamaLLMConfig, + "azure": AzureLLMConfig, "huggingface": HFLLMConfig, + "vllm": VLLMLLMConfig, + "huggingface_singleton": HFLLMConfig, # Add singleton support + "qwen": QwenLLMConfig, + "deepseek": DeepSeekLLMConfig, } @field_validator("backend") diff --git a/src/memos/configs/mem_cube.py b/src/memos/configs/mem_cube.py index d2d35afda..b9868fa99 100644 --- a/src/memos/configs/mem_cube.py +++ b/src/memos/configs/mem_cube.py @@ -70,7 +70,7 @@ def validate_text_mem(cls, text_mem: MemoryConfigFactory) -> MemoryConfigFactory @classmethod def validate_act_mem(cls, act_mem: MemoryConfigFactory) -> MemoryConfigFactory: """Validate the act_mem field.""" - allowed_backends = ["kv_cache", "uninitialized"] + allowed_backends = ["kv_cache", "vllm_kv_cache", "uninitialized"] if act_mem.backend not in allowed_backends: raise ConfigurationError( f"GeneralMemCubeConfig requires act_mem backend to be one of {allowed_backends}, got '{act_mem.backend}'" diff --git a/src/memos/configs/mem_scheduler.py b/src/memos/configs/mem_scheduler.py index 6aff65188..83799de74 100644 --- a/src/memos/configs/mem_scheduler.py +++ b/src/memos/configs/mem_scheduler.py @@ -1,13 +1,17 @@ +import os + +from pathlib import Path from typing import Any, ClassVar from pydantic import ConfigDict, Field, field_validator, model_validator from memos.configs.base import BaseConfig from memos.mem_scheduler.modules.schemas import ( + BASE_DIR, DEFAULT_ACT_MEM_DUMP_PATH, - DEFAULT_ACTIVATION_MEM_SIZE, DEFAULT_CONSUME_INTERVAL_SECONDS, DEFAULT_THREAD__POOL_MAX_WORKERS, + DictConversionMixin, ) @@ -33,6 +37,10 @@ class BaseSchedulerConfig(BaseConfig): le=60, description=f"Interval for consuming messages from queue in seconds (default: {DEFAULT_CONSUME_INTERVAL_SECONDS})", ) + auth_config_path: str | None = Field( + default=None, + description="Path to the authentication configuration file containing private credentials", + ) class GeneralSchedulerConfig(BaseSchedulerConfig): @@ -42,14 +50,13 @@ class GeneralSchedulerConfig(BaseSchedulerConfig): context_window_size: int | None = Field( default=5, description="Size of the context window for conversation history" ) - activation_mem_size: int | None = Field( - default=DEFAULT_ACTIVATION_MEM_SIZE, # Assuming DEFAULT_ACTIVATION_MEM_SIZE is 1000 - description="Maximum size of the activation memory", - ) act_mem_dump_path: str | None = Field( default=DEFAULT_ACT_MEM_DUMP_PATH, # Replace with DEFAULT_ACT_MEM_DUMP_PATH description="File path for dumping activation memory", ) + enable_act_memory_update: bool = Field( + default=False, description="Whether to enable automatic activation memory updates" + ) class SchedulerConfigFactory(BaseConfig): @@ -76,3 +83,82 @@ def create_config(self) -> "SchedulerConfigFactory": config_class = self.backend_to_class[self.backend] self.config = config_class(**self.config) return self + + +# ************************* Auth ************************* +class RabbitMQConfig( + BaseConfig, +): + host_name: str = Field(default="", description="Endpoint for RabbitMQ instance access") + user_name: str = Field(default="", description="Static username for RabbitMQ instance") + password: str = Field(default="", description="Password for the static username") + virtual_host: str = Field(default="", description="Vhost name for RabbitMQ instance") + erase_on_connect: bool = Field( + default=True, description="Whether to clear connection state or buffers upon connecting" + ) + port: int = Field( + default=5672, + description="Port number for RabbitMQ instance access", + ge=1, # Port must be >= 1 + le=65535, # Port must be <= 65535 + ) + + +class GraphDBAuthConfig(BaseConfig): + uri: str = Field(default="localhost", description="URI for graph database access") + + +class OpenAIConfig(BaseConfig): + api_key: str = Field(default="", description="API key for OpenAI service") + base_url: str = Field(default="", description="Base URL for API endpoint") + default_model: str = Field(default="", description="Default model to use") + + +class AuthConfig(BaseConfig, DictConversionMixin): + rabbitmq: RabbitMQConfig + openai: OpenAIConfig + graph_db: GraphDBAuthConfig + default_config_path: ClassVar[str] = ( + f"{BASE_DIR}/examples/data/config/mem_scheduler/scheduler_auth.yaml" + ) + + @classmethod + def from_local_yaml(cls, config_path: str | None = None) -> "AuthConfig": + """ + Load configuration from YAML file + + Args: + config_path: Path to YAML configuration file + + Returns: + AuthConfig instance + + Raises: + FileNotFoundError: If config file doesn't exist + ValueError: If YAML parsing or validation fails + """ + + if config_path is None: + config_path = cls.default_config_path + + # Check file exists + if not Path(config_path).exists(): + raise FileNotFoundError(f"Config file not found: {config_path}") + + return cls.from_yaml_file(yaml_path=config_path) + + def set_openai_config_to_environment(self): + # Set environment variables + os.environ["OPENAI_API_KEY"] = self.openai.api_key + os.environ["OPENAI_BASE_URL"] = self.openai.base_url + os.environ["MODEL"] = self.openai.default_model + + @classmethod + def default_config_exists(cls) -> bool: + """ + Check if the default configuration file exists. + + Returns: + bool: True if the default config file exists, False otherwise + """ + return Path(cls.default_config_path).exists() diff --git a/src/memos/configs/memory.py b/src/memos/configs/memory.py index fbc0f4b86..8f824218c 100644 --- a/src/memos/configs/memory.py +++ b/src/memos/configs/memory.py @@ -52,9 +52,9 @@ class KVCacheMemoryConfig(BaseActMemoryConfig): @classmethod def validate_extractor_llm(cls, extractor_llm: LLMConfigFactory) -> LLMConfigFactory: """Validate the extractor_llm field.""" - if extractor_llm.backend != "huggingface": + if extractor_llm.backend not in ["huggingface", "huggingface_singleton", "vllm"]: raise ConfigurationError( - f"KVCacheMemoryConfig requires extractor_llm backend to be 'huggingface', got '{extractor_llm.backend}'" + f"KVCacheMemoryConfig requires extractor_llm backend to be 'huggingface' or 'huggingface_singleton', got '{extractor_llm.backend}'" ) return extractor_llm @@ -84,9 +84,9 @@ class LoRAMemoryConfig(BaseParaMemoryConfig): @classmethod def validate_extractor_llm(cls, extractor_llm: LLMConfigFactory) -> LLMConfigFactory: """Validate the extractor_llm field.""" - if extractor_llm.backend not in ["huggingface"]: + if extractor_llm.backend not in ["huggingface", "huggingface_singleton"]: raise ConfigurationError( - f"LoRAMemoryConfig requires extractor_llm backend to be 'huggingface', got '{extractor_llm.backend}'" + f"LoRAMemoryConfig requires extractor_llm backend to be 'huggingface' or 'huggingface_singleton', got '{extractor_llm.backend}'" ) return extractor_llm @@ -181,6 +181,7 @@ class MemoryConfigFactory(BaseConfig): "general_text": GeneralTextMemoryConfig, "tree_text": TreeTextMemoryConfig, "kv_cache": KVCacheMemoryConfig, + "vllm_kv_cache": KVCacheMemoryConfig, # Use same config as kv_cache "lora": LoRAMemoryConfig, "uninitialized": UninitializedMemoryConfig, } diff --git a/src/memos/dependency.py b/src/memos/dependency.py new file mode 100644 index 000000000..e132eab8d --- /dev/null +++ b/src/memos/dependency.py @@ -0,0 +1,52 @@ +""" +This utility provides tools for managing dependencies in MemOS. +""" + +import functools +import importlib + + +def require_python_package( + import_name: str, install_command: str | None = None, install_link: str | None = None +): + """Check if a package is available and provide installation hints on import failure. + + Args: + import_name (str): The top-level importable module name a package provides. + install_command (str, optional): Installation command. + install_link (str, optional): URL link to installation guide. + + Returns: + Callable: A decorator function that wraps the target function with package availability check. + + Raises: + ImportError: When the specified package is not available, with installation + instructions included in the error message. + + Example: + >>> @require_python_package( + ... import_name='faiss', + ... install_command='pip install faiss-cpu', + ... install_link='https://github.com/facebookresearch/faiss/blob/main/INSTALL.md' + ... ) + ... def create_faiss_index(): + ... from faiss import IndexFlatL2 # Actual import in function + ... return IndexFlatL2(128) + """ + + def decorator(func): + @functools.wraps(func) + def wrapper(*args, **kwargs): + try: + importlib.import_module(import_name) + except ImportError: + error_msg = f"Missing required module - '{import_name}'\n" + error_msg += f"💡 Install command: {install_command}\n" if install_command else "" + error_msg += f"💡 Install guide: {install_link}\n" if install_link else "" + + raise ImportError(error_msg) from None + return func(*args, **kwargs) + + return wrapper + + return decorator diff --git a/src/memos/embedders/ark.py b/src/memos/embedders/ark.py new file mode 100644 index 000000000..db6b42bd4 --- /dev/null +++ b/src/memos/embedders/ark.py @@ -0,0 +1,92 @@ +from memos.configs.embedder import ArkEmbedderConfig +from memos.dependency import require_python_package +from memos.embedders.base import BaseEmbedder +from memos.log import get_logger + + +logger = get_logger(__name__) + + +class ArkEmbedder(BaseEmbedder): + """Ark Embedder class.""" + + @require_python_package( + import_name="volcenginesdkarkruntime", + install_command="pip install 'volcengine-python-sdk[ark]'", + install_link="https://www.volcengine.com/docs/82379/1541595", + ) + def __init__(self, config: ArkEmbedderConfig): + from volcenginesdkarkruntime import Ark + + self.config = config + + if self.config.embedding_dims is not None: + logger.warning( + "Ark does not support specifying embedding dimensions. " + "The embedding dimensions is determined by the model." + "`embedding_dims` will be set to None." + ) + self.config.embedding_dims = None + + # Default model if not specified + if not self.config.model_name_or_path: + self.config.model_name_or_path = "doubao-embedding-vision-250615" + + # Initialize ark client + self.client = Ark(api_key=self.config.api_key, base_url=self.config.api_base) + + def embed(self, texts: list[str]) -> list[list[float]]: + """ + Generate embeddings for the given texts. + + Args: + texts: List of texts to embed. + + Returns: + List of embeddings, each represented as a list of floats. + """ + from volcenginesdkarkruntime.types.multimodal_embedding import ( + MultimodalEmbeddingContentPartTextParam, + ) + + if self.config.multi_modal: + texts_input = [ + MultimodalEmbeddingContentPartTextParam(text=text, type="text") for text in texts + ] + return self.multimodal_embeddings(inputs=texts_input, chunk_size=self.config.chunk_size) + return self.text_embedding(texts, chunk_size=self.config.chunk_size) + + def text_embedding(self, inputs: list[str], chunk_size: int | None = None) -> list[list[float]]: + chunk_size_ = chunk_size or self.config.chunk_size + embeddings: list[list[float]] = [] + for i in range(0, len(inputs), chunk_size_): + response = self.client.embeddings.create( + model=self.config.model_name_or_path, + input=inputs[i : i + chunk_size_], + ) + + data = [response.data] if isinstance(response.data, dict) else response.data + embeddings.extend(r.embedding for r in data) + + return embeddings + + def multimodal_embeddings( + self, inputs: list, chunk_size: int | None = None + ) -> list[list[float]]: + from volcenginesdkarkruntime.types.multimodal_embedding import ( + MultimodalEmbeddingResponse, # noqa: TC002 + ) + + chunk_size_ = chunk_size or self.config.chunk_size + embeddings: list[list[float]] = [] + + for i in range(0, len(inputs), chunk_size_): + response: MultimodalEmbeddingResponse = self.client.multimodal_embeddings.create( + model=self.config.model_name_or_path, + input=inputs[i : i + chunk_size_], + ) + + data = [response.data] if isinstance(response.data, dict) else response.data + embeddings.extend(r["embedding"] for r in data) + + return embeddings diff --git a/src/memos/embedders/factory.py b/src/memos/embedders/factory.py index 977bf95df..b15ad7c47 100644 --- a/src/memos/embedders/factory.py +++ b/src/memos/embedders/factory.py @@ -1,9 +1,11 @@ from typing import Any, ClassVar from memos.configs.embedder import EmbedderConfigFactory +from memos.embedders.ark import ArkEmbedder from memos.embedders.base import BaseEmbedder from memos.embedders.ollama import OllamaEmbedder from memos.embedders.sentence_transformer import SenTranEmbedder +from memos.embedders.universal_api import UniversalAPIEmbedder class EmbedderFactory(BaseEmbedder): @@ -12,6 +14,8 @@ class EmbedderFactory(BaseEmbedder): backend_to_class: ClassVar[dict[str, Any]] = { "ollama": OllamaEmbedder, "sentence_transformer": SenTranEmbedder, + "ark": ArkEmbedder, + "universal_api": UniversalAPIEmbedder, } @classmethod diff --git a/src/memos/embedders/sentence_transformer.py b/src/memos/embedders/sentence_transformer.py index b1dcffc66..1ae818ad6 100644 --- a/src/memos/embedders/sentence_transformer.py +++ b/src/memos/embedders/sentence_transformer.py @@ -1,6 +1,5 @@ -from sentence_transformers import SentenceTransformer - from memos.configs.embedder import SenTranEmbedderConfig +from memos.dependency import require_python_package from memos.embedders.base import BaseEmbedder from memos.log import get_logger @@ -11,7 +10,14 @@ class SenTranEmbedder(BaseEmbedder): """Sentence Transformer Embedder class.""" + @require_python_package( + import_name="sentence_transformers", + install_command="pip install sentence-transformers", + install_link="https://www.sbert.net/docs/installation.html", + ) def __init__(self, config: SenTranEmbedderConfig): + from sentence_transformers import SentenceTransformer + self.config = config self.model = SentenceTransformer( self.config.model_name_or_path, trust_remote_code=self.config.trust_remote_code diff --git a/src/memos/embedders/universal_api.py b/src/memos/embedders/universal_api.py new file mode 100644 index 000000000..72116cf05 --- /dev/null +++ b/src/memos/embedders/universal_api.py @@ -0,0 +1,32 @@ +from openai import AzureOpenAI as AzureClient +from openai import OpenAI as OpenAIClient + +from memos.configs.embedder import UniversalAPIEmbedderConfig +from memos.embedders.base import BaseEmbedder + + +class UniversalAPIEmbedder(BaseEmbedder): + def __init__(self, config: UniversalAPIEmbedderConfig): + self.provider = config.provider + self.config = config + + if self.provider == "openai": + self.client = OpenAIClient(api_key=config.api_key, base_url=config.base_url) + elif self.provider == "azure": + self.client = AzureClient( + azure_endpoint=config.base_url, + api_version="2024-03-01-preview", + api_key=config.api_key, + ) + else: + raise ValueError(f"Unsupported provider: {self.provider}") + + def embed(self, texts: list[str]) -> list[list[float]]: + if self.provider == "openai" or self.provider == "azure": + response = self.client.embeddings.create( + model=getattr(self.config, "model_name_or_path", "text-embedding-3-large"), + input=texts, + ) + return [r.embedding for r in response.data] + else: + raise ValueError(f"Unsupported provider: {self.provider}") diff --git a/src/memos/graph_dbs/base.py b/src/memos/graph_dbs/base.py index 89552e314..d59139ef2 100644 --- a/src/memos/graph_dbs/base.py +++ b/src/memos/graph_dbs/base.py @@ -9,12 +9,12 @@ class BaseGraphDB(ABC): # Node (Memory) Management @abstractmethod - def add_node(self, id: str, content: str, metadata: dict[str, Any]) -> None: + def add_node(self, id: str, memory: str, metadata: dict[str, Any]) -> None: """ Add a memory node to the graph. Args: id: Unique identifier for the memory node. - content: Raw memory content (e.g., text). + memory: Raw memory content (e.g., text). metadata: Dictionary of metadata (e.g., timestamp, tags, source). """ diff --git a/src/memos/graph_dbs/factory.py b/src/memos/graph_dbs/factory.py index c100270da..c4365d16a 100644 --- a/src/memos/graph_dbs/factory.py +++ b/src/memos/graph_dbs/factory.py @@ -3,6 +3,7 @@ from memos.configs.graph_db import GraphDBConfigFactory from memos.graph_dbs.base import BaseGraphDB from memos.graph_dbs.neo4j import Neo4jGraphDB +from memos.graph_dbs.neo4j_community import Neo4jCommunityGraphDB class GraphStoreFactory(BaseGraphDB): @@ -10,6 +11,7 @@ class GraphStoreFactory(BaseGraphDB): backend_to_class: ClassVar[dict[str, Any]] = { "neo4j": Neo4jGraphDB, + "neo4j-community": Neo4jCommunityGraphDB, } @classmethod diff --git a/src/memos/graph_dbs/neo4j.py b/src/memos/graph_dbs/neo4j.py index 252d6e981..1489ff541 100644 --- a/src/memos/graph_dbs/neo4j.py +++ b/src/memos/graph_dbs/neo4j.py @@ -3,9 +3,8 @@ from datetime import datetime from typing import Any, Literal -from neo4j import GraphDatabase - from memos.configs.graph_db import Neo4jGraphDBConfig +from memos.dependency import require_python_package from memos.graph_dbs.base import BaseGraphDB from memos.log import get_logger @@ -13,17 +12,6 @@ logger = get_logger(__name__) -def _parse_node(node_data: dict[str, Any]) -> dict[str, Any]: - node = node_data.copy() - - # Convert Neo4j datetime to string - for time_field in ("created_at", "updated_at"): - if time_field in node and hasattr(node[time_field], "isoformat"): - node[time_field] = node[time_field].isoformat() - - return {"id": node.pop("id"), "memory": node.pop("memory", ""), "metadata": node} - - def _compose_node(item: dict[str, Any]) -> tuple[str, str, dict[str, Any]]: node_id = item["id"] memory = item["memory"] @@ -55,11 +43,37 @@ def _prepare_node_metadata(metadata: dict[str, Any]) -> dict[str, Any]: class Neo4jGraphDB(BaseGraphDB): """Neo4j-based implementation of a graph memory store.""" + @require_python_package( + import_name="neo4j", + install_command="pip install neo4j", + install_link="https://neo4j.com/docs/python-manual/current/install/", + ) def __init__(self, config: Neo4jGraphDBConfig): + """Neo4j-based implementation of a graph memory store. + + Tenant Modes: + - use_multi_db = True: + Dedicated Database Mode (Multi-Database Multi-Tenant). + Each tenant or logical scope uses a separate Neo4j database. + `db_name` is the specific tenant database. + `user_name` can be None (optional). + + - use_multi_db = False: + Shared Database Multi-Tenant Mode. + All tenants share a single Neo4j database. + `db_name` is the shared database. + `user_name` is required to isolate each tenant's data at the node level. + All node queries will enforce `user_name` in WHERE conditions and store it in metadata, + but it will be removed automatically before returning to external consumers. + """ + from neo4j import GraphDatabase + self.config = config self.driver = GraphDatabase.driver(config.uri, auth=(config.user, config.password)) self.db_name = config.db_name + self.user_name = config.user_name + self.system_db_name = "system" if config.use_multi_db else config.db_name if config.auto_create: self._ensure_database_exists() @@ -86,21 +100,38 @@ def get_memory_count(self, memory_type: str) -> int: query = """ MATCH (n:Memory) WHERE n.memory_type = $memory_type - RETURN COUNT(n) AS count """ + if not self.config.use_multi_db and self.config.user_name: + query += "\nAND n.user_name = $user_name" + query += "\nRETURN COUNT(n) AS count" with self.driver.session(database=self.db_name) as session: - result = session.run(query, memory_type=memory_type) + result = session.run( + query, + { + "memory_type": memory_type, + "user_name": self.config.user_name if self.config.user_name else None, + }, + ) return result.single()["count"] def count_nodes(self, scope: str) -> int: query = """ MATCH (n:Memory) WHERE n.memory_type = $scope - RETURN count(n) AS count """ + if not self.config.use_multi_db and self.config.user_name: + query += "\nAND n.user_name = $user_name" + query += "\nRETURN count(n) AS count" + with self.driver.session(database=self.db_name) as session: - result = session.run(query, {"scope": scope}).single() - return result["count"] + result = session.run( + query, + { + "scope": scope, + "user_name": self.config.user_name if self.config.user_name else None, + }, + ) + return result.single()["count"] def remove_oldest_memory(self, memory_type: str, keep_latest: int) -> None: """ @@ -113,14 +144,22 @@ def remove_oldest_memory(self, memory_type: str, keep_latest: int) -> None: query = f""" MATCH (n:Memory) WHERE n.memory_type = '{memory_type}' - WITH n ORDER BY n.updated_at DESC - SKIP {keep_latest} - DETACH DELETE n + """ + if not self.config.use_multi_db and self.config.user_name: + query += f"\nAND n.user_name = '{self.config.user_name}'" + + query += f""" + WITH n ORDER BY n.updated_at DESC + SKIP {keep_latest} + DETACH DELETE n """ with self.driver.session(database=self.db_name) as session: session.run(query) def add_node(self, id: str, memory: str, metadata: dict[str, Any]) -> None: + if not self.config.use_multi_db and self.config.user_name: + metadata["user_name"] = self.config.user_name + # Safely process metadata metadata = _prepare_node_metadata(metadata) @@ -162,10 +201,14 @@ def update_node(self, id: str, fields: dict[str, Any]) -> None: set_clauses.append("n += $fields") # Merge remaining fields set_clause_str = ",\n ".join(set_clauses) - query = f""" - MATCH (n:Memory {{id: $id}}) - SET {set_clause_str} + query = """ + MATCH (n:Memory {id: $id}) """ + if not self.config.use_multi_db and self.config.user_name: + query += "\nWHERE n.user_name = $user_name" + params["user_name"] = self.config.user_name + + query += f"\nSET {set_clause_str}" with self.driver.session(database=self.db_name) as session: session.run(query, **params) @@ -176,8 +219,17 @@ def delete_node(self, id: str) -> None: Args: id: Node identifier to delete. """ + query = "MATCH (n:Memory {id: $id})" + + params = {"id": id} + if not self.config.use_multi_db and self.config.user_name: + query += " WHERE n.user_name = $user_name" + params["user_name"] = self.config.user_name + + query += " DETACH DELETE n" + with self.driver.session(database=self.db_name) as session: - session.run("MATCH (n:Memory {id: $id}) DETACH DELETE n", id=id) + session.run(query, **params) # Edge (Relationship) Management def add_edge(self, source_id: str, target_id: str, type: str) -> None: @@ -188,15 +240,21 @@ def add_edge(self, source_id: str, target_id: str, type: str) -> None: target_id: ID of the target node. type: Relationship type (e.g., 'RELATE_TO', 'PARENT'). """ + query = """ + MATCH (a:Memory {id: $source_id}) + MATCH (b:Memory {id: $target_id}) + """ + params = {"source_id": source_id, "target_id": target_id} + if not self.config.use_multi_db and self.config.user_name: + query += """ + WHERE a.user_name = $user_name AND b.user_name = $user_name + """ + params["user_name"] = self.config.user_name + + query += f"\nMERGE (a)-[:{type}]->(b)" + with self.driver.session(database=self.db_name) as session: - session.run( - f""" - MATCH (a:Memory {{id: $source_id}}) - MATCH (b:Memory {{id: $target_id}}) - MERGE (a)-[:{type}]->(b) - """, - {"source_id": source_id, "target_id": target_id}, - ) + session.run(query, params) def delete_edge(self, source_id: str, target_id: str, type: str) -> None: """ @@ -206,12 +264,21 @@ def delete_edge(self, source_id: str, target_id: str, type: str) -> None: target_id: ID of the target node. type: Relationship type to remove. """ + query = f""" + MATCH (a:Memory {{id: $source}}) + -[r:{type}]-> + (b:Memory {{id: $target}}) + """ + params = {"source": source_id, "target": target_id} + + if not self.config.use_multi_db and self.config.user_name: + query += "\nWHERE a.user_name = $user_name AND b.user_name = $user_name" + params["user_name"] = self.config.user_name + + query += "\nDELETE r" + with self.driver.session(database=self.db_name) as session: - session.run( - f"MATCH (a:Memory {{id: $source}})-[r:{type}]->(b:Memory {{id: $target}})\nDELETE r", - source=source_id, - target=target_id, - ) + session.run(query, params) def edge_exists( self, source_id: str, target_id: str, type: str = "ANY", direction: str = "OUTGOING" @@ -241,14 +308,18 @@ def edge_exists( raise ValueError( f"Invalid direction: {direction}. Must be 'OUTGOING', 'INCOMING', or 'ANY'." ) + query = f"MATCH {pattern}" + params = {"source": source_id, "target": target_id} + + if not self.config.use_multi_db and self.config.user_name: + query += "\nWHERE a.user_name = $user_name AND b.user_name = $user_name" + params["user_name"] = self.config.user_name + + query += "\nRETURN r" # Run the Cypher query with self.driver.session(database=self.db_name) as session: - result = session.run( - f"MATCH {pattern} RETURN r", - source=source_id, - target=target_id, - ) + result = session.run(query, params) return result.single() is not None # Graph Query & Reasoning @@ -260,10 +331,17 @@ def get_node(self, id: str) -> dict[str, Any] | None: Returns: Dictionary of node fields, or None if not found. """ + where_user = "" + params = {"id": id} + if not self.config.use_multi_db and self.config.user_name: + where_user = " AND n.user_name = $user_name" + params["user_name"] = self.config.user_name + + query = f"MATCH (n:Memory) WHERE n.id = $id {where_user} RETURN n" + with self.driver.session(database=self.db_name) as session: - result = session.run("MATCH (n:Memory {id: $id}) RETURN n", id=id) - record = result.single() - return _parse_node(dict(record["n"])) if record else None + record = session.run(query, params).single() + return self._parse_node(dict(record["n"])) if record else None def get_nodes(self, ids: list[str]) -> list[dict[str, Any]]: """ @@ -280,10 +358,18 @@ def get_nodes(self, ids: list[str]) -> list[dict[str, Any]]: if not ids: return [] - query = "MATCH (n:Memory) WHERE n.id IN $ids RETURN n" + where_user = "" + params = {"ids": ids} + + if not self.config.use_multi_db and self.config.user_name: + where_user = " AND n.user_name = $user_name" + params["user_name"] = self.config.user_name + + query = f"MATCH (n:Memory) WHERE n.id IN $ids{where_user} RETURN n" + with self.driver.session(database=self.db_name) as session: - results = session.run(query, {"ids": ids}) - return [_parse_node(dict(record["n"])) for record in results] + results = session.run(query, params) + return [self._parse_node(dict(record["n"])) for record in results] def get_edges(self, id: str, type: str = "ANY", direction: str = "ANY") -> list[dict[str, str]]: """ @@ -317,14 +403,20 @@ def get_edges(self, id: str, type: str = "ANY", direction: str = "ANY") -> list[ else: raise ValueError("Invalid direction. Must be 'OUTGOING', 'INCOMING', or 'ANY'.") + params = {"id": id} + + if not self.config.use_multi_db and self.config.user_name: + where_clause += " AND a.user_name = $user_name AND b.user_name = $user_name" + params["user_name"] = self.config.user_name + query = f""" - MATCH {pattern} - WHERE {where_clause} - RETURN a.id AS from_id, b.id AS to_id, type(r) AS type - """ + MATCH {pattern} + WHERE {where_clause} + RETURN a.id AS from_id, b.id AS to_id, type(r) AS type + """ with self.driver.session(database=self.db_name) as session: - result = session.run(query, id=id) + result = session.run(query, params) edges = [] for record in result: edges.append( @@ -365,19 +457,7 @@ def get_neighbors_by_tag( Returns: List of dicts with node details and overlap count. """ - query = """ - MATCH (n:Memory) - WHERE NOT n.id IN $exclude_ids - AND n.status = 'activated' - AND n.type <> 'reasoning' - AND n.memory_type <> 'WorkingMemory' - WITH n, [tag IN n.tags WHERE tag IN $tags] AS overlap_tags - WHERE size(overlap_tags) >= $min_overlap - RETURN n, size(overlap_tags) AS overlap_count - ORDER BY overlap_count DESC - LIMIT $top_k - """ - + where_user = "" params = { "tags": tags, "exclude_ids": exclude_ids, @@ -385,18 +465,47 @@ def get_neighbors_by_tag( "top_k": top_k, } + if not self.config.use_multi_db and self.config.user_name: + where_user = "AND n.user_name = $user_name" + params["user_name"] = self.config.user_name + + query = f""" + MATCH (n:Memory) + WHERE NOT n.id IN $exclude_ids + AND n.status = 'activated' + AND n.type <> 'reasoning' + AND n.memory_type <> 'WorkingMemory' + {where_user} + WITH n, [tag IN n.tags WHERE tag IN $tags] AS overlap_tags + WHERE size(overlap_tags) >= $min_overlap + RETURN n, size(overlap_tags) AS overlap_count + ORDER BY overlap_count DESC + LIMIT $top_k + """ + with self.driver.session(database=self.db_name) as session: result = session.run(query, params) - return [_parse_node(dict(record["n"])) for record in result] + return [self._parse_node(dict(record["n"])) for record in result] + + def get_children_with_embeddings(self, id: str) -> list[dict[str, Any]]: + where_user = "" + params = {"id": id} + + if not self.config.use_multi_db and self.config.user_name: + where_user = "AND p.user_name = $user_name AND c.user_name = $user_name" + params["user_name"] = self.config.user_name + + query = f""" + MATCH (p:Memory)-[:PARENT]->(c:Memory) + WHERE p.id = $id {where_user} + RETURN c.id AS id, c.embedding AS embedding, c.memory AS memory + """ - def get_children_with_embeddings(self, id: str) -> list[str]: - query = """ - MATCH (p:Memory)-[:PARENT]->(c:Memory) - WHERE p.id = $id - RETURN c.id AS id, c.embedding AS embedding, c.memory AS memory - """ with self.driver.session(database=self.db_name) as session: - return list(session.run(query, id=id)) + result = session.run(query, params) + return [ + {"id": r["id"], "embedding": r["embedding"], "memory": r["memory"]} for r in result + ] def get_path(self, source_id: str, target_id: str, max_depth: int = 3) -> list[str]: """ @@ -427,16 +536,29 @@ def get_subgraph( } """ with self.driver.session(database=self.db_name) as session: - status_clause = f", status: '{center_status}'" if center_status else "" + params = {"center_id": center_id} + center_user_clause = "" + neighbor_user_clause = "" + + if not self.config.use_multi_db and self.config.user_name: + center_user_clause = " AND center.user_name = $user_name" + neighbor_user_clause = " WHERE neighbor.user_name = $user_name" + params["user_name"] = self.config.user_name + status_clause = f" AND center.status = '{center_status}'" if center_status else "" + query = f""" - MATCH (center:Memory {{id: $center_id{status_clause}}}) - OPTIONAL MATCH (center)-[r*1..{depth}]-(neighbor:Memory) - WITH collect(DISTINCT center) AS centers, - collect(DISTINCT neighbor) AS neighbors, - collect(DISTINCT r) AS rels - RETURN centers, neighbors, rels + MATCH (center:Memory) + WHERE center.id = $center_id{status_clause}{center_user_clause} + + OPTIONAL MATCH (center)-[r*1..{depth}]-(neighbor:Memory) + {neighbor_user_clause} + + WITH collect(DISTINCT center) AS centers, + collect(DISTINCT neighbor) AS neighbors, + collect(DISTINCT r) AS rels + RETURN centers, neighbors, rels """ - record = session.run(query, {"center_id": center_id}).single() + record = session.run(query, params).single() if not record: return {"core_node": None, "neighbors": [], "edges": []} @@ -445,8 +567,8 @@ def get_subgraph( if not centers or centers[0] is None: return {"core_node": None, "neighbors": [], "edges": []} - core_node = _parse_node(dict(centers[0])) - neighbors = [_parse_node(dict(n)) for n in record["neighbors"] if n] + core_node = self._parse_node(dict(centers[0])) + neighbors = [self._parse_node(dict(n)) for n in record["neighbors"] if n] edges = [] for rel_chain in record["rels"]: for rel in rel_chain: @@ -508,6 +630,8 @@ def search_by_embedding( where_clauses.append("node.memory_type = $scope") if status: where_clauses.append("node.status = $status") + if not self.config.use_multi_db and self.config.user_name: + where_clauses.append("node.user_name = $user_name") where_clause = "" if where_clauses: @@ -525,6 +649,8 @@ def search_by_embedding( parameters["scope"] = scope if status: parameters["status"] = status + if not self.config.use_multi_db and self.config.user_name: + parameters["user_name"] = self.config.user_name with self.driver.session(database=self.db_name) as session: result = session.run(query, parameters) @@ -592,6 +718,10 @@ def get_by_metadata(self, filters: list[dict[str, Any]]) -> list[str]: else: raise ValueError(f"Unsupported operator: {op}") + if not self.config.use_multi_db and self.config.user_name: + where_clauses.append("n.user_name = $user_name") + params["user_name"] = self.config.user_name + where_str = " AND ".join(where_clauses) query = f"MATCH (n:Memory) WHERE {where_str} RETURN n.id AS id" @@ -620,6 +750,20 @@ def get_grouped_counts( if not group_fields: raise ValueError("group_fields cannot be empty") + final_params = params.copy() if params else {} + + if not self.config.use_multi_db and self.config.user_name: + user_clause = "n.user_name = $user_name" + final_params["user_name"] = self.config.user_name + if where_clause: + where_clause = where_clause.strip() + if where_clause.upper().startswith("WHERE"): + where_clause += f" AND {user_clause}" + else: + where_clause = f"WHERE {where_clause} AND {user_clause}" + else: + where_clause = f"WHERE {user_clause}" + # Force RETURN field AS field to guarantee key match group_fields_cypher = ", ".join([f"n.{field} AS {field}" for field in group_fields]) @@ -630,7 +774,7 @@ def get_grouped_counts( """ with self.driver.session(database=self.db_name) as session: - result = session.run(query, params or {}) + result = session.run(query, final_params) return [ {**{field: record[field] for field in group_fields}, "count": record["count"]} for record in result @@ -669,17 +813,16 @@ def clear(self) -> None: Clear the entire graph if the target database exists. """ try: - # Step 1: Check if the database exists - with self.driver.session(database="system") as session: - result = session.run("SHOW DATABASES YIELD name RETURN name") - db_names = [record["name"] for record in result] - if self.db_name not in db_names: - logger.info(f"[Skip] Database '{self.db_name}' does not exist.") - return + if not self.config.use_multi_db and self.config.user_name: + query = "MATCH (n:Memory) WHERE n.user_name = $user_name DETACH DELETE n" + params = {"user_name": self.config.user_name} + else: + query = "MATCH (n) DETACH DELETE n" + params = {} # Step 2: Clear the graph in that database with self.driver.session(database=self.db_name) as session: - session.run("MATCH (n) DETACH DELETE n") + session.run(query, params) logger.info(f"Cleared all nodes from database '{self.db_name}'.") except Exception as e: @@ -698,14 +841,22 @@ def export_graph(self) -> dict[str, Any]: """ with self.driver.session(database=self.db_name) as session: # Export nodes - node_result = session.run("MATCH (n:Memory) RETURN n") - nodes = [_parse_node(dict(record["n"])) for record in node_result] + node_query = "MATCH (n:Memory)" + edge_query = "MATCH (a:Memory)-[r]->(b:Memory)" + params = {} + + if not self.config.use_multi_db and self.config.user_name: + node_query += " WHERE n.user_name = $user_name" + edge_query += " WHERE a.user_name = $user_name AND b.user_name = $user_name" + params["user_name"] = self.config.user_name + + node_result = session.run(f"{node_query} RETURN n", params) + nodes = [self._parse_node(dict(record["n"])) for record in node_result] # Export edges - edge_result = session.run(""" - MATCH (a:Memory)-[r]->(b:Memory) - RETURN a.id AS source, b.id AS target, type(r) AS type - """) + edge_result = session.run( + f"{edge_query} RETURN a.id AS source, b.id AS target, type(r) AS type", params + ) edges = [ {"source": record["source"], "target": record["target"], "type": record["type"]} for record in edge_result @@ -724,6 +875,9 @@ def import_graph(self, data: dict[str, Any]) -> None: for node in data.get("nodes", []): id, memory, metadata = _compose_node(node) + if not self.config.use_multi_db and self.config.user_name: + metadata["user_name"] = self.config.user_name + metadata = _prepare_node_metadata(metadata) # Merge node and set metadata @@ -769,15 +923,22 @@ def get_all_memory_items(self, scope: str) -> list[dict]: if scope not in {"WorkingMemory", "LongTermMemory", "UserMemory"}: raise ValueError(f"Unsupported memory type scope: {scope}") - query = """ - MATCH (n:Memory) - WHERE n.memory_type = $scope - RETURN n - """ + where_clause = "WHERE n.memory_type = $scope" + params = {"scope": scope} + + if not self.config.use_multi_db and self.config.user_name: + where_clause += " AND n.user_name = $user_name" + params["user_name"] = self.config.user_name + + query = f""" + MATCH (n:Memory) + {where_clause} + RETURN n + """ with self.driver.session(database=self.db_name) as session: - results = session.run(query, {"scope": scope}) - return [_parse_node(dict(record["n"])) for record in results] + results = session.run(query, params) + return [self._parse_node(dict(record["n"])) for record in results] def get_structure_optimization_candidates(self, scope: str) -> list[dict]: """ @@ -785,37 +946,62 @@ def get_structure_optimization_candidates(self, scope: str) -> list[dict]: - Isolated nodes, nodes with empty background, or nodes with exactly one child. - Plus: the child of any parent node that has exactly one child. """ - query = """ - MATCH (n:Memory) + where_clause = """ WHERE n.memory_type = $scope AND n.status = 'activated' AND NOT ( (n)-[:PARENT]->() OR ()-[:PARENT]->(n) ) - RETURN n.id AS id, n AS node - """ + """ + params = {"scope": scope} + + if not self.config.use_multi_db and self.config.user_name: + where_clause += " AND n.user_name = $user_name" + params["user_name"] = self.config.user_name + + query = f""" + MATCH (n:Memory) + {where_clause} + RETURN n.id AS id, n AS node + """ with self.driver.session(database=self.db_name) as session: - results = session.run(query, {"scope": scope}) - return [_parse_node({"id": record["id"], **dict(record["node"])}) for record in results] + results = session.run(query, params) + return [ + self._parse_node({"id": record["id"], **dict(record["node"])}) for record in results + ] def drop_database(self) -> None: """ Permanently delete the entire database this instance is using. WARNING: This operation is destructive and cannot be undone. """ - if self.db_name in ("system", "neo4j"): - raise ValueError(f"Refusing to drop protected database: {self.db_name}") + if self.config.use_multi_db: + if self.db_name in ("system", "neo4j"): + raise ValueError(f"Refusing to drop protected database: {self.db_name}") - with self.driver.session(database="system") as session: - session.run(f"DROP DATABASE {self.db_name} IF EXISTS") - print(f"Database '{self.db_name}' has been dropped.") + with self.driver.session(database=self.system_db_name) as session: + session.run(f"DROP DATABASE {self.db_name} IF EXISTS") + print(f"Database '{self.db_name}' has been dropped.") + else: + raise ValueError( + f"Refusing to drop protected database: {self.db_name} in " + f"Shared Database Multi-Tenant mode" + ) def _ensure_database_exists(self): - with self.driver.session(database="system") as session: - session.run(f"CREATE DATABASE $db_name IF NOT EXISTS", db_name=self.db_name) + from neo4j.exceptions import ClientError + + try: + with self.driver.session(database="system") as session: + session.run(f"CREATE DATABASE `{self.db_name}` IF NOT EXISTS") + except ClientError as e: + if "ExistingDatabaseFound" in str(e): + pass # Ignore, database already exists + else: + raise # Wait until the database is available for _ in range(10): - with self.driver.session(database="system") as session: + with self.driver.session(database=self.system_db_name) as session: result = session.run( "SHOW DATABASES YIELD name, currentStatus RETURN name, currentStatus" ) @@ -857,7 +1043,10 @@ def _create_vector_index( def _create_basic_property_indexes(self) -> None: """ - Create standard B-tree indexes on memory_type, created_at, and updated_at fields. + Create standard B-tree indexes on memory_type, created_at, + and updated_at fields. + Create standard B-tree indexes on user_name when use Shared Database + Multi-Tenant Mode """ try: with self.driver.session(database=self.db_name) as session: @@ -878,6 +1067,15 @@ def _create_basic_property_indexes(self) -> None: FOR (n:Memory) ON (n.updated_at) """) logger.debug("Index 'memory_updated_at_index' ensured.") + + if not self.config.use_multi_db and self.config.user_name: + session.run( + """ + CREATE INDEX memory_user_name_index IF NOT EXISTS + FOR (n:Memory) ON (n.user_name) + """ + ) + logger.debug("Index 'memory_user_name_index' ensured.") except Exception as e: logger.warning(f"Failed to create basic property indexes: {e}") @@ -892,3 +1090,14 @@ def _index_exists(self, index_name: str) -> bool: if record["name"] == index_name: return True return False + + def _parse_node(self, node_data: dict[str, Any]) -> dict[str, Any]: + node = node_data.copy() + + # Convert Neo4j datetime to string + for time_field in ("created_at", "updated_at"): + if time_field in node and hasattr(node[time_field], "isoformat"): + node[time_field] = node[time_field].isoformat() + node.pop("user_name", None) + + return {"id": node.pop("id"), "memory": node.pop("memory", ""), "metadata": node} diff --git a/src/memos/graph_dbs/neo4j_community.py b/src/memos/graph_dbs/neo4j_community.py new file mode 100644 index 000000000..98d9723bb --- /dev/null +++ b/src/memos/graph_dbs/neo4j_community.py @@ -0,0 +1,300 @@ +from typing import Any + +from memos.configs.graph_db import Neo4jGraphDBConfig +from memos.graph_dbs.neo4j import Neo4jGraphDB, _prepare_node_metadata +from memos.log import get_logger +from memos.vec_dbs.factory import VecDBFactory +from memos.vec_dbs.item import VecDBItem + + +logger = get_logger(__name__) + + +class Neo4jCommunityGraphDB(Neo4jGraphDB): + """ + Neo4j Community Edition graph memory store. + + Note: + This class avoids Enterprise-only features: + - No multi-database support + - No vector index + - No CREATE DATABASE + """ + + def __init__(self, config: Neo4jGraphDBConfig): + assert config.auto_create is False + assert config.use_multi_db is False + # Init vector database + self.vec_db = VecDBFactory.from_config(config.vec_config) + # Call parent init + super().__init__(config) + + def create_index( + self, + label: str = "Memory", + vector_property: str = "embedding", + dimensions: int = 1536, + index_name: str = "memory_vector_index", + ) -> None: + """ + Create the vector index for embedding and datetime indexes for created_at and updated_at fields. + """ + # Create indexes + self._create_basic_property_indexes() + + def add_node(self, id: str, memory: str, metadata: dict[str, Any]) -> None: + if not self.config.use_multi_db and self.config.user_name: + metadata["user_name"] = self.config.user_name + + # Safely process metadata + metadata = _prepare_node_metadata(metadata) + + # Extract required fields + embedding = metadata.pop("embedding", None) + if embedding is None: + raise ValueError(f"Missing 'embedding' in metadata for node {id}") + + # Merge node and set metadata + created_at = metadata.pop("created_at") + updated_at = metadata.pop("updated_at") + vector_sync_status = "success" + + try: + # Write to Vector DB + item = VecDBItem( + id=id, + vector=embedding, + payload={ + "memory": memory, + "vector_sync": vector_sync_status, + **metadata, # unpack all metadata keys to top-level + }, + ) + self.vec_db.add([item]) + except Exception as e: + logger.warning(f"[VecDB] Vector insert failed for node {id}: {e}") + vector_sync_status = "failed" + + metadata["vector_sync"] = vector_sync_status + query = """ + MERGE (n:Memory {id: $id}) + SET n.memory = $memory, + n.created_at = datetime($created_at), + n.updated_at = datetime($updated_at), + n += $metadata + """ + with self.driver.session(database=self.db_name) as session: + session.run( + query, + id=id, + memory=memory, + created_at=created_at, + updated_at=updated_at, + metadata=metadata, + ) + + def get_children_with_embeddings(self, id: str) -> list[dict[str, Any]]: + where_user = "" + params = {"id": id} + + if not self.config.use_multi_db and self.config.user_name: + where_user = "AND p.user_name = $user_name AND c.user_name = $user_name" + params["user_name"] = self.config.user_name + + query = f""" + MATCH (p:Memory)-[:PARENT]->(c:Memory) + WHERE p.id = $id {where_user} + RETURN c.id AS id, c.memory AS memory + """ + + with self.driver.session(database=self.db_name) as session: + result = session.run(query, params) + child_nodes = [{"id": r["id"], "memory": r["memory"]} for r in result] + + # Get embeddings from vector DB + ids = [n["id"] for n in child_nodes] + vec_items = {v.id: v.vector for v in self.vec_db.get_by_ids(ids)} + + # Merge results + for node in child_nodes: + node["embedding"] = vec_items.get(node["id"]) + + return child_nodes + + # Search / recall operations + def search_by_embedding( + self, + vector: list[float], + top_k: int = 5, + scope: str | None = None, + status: str | None = None, + threshold: float | None = None, + ) -> list[dict]: + """ + Retrieve node IDs based on vector similarity using external vector DB. + + Args: + vector (list[float]): The embedding vector representing query semantics. + top_k (int): Number of top similar nodes to retrieve. + scope (str, optional): Memory type filter (e.g., 'WorkingMemory', 'LongTermMemory'). + status (str, optional): Node status filter (e.g., 'activated', 'archived'). + threshold (float, optional): Minimum similarity score threshold (0 ~ 1). + + Returns: + list[dict]: A list of dicts with 'id' and 'score', ordered by similarity. + + Notes: + - This method uses an external vector database (not Neo4j) to perform the search. + - If 'scope' is provided, it restricts results to nodes with matching memory_type. + - If 'status' is provided, it further filters nodes by status. + - If 'threshold' is provided, only results with score >= threshold will be returned. + - The returned IDs can be used to fetch full node data from Neo4j if needed. + """ + # Build VecDB filter + vec_filter = {} + if scope: + vec_filter["memory_type"] = scope + if status: + vec_filter["status"] = status + vec_filter["vector_sync"] = "success" + vec_filter["user_name"] = self.config.user_name + + # Perform vector search + results = self.vec_db.search(query_vector=vector, top_k=top_k, filter=vec_filter) + + # Filter by threshold + if threshold is not None: + results = [r for r in results if r.score is None or r.score >= threshold] + + # Return consistent format + return [{"id": r.id, "score": r.score} for r in results] + + def get_all_memory_items(self, scope: str) -> list[dict]: + """ + Retrieve all memory items of a specific memory_type. + + Args: + scope (str): Must be one of 'WorkingMemory', 'LongTermMemory', or 'UserMemory'. + + Returns: + list[dict]: Full list of memory items under this scope. + """ + if scope not in {"WorkingMemory", "LongTermMemory", "UserMemory"}: + raise ValueError(f"Unsupported memory type scope: {scope}") + + where_clause = "WHERE n.memory_type = $scope" + params = {"scope": scope} + + if not self.config.use_multi_db and self.config.user_name: + where_clause += " AND n.user_name = $user_name" + params["user_name"] = self.config.user_name + + query = f""" + MATCH (n:Memory) + {where_clause} + RETURN n + """ + + with self.driver.session(database=self.db_name) as session: + results = session.run(query, params) + return [self._parse_node(dict(record["n"])) for record in results] + + def clear(self) -> None: + """ + Clear the entire graph if the target database exists. + """ + # Step 1: clear Neo4j part via parent logic + super().clear() + + # Step2: Clear the vector db + try: + items = self.vec_db.get_by_filter({"user_name": self.config.user_name}) + if items: + self.vec_db.delete([item.id for item in items]) + logger.info(f"Cleared {len(items)} vectors for user '{self.config.user_name}'.") + else: + logger.info(f"No vectors to clear for user '{self.config.user_name}'.") + except Exception as e: + logger.warning(f"Failed to clear vector DB for user '{self.config.user_name}': {e}") + + def drop_database(self) -> None: + """ + Permanently delete the entire database this instance is using. + WARNING: This operation is destructive and cannot be undone. + """ + raise ValueError( + f"Refusing to drop protected database: {self.db_name} in " + f"Shared Database Multi-Tenant mode" + ) + + # Avoid enterprise feature + def _ensure_database_exists(self): + pass + + def _create_basic_property_indexes(self) -> None: + """ + Create standard B-tree indexes on memory_type, created_at, + and updated_at fields. + Create standard B-tree indexes on user_name when use Shared Database + Multi-Tenant Mode + """ + # Step 1: Neo4j indexes + try: + with self.driver.session(database=self.db_name) as session: + session.run(""" + CREATE INDEX memory_type_index IF NOT EXISTS + FOR (n:Memory) ON (n.memory_type) + """) + logger.debug("Index 'memory_type_index' ensured.") + + session.run(""" + CREATE INDEX memory_created_at_index IF NOT EXISTS + FOR (n:Memory) ON (n.created_at) + """) + logger.debug("Index 'memory_created_at_index' ensured.") + + session.run(""" + CREATE INDEX memory_updated_at_index IF NOT EXISTS + FOR (n:Memory) ON (n.updated_at) + """) + logger.debug("Index 'memory_updated_at_index' ensured.") + + if not self.config.use_multi_db and self.config.user_name: + session.run( + """ + CREATE INDEX memory_user_name_index IF NOT EXISTS + FOR (n:Memory) ON (n.user_name) + """ + ) + logger.debug("Index 'memory_user_name_index' ensured.") + except Exception as e: + logger.warning(f"Failed to create basic property indexes: {e}") + + # Step 2: VectorDB indexes + try: + if hasattr(self.vec_db, "ensure_payload_indexes"): + self.vec_db.ensure_payload_indexes(["user_name", "memory_type", "status"]) + else: + logger.debug("VecDB does not support payload index creation; skipping.") + except Exception as e: + logger.warning(f"Failed to create VecDB payload indexes: {e}") + + def _parse_node(self, node_data: dict[str, Any]) -> dict[str, Any]: + """Parse Neo4j node and optionally fetch embedding from vector DB.""" + node = node_data.copy() + + # Convert Neo4j datetime to string + for time_field in ("created_at", "updated_at"): + if time_field in node and hasattr(node[time_field], "isoformat"): + node[time_field] = node[time_field].isoformat() + node.pop("user_name", None) + + new_node = {"id": node.pop("id"), "memory": node.pop("memory", ""), "metadata": node} + try: + vec_item = self.vec_db.get_by_id(new_node["id"]) + if vec_item and vec_item.vector: + new_node["metadata"]["embedding"] = vec_item.vector + except Exception as e: + logger.warning(f"Failed to fetch vector for node {new_node['id']}: {e}") + new_node["metadata"]["embedding"] = None + return new_node diff --git a/src/memos/llms/base.py b/src/memos/llms/base.py index 312d19f1b..8c3681e16 100644 --- a/src/memos/llms/base.py +++ b/src/memos/llms/base.py @@ -1,4 +1,5 @@ from abc import ABC, abstractmethod +from collections.abc import Generator from memos.configs.llm import BaseLLMConfig from memos.types import MessageList @@ -14,3 +15,11 @@ def __init__(self, config: BaseLLMConfig): @abstractmethod def generate(self, messages: MessageList, **kwargs) -> str: """Generate a response from the LLM.""" + + @abstractmethod + def generate_stream(self, messages: MessageList, **kwargs) -> Generator[str, None, None]: + """ + (Optional) Generate a streaming response from the LLM. + Subclasses should override this if they support streaming. + By default, this raises NotImplementedError. + """ diff --git a/src/memos/llms/deepseek.py b/src/memos/llms/deepseek.py new file mode 100644 index 000000000..f5ee4842b --- /dev/null +++ b/src/memos/llms/deepseek.py @@ -0,0 +1,54 @@ +from collections.abc import Generator + +from memos.configs.llm import DeepSeekLLMConfig +from memos.llms.openai import OpenAILLM +from memos.llms.utils import remove_thinking_tags +from memos.log import get_logger +from memos.types import MessageList + + +logger = get_logger(__name__) + + +class DeepSeekLLM(OpenAILLM): + """DeepSeek LLM via OpenAI-compatible API.""" + + def __init__(self, config: DeepSeekLLMConfig): + super().__init__(config) + + def generate(self, messages: MessageList) -> str: + """Generate a response from DeepSeek.""" + response = self.client.chat.completions.create( + model=self.config.model_name_or_path, + messages=messages, + temperature=self.config.temperature, + max_tokens=self.config.max_tokens, + top_p=self.config.top_p, + extra_body=self.config.extra_body, + ) + logger.info(f"Response from DeepSeek: {response.model_dump_json()}") + response_content = response.choices[0].message.content + if self.config.remove_think_prefix: + return remove_thinking_tags(response_content) + else: + return response_content + + def generate_stream(self, messages: MessageList, **kwargs) -> Generator[str, None, None]: + """Stream response from DeepSeek.""" + response = self.client.chat.completions.create( + model=self.config.model_name_or_path, + messages=messages, + stream=True, + temperature=self.config.temperature, + max_tokens=self.config.max_tokens, + top_p=self.config.top_p, + extra_body=self.config.extra_body, + ) + # Streaming chunks of text + for chunk in response: + delta = chunk.choices[0].delta + if hasattr(delta, "reasoning_content") and delta.reasoning_content: + yield delta.reasoning_content + + if hasattr(delta, "content") and delta.content: + yield delta.content diff --git a/src/memos/llms/factory.py b/src/memos/llms/factory.py index 4435508ba..0c12a667a 100644 --- a/src/memos/llms/factory.py +++ b/src/memos/llms/factory.py @@ -2,9 +2,13 @@ from memos.configs.llm import LLMConfigFactory from memos.llms.base import BaseLLM +from memos.llms.deepseek import DeepSeekLLM from memos.llms.hf import HFLLM +from memos.llms.hf_singleton import HFSingletonLLM from memos.llms.ollama import OllamaLLM -from memos.llms.openai import OpenAILLM +from memos.llms.openai import AzureLLM, OpenAILLM +from memos.llms.qwen import QwenLLM +from memos.llms.vllm import VLLMLLM class LLMFactory(BaseLLM): @@ -12,8 +16,13 @@ class LLMFactory(BaseLLM): backend_to_class: ClassVar[dict[str, Any]] = { "openai": OpenAILLM, + "azure": AzureLLM, "ollama": OllamaLLM, "huggingface": HFLLM, + "huggingface_singleton": HFSingletonLLM, # Add singleton version + "vllm": VLLMLLM, + "qwen": QwenLLM, + "deepseek": DeepSeekLLM, } @classmethod diff --git a/src/memos/llms/hf.py b/src/memos/llms/hf.py index 895d42627..00081b581 100644 --- a/src/memos/llms/hf.py +++ b/src/memos/llms/hf.py @@ -1,4 +1,5 @@ -import torch +from collections.abc import Generator +from typing import Any from transformers import ( AutoModelForCausalLM, @@ -71,6 +72,26 @@ def generate(self, messages: MessageList, past_key_values: DynamicCache | None = else: return self._generate_with_cache(prompt, past_key_values) + def generate_stream( + self, messages: MessageList, past_key_values: DynamicCache | None = None + ) -> Generator[str, None, None]: + """ + Generate a streaming response from the model. + Args: + messages (MessageList): Chat messages for prompt construction. + past_key_values (DynamicCache | None): Optional KV cache for fast generation. + Yields: + str: Streaming model response chunks. + """ + prompt = self.tokenizer.apply_chat_template( + messages, tokenize=False, add_generation_prompt=self.config.add_generation_prompt + ) + logger.info(f"HFLLM streaming prompt: {prompt}") + if past_key_values is None: + yield from self._generate_full_stream(prompt) + else: + yield from self._generate_with_cache_stream(prompt, past_key_values) + def _generate_full(self, prompt: str) -> str: """ Generate output from scratch using the full prompt. @@ -104,6 +125,73 @@ def _generate_full(self, prompt: str) -> str: else response ) + def _generate_full_stream(self, prompt: str) -> Generator[str, None, None]: + """ + Generate output from scratch using the full prompt with streaming. + Args: + prompt (str): The input prompt string. + Yields: + str: Streaming response chunks. + """ + import torch + + inputs = self.tokenizer([prompt], return_tensors="pt").to(self.model.device) + + # Get generation parameters + max_new_tokens = getattr(self.config, "max_tokens", 128) + remove_think_prefix = getattr(self.config, "remove_think_prefix", False) + + # Manual streaming generation + generated_ids = inputs.input_ids.clone() + accumulated_text = "" + + for _ in range(max_new_tokens): + # Forward pass + with torch.no_grad(): + outputs = self.model( + input_ids=generated_ids, + use_cache=True, + return_dict=True, + ) + + # Get next token logits + next_token_logits = outputs.logits[:, -1, :] + + # Apply logits processors if sampling + if getattr(self.config, "do_sample", True): + batch_size, _ = next_token_logits.size() + dummy_ids = torch.zeros( + (batch_size, 1), dtype=torch.long, device=next_token_logits.device + ) + filtered_logits = self.logits_processors(dummy_ids, next_token_logits) + probs = torch.softmax(filtered_logits, dim=-1) + next_token = torch.multinomial(probs, num_samples=1) + else: + next_token = torch.argmax(next_token_logits, dim=-1, keepdim=True) + + # Check for EOS token + if self._should_stop(next_token): + break + + # Append new token + generated_ids = torch.cat([generated_ids, next_token], dim=-1) + + # Decode and yield the new token + new_token_text = self.tokenizer.decode(next_token[0], skip_special_tokens=True) + if new_token_text: # Only yield non-empty tokens + accumulated_text += new_token_text + + # Apply thinking tag removal if enabled + if remove_think_prefix: + processed_text = remove_thinking_tags(accumulated_text) + # Only yield the difference (new content) + if len(processed_text) > len(accumulated_text) - len(new_token_text): + yield processed_text[len(accumulated_text) - len(new_token_text) :] + else: + yield new_token_text + else: + yield new_token_text + def _generate_with_cache(self, query: str, kv: DynamicCache) -> str: """ Generate output incrementally using an existing KV cache. @@ -113,6 +201,8 @@ def _generate_with_cache(self, query: str, kv: DynamicCache) -> str: Returns: str: Model response. """ + import torch + query_ids = self.tokenizer( query, return_tensors="pt", add_special_tokens=False ).input_ids.to(self.model.device) @@ -137,10 +227,70 @@ def _generate_with_cache(self, query: str, kv: DynamicCache) -> str: else response ) - @torch.no_grad() - def _prefill( - self, input_ids: torch.Tensor, kv: DynamicCache - ) -> tuple[torch.Tensor, DynamicCache]: + def _generate_with_cache_stream( + self, query: str, kv: DynamicCache + ) -> Generator[str, None, None]: + """ + Generate output incrementally using an existing KV cache with streaming. + Args: + query (str): The new user query string. + kv (DynamicCache): The prefilled KV cache. + Yields: + str: Streaming response chunks. + """ + query_ids = self.tokenizer( + query, return_tensors="pt", add_special_tokens=False + ).input_ids.to(self.model.device) + + max_new_tokens = getattr(self.config, "max_tokens", 128) + remove_think_prefix = getattr(self.config, "remove_think_prefix", False) + + # Initial forward pass + logits, kv = self._prefill(query_ids, kv) + next_token = self._select_next_token(logits) + + # Yield first token + first_token_text = self.tokenizer.decode(next_token[0], skip_special_tokens=True) + accumulated_text = "" + if first_token_text: + accumulated_text += first_token_text + if remove_think_prefix: + processed_text = remove_thinking_tags(accumulated_text) + if len(processed_text) > len(accumulated_text) - len(first_token_text): + yield processed_text[len(accumulated_text) - len(first_token_text) :] + else: + yield first_token_text + else: + yield first_token_text + + generated = [next_token] + + # Continue generation + for _ in range(max_new_tokens - 1): + if self._should_stop(next_token): + break + logits, kv = self._prefill(next_token, kv) + next_token = self._select_next_token(logits) + + # Decode and yield the new token + new_token_text = self.tokenizer.decode(next_token[0], skip_special_tokens=True) + if new_token_text: + accumulated_text += new_token_text + + # Apply thinking tag removal if enabled + if remove_think_prefix: + processed_text = remove_thinking_tags(accumulated_text) + # Only yield the difference (new content) + if len(processed_text) > len(accumulated_text) - len(new_token_text): + yield processed_text[len(accumulated_text) - len(new_token_text) :] + else: + yield new_token_text + else: + yield new_token_text + + generated.append(next_token) + + def _prefill(self, input_ids: Any, kv: DynamicCache) -> tuple[Any, DynamicCache]: """ Forward the model once, returning last-step logits and updated KV cache. Args: @@ -149,15 +299,18 @@ def _prefill( Returns: tuple[torch.Tensor, DynamicCache]: (last-step logits, updated KV cache) """ - out = self.model( - input_ids=input_ids, - use_cache=True, - past_key_values=kv, - return_dict=True, - ) + import torch + + with torch.no_grad(): + out = self.model( + input_ids=input_ids, + use_cache=True, + past_key_values=kv, + return_dict=True, + ) return out.logits[:, -1, :], out.past_key_values - def _select_next_token(self, logits: torch.Tensor) -> torch.Tensor: + def _select_next_token(self, logits: Any) -> Any: """ Select the next token from logits using sampling or argmax, depending on config. Args: @@ -165,6 +318,8 @@ def _select_next_token(self, logits: torch.Tensor) -> torch.Tensor: Returns: torch.Tensor: Selected token ID(s). """ + import torch + if getattr(self.config, "do_sample", True): batch_size, _ = logits.size() dummy_ids = torch.zeros((batch_size, 1), dtype=torch.long, device=logits.device) @@ -173,7 +328,7 @@ def _select_next_token(self, logits: torch.Tensor) -> torch.Tensor: return torch.multinomial(probs, num_samples=1) return torch.argmax(logits, dim=-1, keepdim=True) - def _should_stop(self, token: torch.Tensor) -> bool: + def _should_stop(self, token: Any) -> bool: """ Check if the given token is the EOS (end-of-sequence) token. Args: @@ -197,6 +352,8 @@ def build_kv_cache(self, messages) -> DynamicCache: Returns: DynamicCache: The constructed KV cache object. """ + import torch + # Accept multiple input types and convert to standard chat messages if isinstance(messages, str): messages = [ diff --git a/src/memos/llms/hf_singleton.py b/src/memos/llms/hf_singleton.py new file mode 100644 index 000000000..af0b6deab --- /dev/null +++ b/src/memos/llms/hf_singleton.py @@ -0,0 +1,114 @@ +import threading + +from typing import ClassVar + +from memos.configs.llm import HFLLMConfig +from memos.llms.hf import HFLLM +from memos.log import get_logger + + +logger = get_logger(__name__) + + +class HFSingletonLLM(HFLLM): + """ + Singleton version of HFLLM that prevents multiple loading of the same model. + This class inherits from HFLLM and adds singleton behavior. + """ + + _instances: ClassVar[dict[str, "HFSingletonLLM"]] = {} + _lock: ClassVar[threading.Lock] = threading.Lock() + + def __new__(cls, config: HFLLMConfig): + """ + Singleton pattern implementation. + Returns existing instance if config already exists, otherwise creates new one. + """ + config_key = cls._get_config_key(config) + + if config_key in cls._instances: + logger.debug(f"Reusing existing HF model: {config.model_name_or_path}") + return cls._instances[config_key] + + with cls._lock: + # Double-check pattern to prevent race conditions + if config_key in cls._instances: + logger.debug(f"Reusing existing HF model: {config.model_name_or_path}") + return cls._instances[config_key] + + logger.info(f"Creating new HF model: {config.model_name_or_path}") + instance = super().__new__(cls) + cls._instances[config_key] = instance + return instance + + def __init__(self, config: HFLLMConfig): + """ + Initialize the singleton HFLLM instance. + Only initializes if this is a new instance. + """ + # Check if already initialized + if hasattr(self, "_initialized"): + return + + # Call parent constructor + super().__init__(config) + self._initialized = True + + @classmethod + def _get_config_key(cls, config: HFLLMConfig) -> str: + """ + Generate a unique key for the HF model configuration. + + Args: + config: The HFLLM configuration + + Returns: + A unique string key representing the configuration + """ + # Create a unique key based on model path and key parameters + key_parts = [config.model_name_or_path] + return "|".join(key_parts) + + @classmethod + def get_instance_count(cls) -> int: + """ + Get the number of unique HF model instances currently managed. + + Returns: + Number of HF model instances + """ + return len(cls._instances) + + @classmethod + def get_instance_info(cls) -> dict[str, str]: + """ + Get information about all managed HF model instances. + + Returns: + Dictionary mapping config keys to model paths + """ + return {key: instance.config.model_name_or_path for key, instance in cls._instances.items()} + + @classmethod + def clear_all(cls) -> None: + """ + Clear all HF model instances from memory. + This should be used carefully as it will force reloading of models. + """ + with cls._lock: + cls._instances.clear() + logger.info("All HF model instances cleared from singleton manager") + + +# Convenience function to get singleton manager info +def get_hf_singleton_info() -> dict[str, int]: + """ + Get information about the HF singleton manager. + + Returns: + Dictionary with instance count and info + """ + return { + "instance_count": HFSingletonLLM.get_instance_count(), + "instance_info": HFSingletonLLM.get_instance_info(), + } diff --git a/src/memos/llms/ollama.py b/src/memos/llms/ollama.py index 00e230789..050b7a253 100644 --- a/src/memos/llms/ollama.py +++ b/src/memos/llms/ollama.py @@ -1,3 +1,4 @@ +from collections.abc import Generator from typing import Any from ollama import Client @@ -80,3 +81,6 @@ def generate(self, messages: MessageList) -> Any: return remove_thinking_tags(str_response) else: return str_response + + def generate_stream(self, messages: MessageList, **kwargs) -> Generator[str, None, None]: + raise NotImplementedError diff --git a/src/memos/llms/openai.py b/src/memos/llms/openai.py index 492b9069f..148f6a2ca 100644 --- a/src/memos/llms/openai.py +++ b/src/memos/llms/openai.py @@ -1,6 +1,8 @@ +from collections.abc import Generator + import openai -from memos.configs.llm import OpenAILLMConfig +from memos.configs.llm import AzureLLMConfig, OpenAILLMConfig from memos.llms.base import BaseLLM from memos.llms.utils import remove_thinking_tags from memos.log import get_logger @@ -33,3 +35,67 @@ def generate(self, messages: MessageList) -> str: return remove_thinking_tags(response_content) else: return response_content + + def generate_stream(self, messages: MessageList, **kwargs) -> Generator[str, None, None]: + """Stream response from OpenAI LLM with optional reasoning support.""" + response = self.client.chat.completions.create( + model=self.config.model_name_or_path, + messages=messages, + stream=True, + temperature=self.config.temperature, + max_tokens=self.config.max_tokens, + top_p=self.config.top_p, + extra_body=self.config.extra_body, + ) + + reasoning_started = False + + for chunk in response: + delta = chunk.choices[0].delta + + # Support for custom 'reasoning_content' (if present in OpenAI-compatible models like Qwen) + if hasattr(delta, "reasoning_content") and delta.reasoning_content: + if not reasoning_started and not self.config.remove_think_prefix: + yield "" + reasoning_started = True + yield delta.reasoning_content + elif hasattr(delta, "content") and delta.content: + if reasoning_started and not self.config.remove_think_prefix: + yield "" + reasoning_started = False + yield delta.content + + # Ensure we close the block if not already done + if reasoning_started and not self.config.remove_think_prefix: + yield "" + + +class AzureLLM(BaseLLM): + """Azure OpenAI LLM class.""" + + def __init__(self, config: AzureLLMConfig): + self.config = config + self.client = openai.AzureOpenAI( + azure_endpoint=config.base_url, + api_version=config.api_version, + api_key=config.api_key, + ) + + def generate(self, messages: MessageList) -> str: + """Generate a response from Azure OpenAI LLM.""" + response = self.client.chat.completions.create( + model=self.config.model_name_or_path, + messages=messages, + temperature=self.config.temperature, + max_tokens=self.config.max_tokens, + top_p=self.config.top_p, + ) + logger.info(f"Response from Azure OpenAI: {response.model_dump_json()}") + response_content = response.choices[0].message.content + if self.config.remove_think_prefix: + return remove_thinking_tags(response_content) + else: + return response_content + + def generate_stream(self, messages: MessageList, **kwargs) -> Generator[str, None, None]: + raise NotImplementedError diff --git a/src/memos/llms/qwen.py b/src/memos/llms/qwen.py new file mode 100644 index 000000000..a47fcdf36 --- /dev/null +++ b/src/memos/llms/qwen.py @@ -0,0 +1,63 @@ +from collections.abc import Generator + +from memos.configs.llm import QwenLLMConfig +from memos.llms.openai import OpenAILLM +from memos.llms.utils import remove_thinking_tags +from memos.log import get_logger +from memos.types import MessageList + + +logger = get_logger(__name__) + + +class QwenLLM(OpenAILLM): + """Qwen (DashScope) LLM class via OpenAI-compatible API.""" + + def __init__(self, config: QwenLLMConfig): + super().__init__(config) + + def generate(self, messages: MessageList) -> str: + """Generate a response from Qwen LLM.""" + response = self.client.chat.completions.create( + model=self.config.model_name_or_path, + messages=messages, + extra_body=self.config.extra_body, + temperature=self.config.temperature, + max_tokens=self.config.max_tokens, + top_p=self.config.top_p, + ) + logger.info(f"Response from Qwen: {response.model_dump_json()}") + response_content = response.choices[0].message.content + if self.config.remove_think_prefix: + return remove_thinking_tags(response_content) + else: + return response_content + + def generate_stream(self, messages: MessageList, **kwargs) -> Generator[str, None, None]: + """Stream response from Qwen LLM.""" + response = self.client.chat.completions.create( + model=self.config.model_name_or_path, + messages=messages, + stream=True, + temperature=self.config.temperature, + max_tokens=self.config.max_tokens, + top_p=self.config.top_p, + extra_body=self.config.extra_body, + ) + + reasoning_started = False + for chunk in response: + delta = chunk.choices[0].delta + + # Some models may have separate `reasoning_content` vs `content` + # For Qwen (DashScope), likely only `content` is used + if hasattr(delta, "reasoning_content") and delta.reasoning_content: + if not reasoning_started and not self.config.remove_think_prefix: + yield "" + reasoning_started = True + yield delta.reasoning_content + elif hasattr(delta, "content") and delta.content: + if reasoning_started and not self.config.remove_think_prefix: + yield "" + reasoning_started = False + yield delta.content diff --git a/src/memos/llms/vllm.py b/src/memos/llms/vllm.py new file mode 100644 index 000000000..167569a40 --- /dev/null +++ b/src/memos/llms/vllm.py @@ -0,0 +1,153 @@ +from typing import Any, cast + +from memos.configs.llm import VLLMLLMConfig +from memos.llms.base import BaseLLM +from memos.llms.utils import remove_thinking_tags +from memos.log import get_logger +from memos.types import MessageDict + + +logger = get_logger(__name__) + + +class VLLMLLM(BaseLLM): + """ + VLLM LLM class for connecting to existing vLLM servers. + """ + + def __init__(self, config: VLLMLLMConfig): + """ + Initialize the VLLM LLM to connect to an existing vLLM server. + """ + self.config = config + + # Initialize OpenAI client for API calls + self.client = None + api_key = getattr(self.config, "api_key", "dummy") + if not api_key: + api_key = "dummy" + + import openai + + self.client = openai.Client( + api_key=api_key, base_url=getattr(self.config, "api_base", "http://localhost:8088/v1") + ) + + def build_vllm_kv_cache(self, messages: Any) -> str: + """ + Build a KV cache from chat messages via one vLLM request. + Handles str, list[str], and MessageList formats. + """ + # 1. Normalize input to a MessageList + processed_messages: list[MessageDict] = [] + if isinstance(messages, str): + processed_messages = [ + { + "role": "system", + "content": f"Below is some information about the user.\n{messages}", + } + ] + elif isinstance(messages, list): + if not messages: + pass # Empty list + elif isinstance(messages[0], str): + str_content = " ".join(str(msg) for msg in messages) + processed_messages = [ + { + "role": "system", + "content": f"Below is some information about the user.\n{str_content}", + } + ] + elif isinstance(messages[0], dict): + processed_messages = cast("list[MessageDict]", messages) + + # 2. Convert to prompt for logging/return value. + prompt = self._messages_to_prompt(processed_messages) + + if not prompt.strip(): + raise ValueError("Prompt is empty, cannot build KV cache.") + + # 3. Send request to vLLM server to preload the KV cache + if self.client: + try: + # Use the processed messages for the API call + prefill_kwargs = { + "model": self.config.model_name_or_path, + "messages": processed_messages, + "max_tokens": 2, + "temperature": 0.0, + "top_p": 1.0, + } + self.client.chat.completions.create(**prefill_kwargs) + logger.info(f"vLLM KV cache prefill completed for prompt: '{prompt[:100]}...'") + except Exception as e: + logger.warning(f"Failed to prefill vLLM KV cache: {e}") + + return prompt + + def generate(self, messages: list[MessageDict]) -> str: + """ + Generate a response from the model. + """ + if self.client: + return self._generate_with_api_client(messages) + else: + raise RuntimeError("API client is not available") + + def _generate_with_api_client(self, messages: list[MessageDict]) -> str: + """ + Generate response using vLLM API client. + """ + if self.client: + completion_kwargs = { + "model": self.config.model_name_or_path, + "messages": messages, + "temperature": float(getattr(self.config, "temperature", 0.8)), + "max_tokens": int(getattr(self.config, "max_tokens", 1024)), + "top_p": float(getattr(self.config, "top_p", 0.9)), + } + + response = self.client.chat.completions.create(**completion_kwargs) + response_text = response.choices[0].message.content or "" + logger.info(f"VLLM API response: {response_text}") + return ( + remove_thinking_tags(response_text) + if getattr(self.config, "remove_think_prefix", False) + else response_text + ) + else: + raise RuntimeError("API client is not available") + + def _messages_to_prompt(self, messages: list[MessageDict]) -> str: + """ + Convert messages to prompt string. + """ + prompt_parts = [] + for msg in messages: + role = msg["role"] + content = msg["content"] + prompt_parts.append(f"{role.capitalize()}: {content}") + return "\n".join(prompt_parts) + + def generate_stream(self, messages: list[MessageDict]): + """ + Generate a response from the model using streaming. + Yields content chunks as they are received. + """ + if self.client: + completion_kwargs = { + "model": self.config.model_name_or_path, + "messages": messages, + "temperature": float(getattr(self.config, "temperature", 0.8)), + "max_tokens": int(getattr(self.config, "max_tokens", 1024)), + "top_p": float(getattr(self.config, "top_p", 0.9)), + "stream": True, # Enable streaming + } + + stream = self.client.chat.completions.create(**completion_kwargs) + for chunk in stream: + content = chunk.choices[0].delta.content + if content: + yield content + else: + raise RuntimeError("API client is not available") diff --git a/src/memos/mem_cube/general.py b/src/memos/mem_cube/general.py index d44aab915..7217c354b 100644 --- a/src/memos/mem_cube/general.py +++ b/src/memos/mem_cube/general.py @@ -1,11 +1,13 @@ import os +from typing import Literal + from memos.configs.mem_cube import GeneralMemCubeConfig from memos.configs.utils import get_json_file_model_schema from memos.exceptions import ConfigurationError, MemCubeError from memos.log import get_logger from memos.mem_cube.base import BaseMemCube -from memos.mem_cube.utils import download_repo +from memos.mem_cube.utils import download_repo, merge_config_with_default from memos.memories.activation.base import BaseActMemory from memos.memories.factory import MemoryFactory from memos.memories.parametric.base import BaseParaMemory @@ -37,10 +39,15 @@ def __init__(self, config: GeneralMemCubeConfig): else None ) - def load(self, dir: str) -> None: + def load( + self, dir: str, memory_types: list[Literal["text_mem", "act_mem", "para_mem"]] | None = None + ) -> None: """Load memories. Args: dir (str): The directory containing the memory files. + memory_types (list[str], optional): List of memory types to load. + If None, loads all available memory types. + Options: ["text_mem", "act_mem", "para_mem"] """ loaded_schema = get_json_file_model_schema(os.path.join(dir, self.config.config_filename)) if loaded_schema != self.config.model_schema: @@ -48,60 +55,114 @@ def load(self, dir: str) -> None: f"Configuration schema mismatch. Expected {self.config.model_schema}, " f"but found {loaded_schema}." ) - self.text_mem.load(dir) if self.text_mem else None - self.act_mem.load(dir) if self.act_mem else None - self.para_mem.load(dir) if self.para_mem else None - logger.info(f"MemCube loaded successfully from {dir}") + # If no specific memory types specified, load all + if memory_types is None: + memory_types = ["text_mem", "act_mem", "para_mem"] + + # Load specified memory types + if "text_mem" in memory_types and self.text_mem: + self.text_mem.load(dir) + logger.debug(f"Loaded text_mem from {dir}") + + if "act_mem" in memory_types and self.act_mem: + self.act_mem.load(dir) + logger.info(f"Loaded act_mem from {dir}") - def dump(self, dir: str) -> None: + if "para_mem" in memory_types and self.para_mem: + self.para_mem.load(dir) + logger.info(f"Loaded para_mem from {dir}") + + logger.info(f"MemCube loaded successfully from {dir} (types: {memory_types})") + + def dump( + self, dir: str, memory_types: list[Literal["text_mem", "act_mem", "para_mem"]] | None = None + ) -> None: """Dump memories. Args: dir (str): The directory where the memory files will be saved. + memory_types (list[str], optional): List of memory types to dump. + If None, dumps all available memory types. + Options: ["text_mem", "act_mem", "para_mem"] """ if os.path.exists(dir) and os.listdir(dir): raise MemCubeError( f"Directory {dir} is not empty. Please provide an empty directory for dumping." ) + # Always dump config self.config.to_json_file(os.path.join(dir, self.config.config_filename)) - self.text_mem.dump(dir) if self.text_mem else None - self.act_mem.dump(dir) if self.act_mem else None - self.para_mem.dump(dir) if self.para_mem else None - logger.info(f"MemCube dumped successfully to {dir}") + # If no specific memory types specified, dump all + if memory_types is None: + memory_types = ["text_mem", "act_mem", "para_mem"] + + # Dump specified memory types + if "text_mem" in memory_types and self.text_mem: + self.text_mem.dump(dir) + logger.info(f"Dumped text_mem to {dir}") + + if "act_mem" in memory_types and self.act_mem: + self.act_mem.dump(dir) + logger.info(f"Dumped act_mem to {dir}") + + if "para_mem" in memory_types and self.para_mem: + self.para_mem.dump(dir) + logger.info(f"Dumped para_mem to {dir}") + + logger.info(f"MemCube dumped successfully to {dir} (types: {memory_types})") @staticmethod - def init_from_dir(dir: str) -> "GeneralMemCube": + def init_from_dir( + dir: str, + memory_types: list[Literal["text_mem", "act_mem", "para_mem"]] | None = None, + default_config: GeneralMemCubeConfig | None = None, + ) -> "GeneralMemCube": """Create a MemCube instance from a MemCube directory. Args: dir (str): The directory containing the memory files. + memory_types (list[str], optional): List of memory types to load. + If None, loads all available memory types. + default_config (GeneralMemCubeConfig, optional): Default configuration to merge with existing config. + If provided, will merge general settings while preserving critical user-specific fields. Returns: MemCube: An instance of MemCube loaded with memories from the specified directory. """ config_path = os.path.join(dir, "config.json") config = GeneralMemCubeConfig.from_json_file(config_path) + + # Merge with default config if provided + if default_config is not None: + config = merge_config_with_default(config, default_config) + logger.info(f"Applied default config to cube {config.cube_id}") + mem_cube = GeneralMemCube(config) - mem_cube.load(dir) + mem_cube.load(dir, memory_types) return mem_cube @staticmethod def init_from_remote_repo( - cube_id: str, base_url: str = "https://huggingface.co/datasets" + cube_id: str, + base_url: str = "https://huggingface.co/datasets", + memory_types: list[Literal["text_mem", "act_mem", "para_mem"]] | None = None, + default_config: GeneralMemCubeConfig | None = None, ) -> "GeneralMemCube": """Create a MemCube instance from a remote repository. Args: - repo (str): The repository name. + cube_id (str): The repository name. base_url (str): The base URL of the remote repository. + memory_types (list[str], optional): List of memory types to load. + If None, loads all available memory types. + default_config (GeneralMemCubeConfig, optional): Default configuration to merge with existing config. Returns: MemCube: An instance of MemCube loaded with memories from the specified remote repository. """ dir = download_repo(cube_id, base_url) - return GeneralMemCube.init_from_dir(dir) + return GeneralMemCube.init_from_dir(dir, memory_types, default_config) @property def text_mem(self) -> "BaseTextMemory | None": diff --git a/src/memos/mem_cube/utils.py b/src/memos/mem_cube/utils.py index c6820b771..0e7afaf39 100644 --- a/src/memos/mem_cube/utils.py +++ b/src/memos/mem_cube/utils.py @@ -1,6 +1,13 @@ +import copy +import logging import subprocess import tempfile +from memos.configs.mem_cube import GeneralMemCubeConfig + + +logger = logging.getLogger(__name__) + def download_repo(repo: str, base_url: str, dir: str | None = None) -> str: """Download a repository from a remote source. @@ -22,3 +29,98 @@ def download_repo(repo: str, base_url: str, dir: str | None = None) -> str: subprocess.run(["git", "clone", repo_url, dir], check=True) return dir + + +def merge_config_with_default( + existing_config: GeneralMemCubeConfig, default_config: GeneralMemCubeConfig +) -> GeneralMemCubeConfig: + """ + Merge existing cube config with default config, preserving critical fields. + + This method updates general configuration fields (like API keys, model parameters) + while preserving critical user-specific fields (like user_id, cube_id, graph_db settings). + + Args: + existing_config (GeneralMemCubeConfig): The existing cube configuration loaded from file + default_config (GeneralMemCubeConfig): The default configuration to merge from + + Returns: + GeneralMemCubeConfig: Merged configuration + """ + + # Convert configs to dictionaries + existing_dict = existing_config.model_dump(mode="json") + default_dict = default_config.model_dump(mode="json") + + logger.info( + f"Starting config merge for user {existing_config.user_id}, cube {existing_config.cube_id}" + ) + + # Define fields that should be preserved from existing config + preserve_fields = {"user_id", "cube_id", "config_filename", "model_schema"} + + # Preserve graph_db from existing config if it exists, but merge some fields + preserved_graph_db = None + if "text_mem" in existing_dict and "text_mem" in default_dict: + existing_text_config = existing_dict["text_mem"].get("config", {}) + default_text_config = default_dict["text_mem"].get("config", {}) + + if "graph_db" in existing_text_config and "graph_db" in default_text_config: + existing_graph_config = existing_text_config["graph_db"]["config"] + default_graph_config = default_text_config["graph_db"]["config"] + + # Define graph_db fields to preserve (user-specific) + preserve_graph_fields = { + "uri", + "user", + "password", + "db_name", + "auto_create", + "user_name", + "use_multi_db", + } + + # Create merged graph_db config + merged_graph_config = copy.deepcopy(existing_graph_config) + for key, value in default_graph_config.items(): + if key not in preserve_graph_fields: + merged_graph_config[key] = value + logger.debug( + f"Updated graph_db field '{key}': {existing_graph_config.get(key)} -> {value}" + ) + if not default_graph_config.get("use_multi_db", True): + # set original use_multi_db to False if default_graph_config.use_multi_db is False + if merged_graph_config.get("use_multi_db", True): + merged_graph_config["use_multi_db"] = False + merged_graph_config["user_name"] = merged_graph_config.get("db_name") + merged_graph_config["db_name"] = default_graph_config.get("db_name") + else: + logger.info("use_multi_db is already False, no need to change") + + preserved_graph_db = { + "backend": existing_text_config["graph_db"]["backend"], + "config": merged_graph_config, + } + + # Use default config as base + merged_dict = copy.deepcopy(default_dict) + + # Restore preserved fields from existing config + for field in preserve_fields: + if field in existing_dict: + merged_dict[field] = existing_dict[field] + logger.debug(f"Preserved field '{field}': {existing_dict[field]}") + + # Restore graph_db if it was preserved + if preserved_graph_db and "text_mem" in merged_dict: + merged_dict["text_mem"]["config"]["graph_db"] = preserved_graph_db + logger.debug(f"Preserved graph_db with merged config: {preserved_graph_db}") + + # Create new config from merged dictionary + merged_config = GeneralMemCubeConfig.model_validate(merged_dict) + + logger.info( + f"Successfully merged cube config for user {merged_config.user_id}, cube {merged_config.cube_id}" + ) + + return merged_config diff --git a/src/memos/mem_os/core.py b/src/memos/mem_os/core.py index 07fa4ba97..d09380332 100644 --- a/src/memos/mem_os/core.py +++ b/src/memos/mem_os/core.py @@ -1,4 +1,6 @@ +import json import os +import uuid from datetime import datetime from pathlib import Path @@ -11,7 +13,11 @@ from memos.mem_cube.general import GeneralMemCube from memos.mem_reader.factory import MemReaderFactory from memos.mem_scheduler.general_scheduler import GeneralScheduler -from memos.mem_scheduler.modules.schemas import ANSWER_LABEL, QUERY_LABEL, ScheduleMessageItem +from memos.mem_scheduler.modules.schemas import ( + ADD_LABEL, + ANSWER_LABEL, + ScheduleMessageItem, +) from memos.mem_scheduler.scheduler_factory import SchedulerFactory from memos.mem_user.user_manager import UserManager, UserRole from memos.memories.activation.item import ActivationMemoryItem @@ -30,7 +36,7 @@ class MOSCore: MOSCore acts as an operating system layer for handling and orchestrating MemCube instances. """ - def __init__(self, config: MOSConfig): + def __init__(self, config: MOSConfig, user_manager: UserManager | None = None): self.config = config self.user_id = config.user_id self.session_id = config.session_id @@ -39,7 +45,12 @@ def __init__(self, config: MOSConfig): self.mem_reader = MemReaderFactory.from_config(config.mem_reader) self.chat_history_manager: dict[str, ChatHistory] = {} self._register_chat_history() - self.user_manager = UserManager(user_id=self.user_id if self.user_id else "root") + + # Use provided user_manager or create a new one + if user_manager is not None: + self.user_manager = user_manager + else: + self.user_manager = UserManager(user_id=self.user_id if self.user_id else "root") # Validate user exists if not self.user_manager.validate_user(self.user_id): @@ -50,7 +61,7 @@ def __init__(self, config: MOSConfig): # Lazy initialization marker self._mem_scheduler_lock = Lock() self.enable_mem_scheduler = self.config.get("enable_mem_scheduler", False) - self._mem_scheduler = None + self._mem_scheduler: GeneralScheduler = None logger.info(f"MOS initialized for user: {self.user_id}") @property @@ -58,6 +69,7 @@ def mem_scheduler(self) -> GeneralScheduler: """Lazy-loaded property for memory scheduler.""" if self.enable_mem_scheduler and self._mem_scheduler is None: self._initialize_mem_scheduler() + self._mem_scheduler.mem_cubes = self.mem_cubes return self._mem_scheduler @mem_scheduler.setter @@ -74,6 +86,7 @@ def mem_scheduler(self, value: GeneralScheduler | None) -> None: raise TypeError(f"Expected GeneralScheduler or None, got {type(value)}") self._mem_scheduler = value + self._mem_scheduler.mem_cubes = self.mem_cubes if value: logger.info("Memory scheduler manually set") @@ -92,7 +105,18 @@ def _initialize_mem_scheduler(self): logger.info("Initializing memory scheduler...") scheduler_config = self.config.mem_scheduler self._mem_scheduler = SchedulerFactory.from_config(scheduler_config) - self._mem_scheduler.initialize_modules(chat_llm=self.chat_llm) + # Validate required components + if not hasattr(self.mem_reader, "llm"): + raise AttributeError( + f"Memory reader of type {type(self.mem_reader).__name__} " + "missing required 'llm' attribute" + ) + self._mem_scheduler.initialize_modules(chat_llm=self.chat_llm) + else: + # Configure scheduler modules + self._mem_scheduler.initialize_modules( + chat_llm=self.chat_llm, process_llm=self.mem_reader.llm + ) self._mem_scheduler.start() def mem_scheduler_on(self) -> bool: @@ -123,6 +147,17 @@ def mem_scheduler_off(self) -> bool: logger.error(f"Failed to stop scheduler: {e!s}") return False + def mem_reorganizer_on(self) -> bool: + pass + + def mem_reorganizer_off(self) -> bool: + """temporally implement""" + for mem_cube in self.mem_cubes.values(): + logger.info(f"try to close reorganizer for {mem_cube.text_mem.config.cube_id}") + if mem_cube.text_mem and mem_cube.text_mem.is_reorganize: + logger.info(f"close reorganizer for {mem_cube.text_mem.config.cube_id}") + mem_cube.text_mem.memory_manager.close() + def _register_chat_history(self, user_id: str | None = None) -> None: """Initialize chat history with user ID.""" if user_id is None: @@ -186,12 +221,16 @@ def _get_all_documents(self, path: str) -> list[str]: documents.append(str(file_path)) return documents - def chat(self, query: str, user_id: str | None = None) -> str: + def chat(self, query: str, user_id: str | None = None, base_prompt: str | None = None) -> str: """ Chat with the MOS. Args: query (str): The user's query. + user_id (str, optional): The user ID for the chat session. Defaults to the user ID from the config. + base_prompt (str, optional): A custom base prompt to use for the chat. + It can be a template string with a `{memories}` placeholder. + If not provided, a default prompt is used. Returns: str: The response from the MOS. @@ -218,7 +257,7 @@ def chat(self, query: str, user_id: str | None = None) -> str: user_id=target_user_id, mem_cube_id=mem_cube_id, mem_cube=mem_cube, - label=QUERY_LABEL, + label=ADD_LABEL, content=query, timestamp=datetime.now(), ) @@ -227,9 +266,9 @@ def chat(self, query: str, user_id: str | None = None) -> str: memories = mem_cube.text_mem.search(query, top_k=self.config.top_k) memories_all.extend(memories) logger.info(f"🧠 [Memory] Searched memories:\n{self._str_memories(memories_all)}\n") - system_prompt = self._build_system_prompt(memories_all) + system_prompt = self._build_system_prompt(memories_all, base_prompt=base_prompt) else: - system_prompt = self._build_system_prompt() + system_prompt = self._build_system_prompt(base_prompt=base_prompt) current_messages = [ {"role": "system", "content": system_prompt}, *chat_history.chat_history, @@ -261,8 +300,8 @@ def chat(self, query: str, user_id: str | None = None) -> str: self.chat_history_manager[user_id] = chat_history # submit message to scheduler - if len(accessible_cubes) == 1: - mem_cube_id = accessible_cubes[0].cube_id + for accessible_mem_cube in accessible_cubes: + mem_cube_id = accessible_mem_cube.cube_id mem_cube = self.mem_cubes[mem_cube_id] if self.enable_mem_scheduler and self.mem_scheduler is not None: message_item = ScheduleMessageItem( @@ -277,20 +316,39 @@ def chat(self, query: str, user_id: str | None = None) -> str: return response - def _build_system_prompt(self, memories: list | None = None) -> str: + def _build_system_prompt( + self, + memories: list[TextualMemoryItem] | list[str] | None = None, + base_prompt: str | None = None, + ) -> str: """Build system prompt with optional memories context.""" - base_prompt = ( - "You are a knowledgeable and helpful AI assistant. " - "You have access to conversation memories that help you provide more personalized responses. " - "Use the memories to understand the user's context, preferences, and past interactions. " - "If memories are provided, reference them naturally when relevant, but don't explicitly mention having memories." - ) + if base_prompt is None: + base_prompt = ( + "You are a knowledgeable and helpful AI assistant. " + "You have access to conversation memories that help you provide more personalized responses. " + "Use the memories to understand the user's context, preferences, and past interactions. " + "If memories are provided, reference them naturally when relevant, but don't explicitly mention having memories." + ) + memory_context = "" if memories: - memory_context = "\n\n## Memories:\n" + memory_list = [] for i, memory in enumerate(memories, 1): - memory_context += f"{i}. {memory.memory}\n" - return base_prompt + memory_context + if isinstance(memory, TextualMemoryItem): + text_memory = memory.memory + else: + if not isinstance(memory, str): + logger.error("Unexpected memory type.") + text_memory = memory + memory_list.append(f"{i}. {text_memory}") + memory_context = "\n".join(memory_list) + + if "{memories}" in base_prompt: + return base_prompt.format(memories=memory_context) + elif memories: + # For backward compatibility, append memories if no placeholder is found + memory_context_with_header = "\n\n## Memories:\n" + memory_context + return base_prompt + memory_context_with_header return base_prompt def _str_memories( @@ -364,7 +422,10 @@ def create_cube_for_user( return self.user_manager.create_cube(cube_name, owner_id, cube_path, cube_id) def register_mem_cube( - self, mem_cube_name_or_path: str, mem_cube_id: str | None = None, user_id: str | None = None + self, + mem_cube_name_or_path: str | GeneralMemCube, + mem_cube_id: str | None = None, + user_id: str | None = None, ) -> None: """ Register a MemCube with the MOS. @@ -377,12 +438,18 @@ def register_mem_cube( self._validate_user_exists(target_user_id) if mem_cube_id is None: - mem_cube_id = mem_cube_name_or_path + if isinstance(mem_cube_name_or_path, GeneralMemCube): + mem_cube_id = f"cube_{target_user_id}" + else: + mem_cube_id = mem_cube_name_or_path if mem_cube_id in self.mem_cubes: logger.info(f"MemCube with ID {mem_cube_id} already in MOS, skip install.") else: - if os.path.exists(mem_cube_name_or_path): + if isinstance(mem_cube_name_or_path, GeneralMemCube): + self.mem_cubes[mem_cube_id] = mem_cube_name_or_path + logger.info(f"register new cube {mem_cube_id} for user {target_user_id}") + elif os.path.exists(mem_cube_name_or_path): self.mem_cubes[mem_cube_id] = GeneralMemCube.init_from_dir(mem_cube_name_or_path) else: logger.warning( @@ -394,6 +461,14 @@ def register_mem_cube( # Check if cube already exists in database existing_cube = self.user_manager.get_cube(mem_cube_id) + # check the embedder is it consistent with MOSConfig + if self.config.mem_reader.config.embedder != ( + cube_embedder := self.mem_cubes[mem_cube_id].text_mem.config.embedder + ): + logger.warning( + f"Cube Embedder is not consistent with MOSConfig for cube: {mem_cube_id}, will use Cube Embedder: {cube_embedder}" + ) + if existing_cube: # Cube exists, just add user to cube if not already associated if not self.user_manager.validate_user_cube_access(target_user_id, mem_cube_id): @@ -407,10 +482,14 @@ def register_mem_cube( else: # Cube doesn't exist, create it self.create_cube_for_user( - cube_name=mem_cube_name_or_path, + cube_name=mem_cube_name_or_path + if not isinstance(mem_cube_name_or_path, GeneralMemCube) + else mem_cube_id, owner_id=target_user_id, cube_id=mem_cube_id, - cube_path=mem_cube_name_or_path, + cube_path=mem_cube_name_or_path + if not isinstance(mem_cube_name_or_path, GeneralMemCube) + else "init", ) logger.info(f"register new cube {mem_cube_id} for user {target_user_id}") @@ -427,7 +506,11 @@ def unregister_mem_cube(self, mem_cube_id: str, user_id: str | None = None) -> N raise ValueError(f"MemCube with ID {mem_cube_id} does not exist.") def search( - self, query: str, user_id: str | None = None, install_cube_ids: list[str] | None = None + self, + query: str, + user_id: str | None = None, + install_cube_ids: list[str] | None = None, + top_k: int | None = None, ) -> MOSSearchResult: """ Search for textual memories across all registered MemCubes. @@ -464,18 +547,10 @@ def search( and (mem_cube.text_mem is not None) and self.config.enable_textual_memory ): - memories = mem_cube.text_mem.search(query, top_k=self.config.top_k) - result["text_mem"].append({"cube_id": mem_cube_id, "memories": memories}) - logger.info( - f"🧠 [Memory] Searched memories from {mem_cube_id}:\n{self._str_memories(memories)}\n" + memories = mem_cube.text_mem.search( + query, top_k=top_k if top_k else self.config.top_k ) - if ( - (mem_cube_id in install_cube_ids) - and (mem_cube.act_mem is not None) - and self.config.enable_activation_memory - ): - memories = mem_cube.act_mem.extract(query) - result["act_mem"].append({"cube_id": mem_cube_id, "memories": [memories]}) + result["text_mem"].append({"cube_id": mem_cube_id, "memories": memories}) logger.info( f"🧠 [Memory] Searched memories from {mem_cube_id}:\n{self._str_memories(memories)}\n" ) @@ -538,10 +613,25 @@ def add( memories = self.mem_reader.get_memory( messages_list, type="chat", - info={"user_id": target_user_id, "session_id": self.session_id}, + info={"user_id": target_user_id, "session_id": str(uuid.uuid4())}, ) for mem in memories: self.mem_cubes[mem_cube_id].text_mem.add(mem) + + # submit messages for scheduler + mem_cube = self.mem_cubes[mem_cube_id] + if self.enable_mem_scheduler and self.mem_scheduler is not None: + text_messages = [message["content"] for message in messages] + message_item = ScheduleMessageItem( + user_id=target_user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + label=ADD_LABEL, + content=json.dumps(text_messages), + timestamp=datetime.now(), + ) + self.mem_scheduler.submit_messages(messages=[message_item]) + if ( (memory_content is not None) and self.config.enable_textual_memory @@ -567,7 +657,7 @@ def add( memories = self.mem_reader.get_memory( messages_list, type="chat", - info={"user_id": target_user_id, "session_id": self.session_id}, + info={"user_id": target_user_id, "session_id": str(uuid.uuid4())}, ) for mem in memories: self.mem_cubes[mem_cube_id].text_mem.add(mem) @@ -580,7 +670,7 @@ def add( doc_memory = self.mem_reader.get_memory( documents, type="doc", - info={"user_id": target_user_id, "session_id": self.session_id}, + info={"user_id": target_user_id, "session_id": str(uuid.uuid4())}, ) for mem in doc_memory: self.mem_cubes[mem_cube_id].text_mem.add(mem) diff --git a/src/memos/mem_os/main.py b/src/memos/mem_os/main.py index 2c479b452..edb3a3333 100644 --- a/src/memos/mem_os/main.py +++ b/src/memos/mem_os/main.py @@ -1,5 +1,6 @@ import concurrent.futures import json +import os from typing import Any @@ -7,6 +8,7 @@ from memos.llms.factory import LLMFactory from memos.log import get_logger from memos.mem_os.core import MOSCore +from memos.mem_os.utils.default_config import get_default from memos.memories.textual.base import BaseTextMemory from memos.templates.mos_prompts import ( COT_DECOMPOSE_PROMPT, @@ -24,20 +26,94 @@ class MOS(MOSCore): This class maintains backward compatibility with the original MOS interface. """ - def __init__(self, config: MOSConfig): + def __init__(self, config: MOSConfig | None = None): + """ + Initialize MOS with optional automatic configuration. + + Args: + config (MOSConfig, optional): MOS configuration. If None, will use automatic configuration from environment variables. + """ + if config is None: + # Auto-configure if no config provided + config, default_cube = self._auto_configure() + self._auto_registered_cube = default_cube + else: + self._auto_registered_cube = None + self.enable_cot = config.PRO_MODE if config.PRO_MODE: print(PRO_MODE_WELCOME_MESSAGE) logger.info(PRO_MODE_WELCOME_MESSAGE) super().__init__(config) - def chat(self, query: str, user_id: str | None = None) -> str: + # Auto-register cube if one was created + if self._auto_registered_cube is not None: + self.register_mem_cube(self._auto_registered_cube) + logger.info( + f"Auto-registered default cube: {self._auto_registered_cube.config.cube_id}" + ) + + def _auto_configure(self, **kwargs) -> tuple[MOSConfig, Any]: + """ + Automatically configure MOS with default settings. + + Returns: + tuple[MOSConfig, Any]: MOS configuration and default MemCube + """ + # Get configuration from environment variables + openai_api_key = os.getenv("OPENAI_API_KEY") + openai_api_base = os.getenv("OPENAI_API_BASE", "https://api.openai.com/v1") + text_mem_type = os.getenv("MOS_TEXT_MEM_TYPE", "general_text") + + if not openai_api_key: + raise ValueError("OPENAI_API_KEY environment variable is required") + + logger.info(f"Auto-configuring MOS with text_mem_type: {text_mem_type}") + return get_default( + openai_api_key=openai_api_key, + openai_api_base=openai_api_base, + text_mem_type=text_mem_type, + ) + + @classmethod + def simple(cls) -> "MOS": + """ + Create a MOS instance with automatic configuration from environment variables. + + This is the simplest way to get started with MemOS. + + Environment variables needed: + - OPENAI_API_KEY: Your OpenAI API key + - OPENAI_API_BASE: OpenAI API base URL (optional, defaults to "https://api.openai.com/v1") + - MOS_TEXT_MEM_TYPE: Text memory type (optional, defaults to "general_text") + + Returns: + MOS: Configured MOS instance with auto-registered default cube + + Example: + ```python + # Set environment variables + export OPENAI_API_KEY="your-api-key" + export MOS_TEXT_MEM_TYPE="general_text" + + # Then use + memory = MOS.simple() + memory.add_memory("Hello world!") + response = memory.chat("What did I just say?") + ``` + """ + return cls() + + def chat(self, query: str, user_id: str | None = None, base_prompt: str | None = None) -> str: """ Enhanced chat method with optional CoT (Chain of Thought) enhancement. Args: query (str): The user's query. user_id (str, optional): User ID for context. + base_prompt (str, optional): A custom base prompt to use for the chat. + It can be a template string with a `{memories}` placeholder. + If not provided, a default prompt is used. Returns: str: The response from the MOS. @@ -46,12 +122,14 @@ def chat(self, query: str, user_id: str | None = None) -> str: if not self.enable_cot: # Use the original chat method from core - return super().chat(query, user_id) + return super().chat(query, user_id, base_prompt=base_prompt) # Enhanced chat with CoT decomposition - return self._chat_with_cot_enhancement(query, user_id) + return self._chat_with_cot_enhancement(query, user_id, base_prompt=base_prompt) - def _chat_with_cot_enhancement(self, query: str, user_id: str | None = None) -> str: + def _chat_with_cot_enhancement( + self, query: str, user_id: str | None = None, base_prompt: str | None = None + ) -> str: """ Chat with CoT enhancement for complex query decomposition. This method includes all the same validation and processing logic as the core chat method. @@ -84,7 +162,7 @@ def _chat_with_cot_enhancement(self, query: str, user_id: str | None = None) -> # Check if the query is complex and needs decomposition if not decomposition_result.get("is_complex", False): logger.info("🔍 [CoT] Query is not complex, using standard chat") - return super().chat(query, user_id) + return super().chat(query, user_id, base_prompt=base_prompt) sub_questions = decomposition_result.get("sub_questions", []) logger.info(f"🔍 [CoT] Decomposed into {len(sub_questions)} sub-questions") @@ -93,7 +171,7 @@ def _chat_with_cot_enhancement(self, query: str, user_id: str | None = None) -> search_engine = self._get_search_engine_for_cot_with_validation(user_cube_ids) if not search_engine: logger.warning("🔍 [CoT] No search engine available, using standard chat") - return super().chat(query, user_id) + return super().chat(query, user_id, base_prompt=base_prompt) # Step 4: Get answers for sub-questions logger.info("🔍 [CoT] Getting answers for sub-questions...") @@ -115,6 +193,7 @@ def _chat_with_cot_enhancement(self, query: str, user_id: str | None = None) -> chat_history=chat_history, user_id=target_user_id, search_engine=search_engine, + base_prompt=base_prompt, ) # Step 6: Update chat history (same as core method) @@ -149,7 +228,7 @@ def _chat_with_cot_enhancement(self, query: str, user_id: str | None = None) -> except Exception as e: logger.error(f"🔍 [CoT] Error in CoT enhancement: {e}") logger.info("🔍 [CoT] Falling back to standard chat") - return super().chat(query, user_id) + return super().chat(query, user_id, base_prompt=base_prompt) def _get_search_engine_for_cot_with_validation( self, user_cube_ids: list[str] @@ -183,6 +262,7 @@ def _generate_enhanced_response_with_context( chat_history: Any, user_id: str | None = None, search_engine: BaseTextMemory | None = None, + base_prompt: str | None = None, ) -> str: """ Generate an enhanced response using sub-questions and their answers, with chat context. @@ -193,6 +273,8 @@ def _generate_enhanced_response_with_context( sub_answers (list[str]): List of answers to sub-questions. chat_history: The user's chat history. user_id (str, optional): User ID for context. + search_engine (BaseTextMemory, optional): Search engine for context retrieval. + base_prompt (str, optional): A custom base prompt for the chat. Returns: str: The enhanced response. @@ -213,10 +295,10 @@ def _generate_enhanced_response_with_context( original_query, top_k=self.config.top_k, mode="fast" ) system_prompt = self._build_system_prompt( - search_memories + search_memories, base_prompt=base_prompt ) # Use the same system prompt builder else: - system_prompt = self._build_system_prompt() + system_prompt = self._build_system_prompt(base_prompt=base_prompt) current_messages = [ {"role": "system", "content": system_prompt + SYNTHESIS_PROMPT.format(qa_text=qa_text)}, *chat_history.chat_history, @@ -261,7 +343,7 @@ def _generate_enhanced_response_with_context( except Exception as e: logger.error(f"🔍 [CoT] Error generating enhanced response: {e}") # Fallback to standard chat - return super().chat(original_query, user_id) + return super().chat(original_query, user_id, base_prompt=base_prompt) @classmethod def cot_decompose( diff --git a/src/memos/mem_os/product.py b/src/memos/mem_os/product.py index b210c30dd..071002f06 100644 --- a/src/memos/mem_os/product.py +++ b/src/memos/mem_os/product.py @@ -1,33 +1,682 @@ import json +import os +import random +import time from collections.abc import Generator -from typing import Literal +from datetime import datetime +from typing import Any, Literal +from dotenv import load_dotenv +from transformers import AutoTokenizer + +from memos.configs.mem_cube import GeneralMemCubeConfig from memos.configs.mem_os import MOSConfig +from memos.log import get_logger +from memos.mem_cube.general import GeneralMemCube from memos.mem_os.core import MOSCore -from memos.memories.activation.item import ActivationMemoryItem -from memos.memories.parametric.item import ParametricMemoryItem -from memos.memories.textual.item import TextualMemoryMetadata, TreeNodeTextualMemoryMetadata +from memos.mem_os.utils.format_utils import ( + convert_graph_to_tree_forworkmem, + filter_nodes_by_tree_ids, + remove_embedding_recursive, + sort_children_by_memory_type, +) +from memos.mem_scheduler.modules.schemas import ANSWER_LABEL, QUERY_LABEL, ScheduleMessageItem +from memos.mem_user.persistent_user_manager import PersistentUserManager +from memos.mem_user.user_manager import UserRole +from memos.memories.textual.item import ( + TextualMemoryItem, +) from memos.types import MessageList +logger = get_logger(__name__) + +load_dotenv() + +CUBE_PATH = os.getenv("MOS_CUBE_PATH", "/tmp/data/") + + class MOSProduct(MOSCore): """ - The MOSProduct class inherits from MOSCore mainly for product usage. + The MOSProduct class inherits from MOSCore and manages multiple users. + Each user has their own configuration and cube access, but shares the same model instances. """ - def __init__(self, config: MOSConfig): - super().__init__(config) + def __init__( + self, + default_config: MOSConfig | None = None, + max_user_instances: int = 100, + default_cube_config: GeneralMemCubeConfig | None = None, + ): + """ + Initialize MOSProduct with an optional default configuration. + + Args: + default_config (MOSConfig | None): Default configuration for new users + max_user_instances (int): Maximum number of user instances to keep in memory + default_cube_config (GeneralMemCubeConfig | None): Default cube configuration for loading cubes + """ + # Initialize with a root config for shared resources + if default_config is None: + # Create a minimal config for root user + root_config = MOSConfig( + user_id="root", + session_id="root_session", + chat_model=default_config.chat_model if default_config else None, + mem_reader=default_config.mem_reader if default_config else None, + enable_mem_scheduler=default_config.enable_mem_scheduler + if default_config + else False, + mem_scheduler=default_config.mem_scheduler if default_config else None, + ) + else: + root_config = default_config.model_copy(deep=True) + root_config.user_id = "root" + root_config.session_id = "root_session" + + # Initialize parent MOSCore with root config + super().__init__(root_config) + + # Product-specific attributes + self.default_config = default_config + self.default_cube_config = default_cube_config + self.max_user_instances = max_user_instances + + # User-specific data structures + self.user_configs: dict[str, MOSConfig] = {} + self.user_cube_access: dict[str, set[str]] = {} # user_id -> set of cube_ids + self.user_chat_histories: dict[str, dict] = {} + + # Use PersistentUserManager for user management + self.global_user_manager = PersistentUserManager(user_id="root") + + # Initialize tiktoken for streaming + try: + # Use gpt2 encoding which is more stable and widely compatible + self.tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B") + logger.info("tokenizer initialized successfully for streaming") + except Exception as e: + logger.warning( + f"Failed to initialize tokenizer, will use character-based chunking: {e}" + ) + self.tokenizer = None + + # Restore user instances from persistent storage + self._restore_user_instances(default_cube_config=default_cube_config) + logger.info(f"User instances restored successfully, now user is {self.mem_cubes.keys()}") + + def _restore_user_instances( + self, default_cube_config: GeneralMemCubeConfig | None = None + ) -> None: + """Restore user instances from persistent storage after service restart. + + Args: + default_cube_config (GeneralMemCubeConfig | None, optional): Default cube configuration. Defaults to None. + """ + try: + # Get all user configurations from persistent storage + user_configs = self.global_user_manager.list_user_configs() + + # Get the raw database records for sorting by updated_at + session = self.global_user_manager._get_session() + try: + from memos.mem_user.persistent_user_manager import UserConfig + + db_configs = session.query(UserConfig).all() + # Create a mapping of user_id to updated_at timestamp + updated_at_map = {config.user_id: config.updated_at for config in db_configs} + + # Sort by updated_at timestamp (most recent first) and limit by max_instances + sorted_configs = sorted( + user_configs.items(), key=lambda x: updated_at_map.get(x[0], ""), reverse=True + )[: self.max_user_instances] + finally: + session.close() + + for user_id, config in sorted_configs: + if user_id != "root": # Skip root user + try: + # Store user config and cube access + self.user_configs[user_id] = config + self._load_user_cube_access(user_id) + + # Pre-load all cubes for this user with default config + self._preload_user_cubes(user_id, default_cube_config) + + logger.info( + f"Restored user configuration and pre-loaded cubes for {user_id}" + ) + + except Exception as e: + logger.error(f"Failed to restore user configuration for {user_id}: {e}") + + except Exception as e: + logger.error(f"Error during user instance restoration: {e}") + + def _preload_user_cubes( + self, user_id: str, default_cube_config: GeneralMemCubeConfig | None = None + ) -> None: + """Pre-load all cubes for a user into memory. + + Args: + user_id (str): The user ID to pre-load cubes for. + default_cube_config (GeneralMemCubeConfig | None, optional): Default cube configuration. Defaults to None. + """ + try: + # Get user's accessible cubes from persistent storage + accessible_cubes = self.global_user_manager.get_user_cubes(user_id) + + for cube in accessible_cubes: + if cube.cube_id not in self.mem_cubes: + try: + if cube.cube_path and os.path.exists(cube.cube_path): + # Pre-load cube with all memory types and default config + self.register_mem_cube( + cube.cube_path, + cube.cube_id, + user_id, + memory_types=["act_mem"] + if self.config.enable_activation_memory + else [], + default_config=default_cube_config, + ) + logger.info(f"Pre-loaded cube {cube.cube_id} for user {user_id}") + else: + logger.warning( + f"Cube path {cube.cube_path} does not exist for cube {cube.cube_id}, skipping pre-load" + ) + except Exception as e: + logger.error( + f"Failed to pre-load cube {cube.cube_id} for user {user_id}: {e}" + ) + + except Exception as e: + logger.error(f"Error pre-loading cubes for user {user_id}: {e}") + + def _load_user_cubes( + self, user_id: str, default_cube_config: GeneralMemCubeConfig | None = None + ) -> None: + """Load all cubes for a user into memory. + + Args: + user_id (str): The user ID to load cubes for. + default_cube_config (GeneralMemCubeConfig | None, optional): Default cube configuration. Defaults to None. + """ + # Get user's accessible cubes from persistent storage + accessible_cubes = self.global_user_manager.get_user_cubes(user_id) + + for cube in accessible_cubes[:1]: + if cube.cube_id not in self.mem_cubes: + try: + if cube.cube_path and os.path.exists(cube.cube_path): + # Use MOSCore's register_mem_cube method directly with default config + # Only load act_mem since text_mem is stored in database + self.register_mem_cube( + cube.cube_path, + cube.cube_id, + user_id, + memory_types=["act_mem"], + default_config=default_cube_config, + ) + else: + logger.warning( + f"Cube path {cube.cube_path} does not exist for cube {cube.cube_id}" + ) + except Exception as e: + logger.error(f"Failed to load cube {cube.cube_id} for user {user_id}: {e}") + + def _ensure_user_instance(self, user_id: str, max_instances: int | None = None) -> None: + """ + Ensure user configuration exists, creating it if necessary. + + Args: + user_id (str): The user ID + max_instances (int): Maximum instances to keep in memory (overrides class default) + """ + if user_id in self.user_configs: + return + + # Try to get config from persistent storage first + stored_config = self.global_user_manager.get_user_config(user_id) + if stored_config: + self.user_configs[user_id] = stored_config + self._load_user_cube_access(user_id) + else: + # Use default config + if not self.default_config: + raise ValueError(f"No configuration available for user {user_id}") + user_config = self.default_config.model_copy(deep=True) + user_config.user_id = user_id + user_config.session_id = f"{user_id}_session" + self.user_configs[user_id] = user_config + self._load_user_cube_access(user_id) + + # Apply LRU eviction if needed + max_instances = max_instances or self.max_user_instances + if len(self.user_configs) > max_instances: + # Remove least recently used instance (excluding root) + user_ids = [uid for uid in self.user_configs if uid != "root"] + if user_ids: + oldest_user_id = user_ids[0] + del self.user_configs[oldest_user_id] + if oldest_user_id in self.user_cube_access: + del self.user_cube_access[oldest_user_id] + logger.info(f"Removed least recently used user configuration: {oldest_user_id}") + + def _load_user_cube_access(self, user_id: str) -> None: + """Load user's cube access permissions.""" + try: + # Get user's accessible cubes from persistent storage + accessible_cubes = self.global_user_manager.get_user_cube_access(user_id) + self.user_cube_access[user_id] = set(accessible_cubes) + except Exception as e: + logger.warning(f"Failed to load cube access for user {user_id}: {e}") + self.user_cube_access[user_id] = set() + + def _get_user_config(self, user_id: str) -> MOSConfig: + """Get user configuration.""" + if user_id not in self.user_configs: + self._ensure_user_instance(user_id) + return self.user_configs[user_id] + + def _validate_user_cube_access(self, user_id: str, cube_id: str) -> None: + """Validate user has access to the cube.""" + if user_id not in self.user_cube_access: + self._load_user_cube_access(user_id) + + if cube_id not in self.user_cube_access.get(user_id, set()): + raise ValueError(f"User '{user_id}' does not have access to cube '{cube_id}'") + + def _validate_user_access(self, user_id: str, cube_id: str | None = None) -> None: + """Validate user access using MOSCore's built-in validation.""" + # Use MOSCore's built-in user validation + if cube_id: + self._validate_cube_access(user_id, cube_id) + else: + self._validate_user_exists(user_id) + + def _create_user_config(self, user_id: str, config: MOSConfig) -> MOSConfig: + """Create a new user configuration.""" + # Create a copy of config with the specific user_id + user_config = config.model_copy(deep=True) + user_config.user_id = user_id + user_config.session_id = f"{user_id}_session" + + # Save configuration to persistent storage + self.global_user_manager.save_user_config(user_id, user_config) + + return user_config + + def _get_or_create_user_config( + self, user_id: str, config: MOSConfig | None = None + ) -> MOSConfig: + """Get existing user config or create a new one.""" + if user_id in self.user_configs: + return self.user_configs[user_id] + + # Try to get config from persistent storage first + stored_config = self.global_user_manager.get_user_config(user_id) + if stored_config: + return self._create_user_config(user_id, stored_config) + + # Use provided config or default config + user_config = config or self.default_config + if not user_config: + raise ValueError(f"No configuration provided for user {user_id}") + + return self._create_user_config(user_id, user_config) + + def _build_system_prompt(self, user_id: str, memories_all: list[TextualMemoryItem]) -> str: + """ + Build custom system prompt for the user with memory references. + + Args: + user_id (str): The user ID. + memories (list[TextualMemoryItem]): The memories to build the system prompt. + + Returns: + str: The custom system prompt. + """ + + # Build base prompt + base_prompt = ( + "You are a knowledgeable and helpful AI assistant with access to user memories. " + "When responding to user queries, you should reference relevant memories using the provided memory IDs. " + "Use the reference format: [1-n:memoriesID] " + "where refid is a sequential number starting from 1 and increments for each reference in your response, " + "and memoriesID is the specific memory ID provided in the available memories list. " + "For example: [1:abc123], [2:def456], [3:ghi789], [4:jkl101], [5:mno112] " + "Only reference memories that are directly relevant to the user's question. " + "Make your responses natural and conversational while incorporating memory references when appropriate." + ) + + # Add memory context if available + if memories_all: + memory_context = "\n\n## Available ID Memories:\n" + for i, memory in enumerate(memories_all, 1): + # Format: [memory_id]: memory_content + memory_id = f"{memory.id.split('-')[0]}" if hasattr(memory, "id") else f"mem_{i}" + memory_content = memory.memory if hasattr(memory, "memory") else str(memory) + memory_context += f"{memory_id}: {memory_content}\n" + return base_prompt + memory_context + + return base_prompt + + def _process_streaming_references_complete(self, text_buffer: str) -> tuple[str, str]: + """ + Complete streaming reference processing to ensure reference tags are never split. + + Args: + text_buffer (str): The accumulated text buffer. + + Returns: + tuple[str, str]: (processed_text, remaining_buffer) + """ + import re + + # Pattern to match complete reference tags: [refid:memoriesID] + complete_pattern = r"\[\d+:[^\]]+\]" + + # Find all complete reference tags + complete_matches = list(re.finditer(complete_pattern, text_buffer)) + + if complete_matches: + # Find the last complete tag + last_match = complete_matches[-1] + end_pos = last_match.end() + + # Return text up to the end of the last complete tag + processed_text = text_buffer[:end_pos] + remaining_buffer = text_buffer[end_pos:] + return processed_text, remaining_buffer + + # Check for incomplete reference tags + # Look for opening bracket with number and colon + opening_pattern = r"\[\d+:" + opening_matches = list(re.finditer(opening_pattern, text_buffer)) + + if opening_matches: + # Find the last opening tag + last_opening = opening_matches[-1] + opening_start = last_opening.start() + + # Check if we have a complete opening pattern + if last_opening.end() <= len(text_buffer): + # We have a complete opening pattern, keep everything in buffer + return "", text_buffer + else: + # Incomplete opening pattern, return text before it + return text_buffer[:opening_start], text_buffer[opening_start:] + + # Check for partial opening pattern (starts with [ but not complete) + if "[" in text_buffer: + ref_start = text_buffer.find("[") + return text_buffer[:ref_start], text_buffer[ref_start:] + + # No reference tags found, return all text + return text_buffer, "" + + def _extract_references_from_response(self, response: str) -> list[dict]: + """ + Extract reference information from the response. + + Args: + response (str): The complete response text. + + Returns: + list[dict]: List of reference information. + """ + import re + + references = [] + # Pattern to match [refid:memoriesID] + pattern = r"\[(\d+):([^\]]+)\]" + + matches = re.findall(pattern, response) + for ref_number, memory_id in matches: + references.append({"memory_id": memory_id, "reference_number": int(ref_number)}) + + return references + + def _chunk_response_with_tiktoken( + self, response: str, chunk_size: int = 5 + ) -> Generator[str, None, None]: + """ + Chunk response using tiktoken for proper token-based streaming. + + Args: + response (str): The response text to chunk. + chunk_size (int): Number of tokens per chunk. + + Yields: + str: Chunked text pieces. + """ + if self.tokenizer: + # Use tiktoken for proper token-based chunking + tokens = self.tokenizer.encode(response) + + for i in range(0, len(tokens), chunk_size): + token_chunk = tokens[i : i + chunk_size] + chunk_text = self.tokenizer.decode(token_chunk) + yield chunk_text + else: + # Fallback to character-based chunking + char_chunk_size = chunk_size * 4 # Approximate character to token ratio + for i in range(0, len(response), char_chunk_size): + yield response[i : i + char_chunk_size] + + def _send_message_to_scheduler( + self, + user_id: str, + mem_cube_id: str, + query: str, + label: str, + ): + """ + Send message to scheduler. + args: + user_id: str, + mem_cube_id: str, + query: str, + """ + + if self.enable_mem_scheduler and (self.mem_scheduler is not None): + message_item = ScheduleMessageItem( + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=self.mem_cubes[mem_cube_id], + label=label, + content=query, + timestamp=datetime.now(), + ) + self.mem_scheduler.submit_messages(messages=[message_item]) + + def register_mem_cube( + self, + mem_cube_name_or_path_or_object: str | GeneralMemCube, + mem_cube_id: str | None = None, + user_id: str | None = None, + memory_types: list[Literal["text_mem", "act_mem", "para_mem"]] | None = None, + default_config: GeneralMemCubeConfig | None = None, + ) -> None: + """ + Register a MemCube with the MOS. + + Args: + mem_cube_name_or_path_or_object (str | GeneralMemCube): The name, path, or GeneralMemCube object to register. + mem_cube_id (str, optional): The identifier for the MemCube. If not provided, a default ID is used. + user_id (str, optional): The user ID to register the cube for. + memory_types (list[str], optional): List of memory types to load. + If None, loads all available memory types. + Options: ["text_mem", "act_mem", "para_mem"] + default_config (GeneralMemCubeConfig, optional): Default configuration for the cube. + """ + # Handle different input types + if isinstance(mem_cube_name_or_path_or_object, GeneralMemCube): + # Direct GeneralMemCube object provided + mem_cube = mem_cube_name_or_path_or_object + if mem_cube_id is None: + mem_cube_id = f"cube_{id(mem_cube)}" # Generate a unique ID + else: + # String path provided + mem_cube_name_or_path = mem_cube_name_or_path_or_object + if mem_cube_id is None: + mem_cube_id = mem_cube_name_or_path + + if mem_cube_id in self.mem_cubes: + logger.info(f"MemCube with ID {mem_cube_id} already in MOS, skip install.") + return + + # Create MemCube from path + if os.path.exists(mem_cube_name_or_path): + mem_cube = GeneralMemCube.init_from_dir( + mem_cube_name_or_path, memory_types, default_config + ) + else: + logger.warning( + f"MemCube {mem_cube_name_or_path} does not exist, try to init from remote repo." + ) + mem_cube = GeneralMemCube.init_from_remote_repo( + mem_cube_name_or_path, memory_types=memory_types, default_config=default_config + ) + + # Register the MemCube + logger.info( + f"Registering MemCube {mem_cube_id} with cube config {mem_cube.config.model_dump(mode='json')}" + ) + self.mem_cubes[mem_cube_id] = mem_cube + + def user_register( + self, + user_id: str, + user_name: str | None = None, + config: MOSConfig | None = None, + interests: str | None = None, + default_mem_cube: GeneralMemCube | None = None, + default_cube_config: GeneralMemCubeConfig | None = None, + ) -> dict[str, str]: + """Register a new user with configuration and default cube. + + Args: + user_id (str): The user ID for registration. + user_name (str): The user name for registration. + config (MOSConfig | None, optional): User-specific configuration. Defaults to None. + interests (str | None, optional): User interests as string. Defaults to None. + default_mem_cube (GeneralMemCube | None, optional): Default memory cube. Defaults to None. + default_cube_config (GeneralMemCubeConfig | None, optional): Default cube configuration. Defaults to None. + + Returns: + dict[str, str]: Registration result with status and message. + """ + try: + # Use provided config or default config + user_config = config or self.default_config + if not user_config: + return { + "status": "error", + "message": "No configuration provided for user registration", + } + if not user_name: + user_name = user_id + + # Create user with configuration using persistent user manager + self.global_user_manager.create_user_with_config( + user_id, user_config, UserRole.USER, user_id + ) + + # Create user configuration + user_config = self._create_user_config(user_id, user_config) + + # Create a default cube for the user using MOSCore's methods + default_cube_name = f"{user_name}_{user_id}_default_cube" + mem_cube_name_or_path = f"{CUBE_PATH}/{default_cube_name}" + default_cube_id = self.create_cube_for_user( + cube_name=default_cube_name, owner_id=user_id, cube_path=mem_cube_name_or_path + ) - def get_suggestion_query(self, user_id: str) -> list[str]: + if default_mem_cube: + try: + default_mem_cube.dump(mem_cube_name_or_path) + except Exception as e: + print(e) + + # Register the default cube with MOS + self.register_mem_cube( + mem_cube_name_or_path_or_object=default_mem_cube, + mem_cube_id=default_cube_id, + user_id=user_id, + memory_types=["act_mem"] if self.config.enable_activation_memory else [], + default_config=default_cube_config, # use default cube config + ) + + # Add interests to the default cube if provided + if interests: + self.add(memory_content=interests, mem_cube_id=default_cube_id, user_id=user_id) + + return { + "status": "success", + "message": f"User {user_name} registered successfully with default cube {default_cube_id}", + "user_id": user_id, + "default_cube_id": default_cube_id, + } + + except Exception as e: + return {"status": "error", "message": f"Failed to register user: {e!s}"} + + def get_suggestion_query(self, user_id: str, language: str = "zh") -> list[str]: """Get suggestion query from LLM. Args: - user_id (str, optional): Custom user ID. + user_id (str): User ID. + language (str): Language for suggestions ("zh" or "en"). Returns: list[str]: The suggestion query list. """ + if language == "zh": + suggestion_prompt = """ + 你是一个有用的助手，可以帮助用户生成建议查询。 + 我将获取用户最近的一些记忆， + 你应该生成一些建议查询，这些查询应该是用户想要查询的内容， + 用户最近的记忆是： + {memories} + 请生成3个建议查询用中文， + 输出应该是json格式，键是"query"，值是一个建议查询列表。 + + 示例： + {{ + "query": ["查询1", "查询2", "查询3"] + }} + """ + else: # English + suggestion_prompt = """ + You are a helpful assistant that can help users to generate suggestion query. + I will get some user recently memories, + you should generate some suggestion query, the query should be user what to query, + user recently memories is: + {memories} + please generate 3 suggestion query in English, + output should be a json format, the key is "query", the value is a list of suggestion query. + + example: + {{ + "query": ["query1", "query2", "query3"] + }} + """ + text_mem_result = super().search("my recently memories", user_id=user_id, top_k=10)[ + "text_mem" + ] + if text_mem_result: + memories = "\n".join([m.memory for m in text_mem_result[0]["memories"]]) + else: + memories = "" + message_list = [{"role": "system", "content": suggestion_prompt.format(memories=memories)}] + response = self.chat_llm.generate(message_list) + response_json = json.loads(response) + + return response_json["query"] + def chat( self, query: str, @@ -38,43 +687,211 @@ def chat( """Chat with LLM SSE Type. Args: query (str): Query string. - user_id (str, optional): Custom user ID. + user_id (str): User ID. cube_id (str, optional): Custom cube ID for user. history (list[dict], optional): Chat history. Returns: Generator[str, None, None]: The response string generator. """ - memories_list = self.search(query)["act_mem"] - content_list = [] - for memory in memories_list: - content_list.append(memory.content) - yield f"data: {json.dumps({'type': 'metadata', 'content': content_list})}\n\n" - llm_response = super().chat(query, user_id) - for chunk in llm_response: - chunk_data: str = f"data: {json.dumps({'type': 'text', 'content': chunk})}\n\n" + # Use MOSCore's built-in validation + if cube_id: + self._validate_cube_access(user_id, cube_id) + else: + self._validate_user_exists(user_id) + + # Load user cubes if not already loaded + self._load_user_cubes(user_id, self.default_cube_config) + time_start = time.time() + memories_list = super().search(query, user_id)["text_mem"] + # Get response from parent MOSCore (returns string, not generator) + response = super().chat(query, user_id) + time_end = time.time() + + # Use tiktoken for proper token-based chunking + for chunk in self._chunk_response_with_tiktoken(response, chunk_size=5): + chunk_data = f"data: {json.dumps({'type': 'text', 'content': chunk})}\n\n" yield chunk_data - reference = [{"id": "1234"}] + + # Prepare reference data + reference = [] + for memories in memories_list: + memories_json = memories.model_dump() + memories_json["metadata"]["ref_id"] = f"[{memories.id.split('-')[0]}]" + memories_json["metadata"]["embedding"] = [] + memories_json["metadata"]["sources"] = [] + reference.append(memories_json) + yield f"data: {json.dumps({'type': 'reference', 'content': reference})}\n\n" + total_time = round(float(time_end - time_start), 1) + + yield f"data: {json.dumps({'type': 'time', 'content': {'total_time': total_time, 'speed_improvement': '23%'}})}\n\n" yield f"data: {json.dumps({'type': 'end'})}\n\n" - def get_all( + def chat_with_references( self, + query: str, user_id: str, - memory_type: Literal["text_mem", "act_mem", "param_mem"], cube_id: str | None = None, - ) -> list[ - dict[ - str, - str - | list[ - TextualMemoryMetadata - | TreeNodeTextualMemoryMetadata - | ActivationMemoryItem - | ParametricMemoryItem - ], + history: MessageList | None = None, + ) -> Generator[str, None, None]: + """ + Chat with LLM with memory references and streaming output. + + Args: + query (str): Query string. + user_id (str): User ID. + cube_id (str, optional): Custom cube ID for user. + history (MessageList, optional): Chat history. + + Returns: + Generator[str, None, None]: The response string generator with reference processing. + """ + + self._load_user_cubes(user_id, self.default_cube_config) + + time_start = time.time() + memories_list = [] + memories_result = super().search( + query, user_id, install_cube_ids=[cube_id] if cube_id else None, top_k=10 + )["text_mem"] + if memories_result: + memories_list = memories_result[0]["memories"] + + # Build custom system prompt with relevant memories + system_prompt = self._build_system_prompt(user_id, memories_list) + + # Get chat history + target_user_id = user_id if user_id is not None else self.user_id + if target_user_id not in self.chat_history_manager: + self._register_chat_history(target_user_id) + + chat_history = self.chat_history_manager[target_user_id] + current_messages = [ + {"role": "system", "content": system_prompt}, + *chat_history.chat_history, + {"role": "user", "content": query}, ] - ]: + + # Generate response with custom prompt + past_key_values = None + response_stream = None + if self.config.enable_activation_memory: + # Handle activation memory (copy MOSCore logic) + for mem_cube_id, mem_cube in self.mem_cubes.items(): + if mem_cube.act_mem and mem_cube_id == cube_id: + kv_cache = next(iter(mem_cube.act_mem.get_all()), None) + past_key_values = ( + kv_cache.memory if (kv_cache and hasattr(kv_cache, "memory")) else None + ) + if past_key_values is not None: + logger.info("past_key_values is not None will apply to chat") + else: + logger.info("past_key_values is None will not apply to chat") + break + if self.config.chat_model.backend == "huggingface": + response_stream = self.chat_llm.generate_stream( + current_messages, past_key_values=past_key_values + ) + elif self.config.chat_model.backend == "vllm": + response_stream = self.chat_llm.generate_stream(current_messages) + else: + if self.config.chat_model.backend in ["huggingface", "vllm"]: + response_stream = self.chat_llm.generate_stream(current_messages) + else: + response_stream = self.chat_llm.generate(current_messages) + + time_end = time.time() + + # Simulate streaming output with proper reference handling using tiktoken + + # Initialize buffer for streaming + buffer = "" + full_response = "" + + # Use tiktoken for proper token-based chunking + if self.config.chat_model.backend not in ["huggingface", "vllm"]: + # For non-huggingface backends, we need to collect the full response first + full_response_text = "" + for chunk in response_stream: + if chunk in ["", ""]: + continue + full_response_text += chunk + response_stream = self._chunk_response_with_tiktoken(full_response_text, chunk_size=5) + for chunk in response_stream: + if chunk in ["", ""]: + continue + buffer += chunk + full_response += chunk + + # Process buffer to ensure complete reference tags + processed_chunk, remaining_buffer = self._process_streaming_references_complete(buffer) + + if processed_chunk: + chunk_data = f"data: {json.dumps({'type': 'text', 'data': processed_chunk}, ensure_ascii=False)}\n\n" + yield chunk_data + buffer = remaining_buffer + + # Process any remaining buffer + if buffer: + processed_chunk, remaining_buffer = self._process_streaming_references_complete(buffer) + if processed_chunk: + chunk_data = f"data: {json.dumps({'type': 'text', 'data': processed_chunk}, ensure_ascii=False)}\n\n" + yield chunk_data + + # Prepare reference data + reference = [] + for memories in memories_list: + memories_json = memories.model_dump() + memories_json["metadata"]["ref_id"] = f"{memories.id.split('-')[0]}" + memories_json["metadata"]["embedding"] = [] + memories_json["metadata"]["sources"] = [] + memories_json["metadata"]["memory"] = memories.memory + reference.append({"metadata": memories_json["metadata"]}) + + yield f"data: {json.dumps({'type': 'reference', 'data': reference})}\n\n" + total_time = round(float(time_end - time_start), 1) + yield f"data: {json.dumps({'type': 'time', 'data': {'total_time': total_time, 'speed_improvement': '23%'}})}\n\n" + chat_history.chat_history.append({"role": "user", "content": query}) + chat_history.chat_history.append({"role": "assistant", "content": full_response}) + self._send_message_to_scheduler( + user_id=user_id, mem_cube_id=cube_id, query=query, label=QUERY_LABEL + ) + self._send_message_to_scheduler( + user_id=user_id, mem_cube_id=cube_id, query=full_response, label=ANSWER_LABEL + ) + self.chat_history_manager[user_id] = chat_history + + yield f"data: {json.dumps({'type': 'end'})}\n\n" + self.add( + user_id=user_id, + messages=[ + { + "role": "user", + "content": query, + "chat_time": str(datetime.now().strftime("%Y-%m-%d %H:%M:%S")), + }, + { + "role": "assistant", + "content": full_response, + "chat_time": str(datetime.now().strftime("%Y-%m-%d %H:%M:%S")), + }, + ], + mem_cube_id=cube_id, + ) + # Keep chat history under 30 messages by removing oldest conversation pair + if len(self.chat_history_manager[user_id].chat_history) > 10: + self.chat_history_manager[user_id].chat_history.pop(0) # Remove oldest user message + self.chat_history_manager[user_id].chat_history.pop( + 0 + ) # Remove oldest assistant response + + def get_all( + self, + user_id: str, + memory_type: Literal["text_mem", "act_mem", "param_mem", "para_mem"], + mem_cube_ids: list[str] | None = None, + ) -> list[dict[str, Any]]: """Get all memory items for a user. Args: @@ -83,7 +900,253 @@ def get_all( memory_type (Literal["text_mem", "act_mem", "param_mem"]): The type of memory to get. Returns: - list[TextualMemoryMetadata | TreeNodeTextualMemoryMetadata | ActivationMemoryItem | ParametricMemoryItem]: A list of memory items. + list[dict[str, Any]]: A list of memory items with cube_id and memories structure. """ - memory_list = super().get_all(user_id, cube_id)[memory_type] - return memory_list + + # Load user cubes if not already loaded + self._load_user_cubes(user_id, self.default_cube_config) + memory_list = super().get_all( + mem_cube_id=mem_cube_ids[0] if mem_cube_ids else None, user_id=user_id + )[memory_type] + reformat_memory_list = [] + if memory_type == "text_mem": + for memory in memory_list: + memories = remove_embedding_recursive(memory["memories"]) + custom_type_ratios = { + "WorkingMemory": 0.20, + "LongTermMemory": 0.40, + "UserMemory": 0.40, + } + tree_result, node_type_count = convert_graph_to_tree_forworkmem( + memories, target_node_count=150, type_ratios=custom_type_ratios + ) + memories_filtered = filter_nodes_by_tree_ids(tree_result, memories) + children = tree_result["children"] + children_sort = sort_children_by_memory_type(children) + tree_result["children"] = children_sort + memories_filtered["tree_structure"] = tree_result + reformat_memory_list.append( + { + "cube_id": memory["cube_id"], + "memories": [memories_filtered], + "memory_statistics": node_type_count, + } + ) + elif memory_type == "act_mem": + memories_list = [] + act_mem_params = self.mem_cubes[mem_cube_ids[0]].act_mem.get_all() + if act_mem_params: + memories_data = act_mem_params[0].model_dump() + records = memories_data.get("records", []) + for record in records["text_memories"]: + memories_list.append( + { + "id": memories_data["id"], + "text": record, + "create_time": records["timestamp"], + "size": random.randint(1, 20), + "modify_times": 1, + } + ) + reformat_memory_list.append( + { + "cube_id": "xxxxxxxxxxxxxxxx" if not mem_cube_ids else mem_cube_ids[0], + "memories": memories_list, + } + ) + elif memory_type == "para_mem": + act_mem_params = self.mem_cubes[mem_cube_ids[0]].act_mem.get_all() + logger.info(f"act_mem_params: {act_mem_params}") + reformat_memory_list.append( + { + "cube_id": "xxxxxxxxxxxxxxxx" if not mem_cube_ids else mem_cube_ids[0], + "memories": act_mem_params[0].model_dump(), + } + ) + return reformat_memory_list + + def _get_subgraph( + self, query: str, mem_cube_id: str, user_id: str | None = None, top_k: int = 5 + ) -> list[dict[str, Any]]: + result = {"para_mem": [], "act_mem": [], "text_mem": []} + if self.config.enable_textual_memory and self.mem_cubes[mem_cube_id].text_mem: + result["text_mem"].append( + { + "cube_id": mem_cube_id, + "memories": self.mem_cubes[mem_cube_id].text_mem.get_relevant_subgraph( + query, top_k=top_k + ), + } + ) + return result + + def get_subgraph( + self, + user_id: str, + query: str, + mem_cube_ids: list[str] | None = None, + ) -> list[dict[str, Any]]: + """Get all memory items for a user. + + Args: + user_id (str): The ID of the user. + cube_id (str | None, optional): The ID of the cube. Defaults to None. + mem_cube_ids (list[str], optional): The IDs of the cubes. Defaults to None. + + Returns: + list[dict[str, Any]]: A list of memory items with cube_id and memories structure. + """ + + # Load user cubes if not already loaded + self._load_user_cubes(user_id, self.default_cube_config) + memory_list = self._get_subgraph( + query=query, mem_cube_id=mem_cube_ids[0], user_id=user_id, top_k=20 + )["text_mem"] + reformat_memory_list = [] + for memory in memory_list: + memories = remove_embedding_recursive(memory["memories"]) + custom_type_ratios = {"WorkingMemory": 0.20, "LongTermMemory": 0.40, "UserMemory": 0.4} + tree_result, node_type_count = convert_graph_to_tree_forworkmem( + memories, target_node_count=150, type_ratios=custom_type_ratios + ) + memories_filtered = filter_nodes_by_tree_ids(tree_result, memories) + children = tree_result["children"] + children_sort = sort_children_by_memory_type(children) + tree_result["children"] = children_sort + memories_filtered["tree_structure"] = tree_result + reformat_memory_list.append( + { + "cube_id": memory["cube_id"], + "memories": [memories_filtered], + "memory_statistics": node_type_count, + } + ) + + return reformat_memory_list + + def search( + self, query: str, user_id: str, install_cube_ids: list[str] | None = None, top_k: int = 20 + ): + """Search memories for a specific user.""" + # Validate user access + self._validate_user_access(user_id) + + # Load user cubes if not already loaded + self._load_user_cubes(user_id, self.default_cube_config) + search_result = super().search(query, user_id, install_cube_ids, top_k) + text_memory_list = search_result["text_mem"] + reformat_memory_list = [] + for memory in text_memory_list: + memories_list = [] + for data in memory["memories"]: + memories = data.model_dump() + memories["ref_id"] = f"[{memories['id'].split('-')[0]}]" + memories["metadata"]["embedding"] = [] + memories["metadata"]["sources"] = [] + memories["metadata"]["ref_id"] = f"[{memories['id'].split('-')[0]}]" + memories["metadata"]["id"] = memories["id"] + memories["metadata"]["memory"] = memories["memory"] + memories_list.append(memories) + reformat_memory_list.append({"cube_id": memory["cube_id"], "memories": memories_list}) + search_result["text_mem"] = reformat_memory_list + + return search_result + + def add( + self, + user_id: str, + messages: MessageList | None = None, + memory_content: str | None = None, + doc_path: str | None = None, + mem_cube_id: str | None = None, + ): + """Add memory for a specific user.""" + # Use MOSCore's built-in user/cube validation + if mem_cube_id: + self._validate_cube_access(user_id, mem_cube_id) + else: + self._validate_user_exists(user_id) + + # Load user cubes if not already loaded + self._load_user_cubes(user_id, self.default_cube_config) + + result = super().add(messages, memory_content, doc_path, mem_cube_id, user_id) + + return result + + def list_users(self) -> list: + """List all registered users.""" + return self.global_user_manager.list_users() + + def get_user_info(self, user_id: str) -> dict: + """Get user information including accessible cubes.""" + # Use MOSCore's built-in user validation + # Validate user access + self._validate_user_access(user_id) + + result = super().get_user_info() + + return result + + def share_cube_with_user(self, cube_id: str, owner_user_id: str, target_user_id: str) -> bool: + """Share a cube with another user.""" + # Use MOSCore's built-in cube access validation + self._validate_cube_access(owner_user_id, cube_id) + + result = super().share_cube_with_user(cube_id, target_user_id) + + return result + + def clear_user_chat_history(self, user_id: str) -> None: + """Clear chat history for a specific user.""" + # Validate user access + self._validate_user_access(user_id) + + super().clear_messages(user_id) + + def update_user_config(self, user_id: str, config: MOSConfig) -> bool: + """Update user configuration. + + Args: + user_id (str): The user ID. + config (MOSConfig): The new configuration. + + Returns: + bool: True if successful, False otherwise. + """ + try: + # Save to persistent storage + success = self.global_user_manager.save_user_config(user_id, config) + if success: + # Update in-memory config + self.user_configs[user_id] = config + logger.info(f"Updated configuration for user {user_id}") + + return success + except Exception as e: + logger.error(f"Failed to update user config for {user_id}: {e}") + return False + + def get_user_config(self, user_id: str) -> MOSConfig | None: + """Get user configuration. + + Args: + user_id (str): The user ID. + + Returns: + MOSConfig | None: The user's configuration or None if not found. + """ + return self.global_user_manager.get_user_config(user_id) + + def get_active_user_count(self) -> int: + """Get the number of active user configurations in memory.""" + return len(self.user_configs) + + def get_user_instance_info(self) -> dict[str, Any]: + """Get information about user configurations in memory.""" + return { + "active_instances": len(self.user_configs), + "max_instances": self.max_user_instances, + "user_ids": list(self.user_configs.keys()), + "lru_order": list(self.user_configs.keys()), # OrderedDict maintains insertion order + } diff --git a/src/memos/mem_os/utils/default_config.py b/src/memos/mem_os/utils/default_config.py new file mode 100644 index 000000000..e64bf2364 --- /dev/null +++ b/src/memos/mem_os/utils/default_config.py @@ -0,0 +1,352 @@ +""" +Default configuration utilities for MemOS. +Provides simplified configuration generation for users. +""" + +from typing import Literal + +from memos.configs.mem_cube import GeneralMemCubeConfig +from memos.configs.mem_os import MOSConfig +from memos.mem_cube.general import GeneralMemCube + + +def get_default_config( + openai_api_key: str, + openai_api_base: str = "https://api.openai.com/v1", + text_mem_type: Literal["tree_text", "general_text"] = "general_text", + user_id: str = "default_user", + **kwargs, +) -> MOSConfig: + """ + Generate a default MOS configuration with minimal user input. + + Args: + openai_api_key (str): OpenAI API key + openai_api_base (str): OpenAI API base URL, defaults to "https://api.openai.com/v1" + text_mem_type (str): Type of text memory, either "tree_text" or "general_text" + user_id (str): User ID for the configuration + **kwargs: Additional configuration overrides + + Returns: + MOSConfig: Complete MOS configuration object + + Example: + ```python + config = get_default_config( + openai_api_key="sk-...", + openai_api_base="https://api.openai.com/v1", + text_mem_type="general_text" + ) + mos = MOS(config) + ``` + """ + + # Base OpenAI configuration + openai_config = { + "model_name_or_path": kwargs.get("model_name", "gpt-4o-mini"), + "temperature": kwargs.get("temperature", 0.8), + "max_tokens": kwargs.get("max_tokens", 1024), + "top_p": kwargs.get("top_p", 0.9), + "top_k": kwargs.get("top_k", 50), + "remove_think_prefix": True, + "api_key": openai_api_key, + "api_base": openai_api_base, + } + + # Universal API embedder configuration (using OpenAI) + embedder_config = { + "backend": "universal_api", + "config": { + "provider": "openai", + "api_key": openai_api_key, + "model_name_or_path": kwargs.get("embedder_model", "text-embedding-3-large"), + "base_url": openai_api_base, + }, + } + + # Base configuration + config_dict = { + "user_id": user_id, + "chat_model": { + "backend": "openai", + "config": openai_config, + }, + "mem_reader": { + "backend": "simple_struct", + "config": { + "llm": { + "backend": "openai", + "config": openai_config, + }, + "embedder": embedder_config, + "chunker": { + "backend": "sentence", + "config": { + "tokenizer_or_token_counter": "gpt2", + "chunk_size": kwargs.get("chunk_size", 512), + "chunk_overlap": kwargs.get("chunk_overlap", 128), + "min_sentences_per_chunk": 1, + }, + }, + }, + }, + "enable_textual_memory": True, + "enable_activation_memory": kwargs.get("enable_activation_memory", False), + "top_k": kwargs.get("top_k", 5), + "max_turns_window": kwargs.get("max_turns_window", 20), + "enable_mem_scheduler": kwargs.get("enable_mem_scheduler", False), + } + + # Note: text_mem configuration is handled in get_default_cube_config + # MOSConfig doesn't have text_mem field, it's only in MemCube config + + # Add scheduler configuration if enabled + if config_dict.get("enable_mem_scheduler", False): + config_dict["mem_scheduler"] = { + "backend": "general_scheduler", + "config": { + "top_k": kwargs.get("scheduler_top_k", 10), + "top_n": kwargs.get("scheduler_top_n", 5), + "act_mem_update_interval": kwargs.get("scheduler_act_mem_update_interval", 300), + "context_window_size": kwargs.get("scheduler_context_window_size", 5), + "thread_pool_max_workers": kwargs.get("scheduler_thread_pool_max_workers", 10), + "consume_interval_seconds": kwargs.get("scheduler_consume_interval_seconds", 3), + "enable_parallel_dispatch": kwargs.get("scheduler_enable_parallel_dispatch", True), + "enable_act_memory_update": True, + }, + } + + # Add activation memory if enabled + if config_dict.get("enable_activation_memory", False): + config_dict["act_mem"] = { + "backend": "kv_cache", + "config": { + "memory_filename": kwargs.get( + "activation_memory_filename", "activation_memory.pickle" + ), + "extractor_llm": { + "backend": "openai", + "config": openai_config, + }, + }, + } + + return MOSConfig(**config_dict) + + +def get_default_cube_config( + openai_api_key: str, + openai_api_base: str = "https://api.openai.com/v1", + text_mem_type: Literal["tree_text", "general_text"] = "general_text", + user_id: str = "default_user", + **kwargs, +) -> GeneralMemCubeConfig: + """ + Generate a default MemCube configuration with minimal user input. + + Args: + openai_api_key (str): OpenAI API key + openai_api_base (str): OpenAI API base URL, defaults to "https://api.openai.com/v1" + text_mem_type (str): Type of text memory, either "tree_text" or "general_text" + user_id (str): User ID for the configuration + **kwargs: Additional configuration overrides + + Returns: + GeneralMemCubeConfig: Complete MemCube configuration object + """ + + # Base OpenAI configuration + openai_config = { + "model_name_or_path": kwargs.get("model_name", "gpt-4o-mini"), + "temperature": kwargs.get("temperature", 0.8), + "max_tokens": kwargs.get("max_tokens", 1024), + "top_p": kwargs.get("top_p", 0.9), + "top_k": kwargs.get("top_k", 50), + "remove_think_prefix": True, + "api_key": openai_api_key, + "api_base": openai_api_base, + } + + # Universal API embedder configuration (using OpenAI) + embedder_config = { + "backend": "universal_api", + "config": { + "provider": "openai", + "api_key": openai_api_key, + "model_name_or_path": kwargs.get("embedder_model", "text-embedding-3-large"), + "base_url": openai_api_base, + }, + } + + # Configure text memory based on type + if text_mem_type == "tree_text": + # Tree text memory requires Neo4j configuration + db_name = f"memos{user_id.replace('-', '').replace('_', '')}" + if not kwargs.get("use_multi_db", False): + db_name = kwargs.get("neo4j_db_name", "defaultdb") + neo4j_config = { + "uri": kwargs.get("neo4j_uri", "bolt://localhost:7687"), + "user": kwargs.get("neo4j_user", "neo4j"), + "db_name": db_name, + "password": kwargs.get("neo4j_password", "12345678"), + "auto_create": True, + "use_multi_db": kwargs.get("use_multi_db", False), + "embedding_dimension": kwargs.get("embedding_dimension", 3072), + } + if not kwargs.get("use_multi_db", False): + neo4j_config["user_name"] = f"memos{user_id.replace('-', '').replace('_', '')}" + + text_mem_config = { + "backend": "tree_text", + "config": { + "extractor_llm": {"backend": "openai", "config": openai_config}, + "dispatcher_llm": {"backend": "openai", "config": openai_config}, + "graph_db": { + "backend": "neo4j", + "config": neo4j_config, + }, + "embedder": embedder_config, + "reorganize": kwargs.get("enable_reorganize", False), + }, + } + + elif text_mem_type == "general_text": + # General text memory with file storage + text_mem_config = { + "backend": "general_text", + "config": { + "cube_id": kwargs.get("cube_id", f"{user_id}_cube"), + "memory_filename": kwargs.get("memory_filename", "textual_memory.json"), + "extractor_llm": {"backend": "openai", "config": openai_config}, + "vector_db": { + "backend": "qdrant", + "config": { + "collection_name": kwargs.get("collection_name", f"{user_id}_collection"), + "vector_dimension": kwargs.get("vector_dimension", 3072), + "distance_metric": "cosine", + }, + }, + "embedder": embedder_config, + }, + } + + # Configure activation memory if enabled + act_mem_config = {} + if kwargs.get("enable_activation_memory", False): + act_mem_config = { + "backend": "kv_cache", + "config": { + "memory_filename": kwargs.get( + "activation_memory_filename", "activation_memory.pickle" + ), + "extractor_llm": { + "backend": "openai", + "config": openai_config, + }, + }, + } + + # Create MemCube configuration + cube_config_dict = { + "user_id": user_id, + "cube_id": kwargs.get("cube_id", f"{user_id}_default_cube"), + "text_mem": text_mem_config, + "act_mem": act_mem_config, + "para_mem": {}, # Empty parametric memory by default + } + + return GeneralMemCubeConfig.model_validate(cube_config_dict) + + +def get_default( + openai_api_key: str, + openai_api_base: str = "https://api.openai.com/v1", + text_mem_type: Literal["tree_text", "general_text"] = "general_text", + user_id: str = "default_user", + **kwargs, +) -> tuple[MOSConfig, GeneralMemCube]: + """ + Generate both MOS configuration and default MemCube with minimal user input. + + This is the main convenience function for getting started with MemOS. + + Args: + openai_api_key (str): OpenAI API key + openai_api_base (str): OpenAI API base URL, defaults to "https://api.openai.com/v1" + text_mem_type (str): Type of text memory, either "tree_text" or "general_text" + user_id (str): User ID for the configuration + **kwargs: Additional configuration overrides + + Returns: + Tuple[MOSConfig, GeneralMemCube]: Complete MOS configuration and MemCube instance + + Example: + ```python + mos_config, default_cube = get_default( + openai_api_key="sk-...", + text_mem_type="general_text" + ) + memory = MOS(mos_config) + memory.register_mem_cube(default_cube) + ``` + """ + + # Generate MOS configuration + mos_config = get_default_config( + openai_api_key=openai_api_key, + openai_api_base=openai_api_base, + text_mem_type=text_mem_type, + user_id=user_id, + **kwargs, + ) + + # Generate MemCube configuration + cube_config = get_default_cube_config( + openai_api_key=openai_api_key, + openai_api_base=openai_api_base, + text_mem_type=text_mem_type, + user_id=user_id, + **kwargs, + ) + + # Create MemCube instance + default_cube = GeneralMemCube(cube_config) + + return mos_config, default_cube + + +def get_simple_config( + openai_api_key: str, + openai_api_base: str = "https://api.openai.com/v1", + text_mem_type: Literal["tree_text", "general_text"] = "general_text", + user_id: str = "default_user", +) -> MOSConfig: + """ + Get a minimal configuration with only essential parameters. + + This is the simplest way to get started with MemOS. + + Args: + openai_api_key (str): OpenAI API key + openai_api_base (str): OpenAI API base URL + text_mem_type (str): Type of text memory + user_id (str): User ID + + Returns: + MOSConfig: Basic MOS configuration + + Example: + ```python + config = get_simple_config( + openai_api_key="sk-...", + text_mem_type="general_text" + ) + mos = MOS(config) + ``` + """ + return get_default_config( + openai_api_key=openai_api_key, + openai_api_base=openai_api_base, + text_mem_type=text_mem_type, + user_id=user_id, + ) diff --git a/src/memos/mem_os/utils/format_utils.py b/src/memos/mem_os/utils/format_utils.py new file mode 100644 index 000000000..9722779f0 --- /dev/null +++ b/src/memos/mem_os/utils/format_utils.py @@ -0,0 +1,1154 @@ +import math +import random + +from typing import Any + +from memos.log import get_logger +from memos.memories.activation.item import KVCacheItem + + +logger = get_logger(__name__) + + +def extract_node_name(memory: str) -> str: + """Extract the first two words from memory as node_name""" + if not memory: + return "" + + words = [word.strip() for word in memory.split() if word.strip()] + + if len(words) >= 2: + return " ".join(words[:2]) + elif len(words) == 1: + return words[0] + else: + return "" + + +def analyze_tree_structure_enhanced(nodes: list[dict], edges: list[dict]) -> dict: + """Enhanced tree structure analysis, focusing on branching degree and leaf distribution""" + # Build adjacency list + adj_list = {} + reverse_adj = {} + for edge in edges: + source, target = edge["source"], edge["target"] + adj_list.setdefault(source, []).append(target) + reverse_adj.setdefault(target, []).append(source) + + # Find all nodes and root nodes + all_nodes = {node["id"] for node in nodes} + target_nodes = {edge["target"] for edge in edges} + root_nodes = all_nodes - target_nodes + + subtree_analysis = {} + + def analyze_subtree_enhanced(root_id: str) -> dict: + """Enhanced subtree analysis, focusing on branching degree and structure quality""" + visited = set() + max_depth = 0 + leaf_count = 0 + total_nodes = 0 + branch_nodes = 0 # Number of branch nodes with multiple children + chain_length = 0 # Longest single chain length + width_per_level = {} # Width per level + + def dfs(node_id: str, depth: int, chain_len: int): + nonlocal max_depth, leaf_count, total_nodes, branch_nodes, chain_length + + if node_id in visited: + return + + visited.add(node_id) + total_nodes += 1 + max_depth = max(max_depth, depth) + chain_length = max(chain_length, chain_len) + + # Record number of nodes per level + width_per_level[depth] = width_per_level.get(depth, 0) + 1 + + children = adj_list.get(node_id, []) + + if not children: # Leaf node + leaf_count += 1 + elif len(children) > 1: # Branch node + branch_nodes += 1 + # Reset chain length because we encountered a branch + for child in children: + dfs(child, depth + 1, 0) + else: # Single child node (chain structure) + for child in children: + dfs(child, depth + 1, chain_len + 1) + + dfs(root_id, 0, 0) + + # Calculate structure quality metrics + avg_width = sum(width_per_level.values()) / len(width_per_level) if width_per_level else 0 + max_width = max(width_per_level.values()) if width_per_level else 0 + + # Calculate branch density: ratio of branch nodes to total nodes + branch_density = branch_nodes / total_nodes if total_nodes > 0 else 0 + + # Calculate depth-width ratio: ideal tree should have moderate depth and good breadth + depth_width_ratio = max_depth / max_width if max_width > 0 else max_depth + + quality_score = calculate_enhanced_quality( + max_depth, + leaf_count, + total_nodes, + branch_nodes, + chain_length, + branch_density, + depth_width_ratio, + max_width, + ) + + return { + "root_id": root_id, + "max_depth": max_depth, + "leaf_count": leaf_count, + "total_nodes": total_nodes, + "branch_nodes": branch_nodes, + "max_chain_length": chain_length, + "branch_density": branch_density, + "max_width": max_width, + "avg_width": avg_width, + "depth_width_ratio": depth_width_ratio, + "nodes_in_subtree": list(visited), + "quality_score": quality_score, + "width_per_level": width_per_level, + } + + for root_id in root_nodes: + subtree_analysis[root_id] = analyze_subtree_enhanced(root_id) + + return subtree_analysis + + +def calculate_enhanced_quality( + max_depth: int, + leaf_count: int, + total_nodes: int, + branch_nodes: int, + max_chain_length: int, + branch_density: float, + depth_width_ratio: float, + max_width: int, +) -> float: + """Enhanced quality calculation, prioritizing branching degree and leaf distribution""" + + if total_nodes <= 1: + return 0.1 + + # 1. Branch quality score (weight: 35%) + # Branch node count score + branch_count_score = min(branch_nodes * 3, 15) # 3 points per branch node, max 15 points + + # Branch density score: ideal density between 20%-60% + if 0.2 <= branch_density <= 0.6: + branch_density_score = 10 + elif branch_density > 0.6: + branch_density_score = max(5, 10 - (branch_density - 0.6) * 20) + else: + branch_density_score = branch_density * 25 # Linear growth for 0-20% + + branch_score = (branch_count_score + branch_density_score) * 0.35 + + # 2. Leaf quality score (weight: 25%) + # Leaf count score + leaf_count_score = min(leaf_count * 2, 20) + + # Leaf distribution score: ideal leaf ratio 30%-70% of total nodes + leaf_ratio = leaf_count / total_nodes + if 0.3 <= leaf_ratio <= 0.7: + leaf_ratio_score = 10 + elif leaf_ratio > 0.7: + leaf_ratio_score = max(3, 10 - (leaf_ratio - 0.7) * 20) + else: + leaf_ratio_score = leaf_ratio * 20 # Linear growth for 0-30% + + leaf_score = (leaf_count_score + leaf_ratio_score) * 0.25 + + # 3. Structure balance score (weight: 25%) + # Depth score: moderate depth is best (3-8 layers) + if 3 <= max_depth <= 8: + depth_score = 15 + elif max_depth < 3: + depth_score = max_depth * 3 # Lower score for 1-2 layers + else: + depth_score = max(5, 15 - (max_depth - 8) * 1.5) # Gradually reduce score beyond 8 layers + + # Width score: larger max width is better, but with upper limit + width_score = min(max_width * 1.5, 15) + + # Depth-width ratio penalty: too large ratio means tree is too "thin" + if depth_width_ratio > 3: + ratio_penalty = (depth_width_ratio - 3) * 2 + structure_score = max(0, (depth_score + width_score - ratio_penalty)) * 0.25 + else: + structure_score = (depth_score + width_score) * 0.25 + + # 4. Chain structure penalty (weight: 15%) + # Longest single chain length penalty: overly long chains severely affect display + if max_chain_length <= 2: + chain_penalty_score = 10 + elif max_chain_length <= 5: + chain_penalty_score = 8 - (max_chain_length - 2) + else: + chain_penalty_score = max(0, 3 - (max_chain_length - 5) * 0.5) + + chain_score = chain_penalty_score * 0.15 + + # 5. Comprehensive calculation + total_score = branch_score + leaf_score + structure_score + chain_score + + # Special case severe penalties + if max_chain_length > total_nodes * 0.8: # If more than 80% are single chains + total_score *= 0.3 + elif branch_density < 0.1 and total_nodes > 5: # Large tree with almost no branches + total_score *= 0.5 + + return total_score + + +def sample_nodes_with_type_balance( + nodes: list[dict], + edges: list[dict], + target_count: int = 150, + type_ratios: dict[str, float] | None = None, +) -> tuple[list[dict], list[dict]]: + """ + Balanced sampling based on type ratios and tree quality + + Args: + nodes: List of nodes + edges: List of edges + target_count: Target number of nodes + type_ratios: Expected ratio for each type, e.g. {'WorkingMemory': 0.15, 'EpisodicMemory': 0.30, ...} + """ + if len(nodes) <= target_count: + return nodes, edges + + # Default type ratio configuration + if type_ratios is None: + type_ratios = { + "WorkingMemory": 0.10, # 10% + "EpisodicMemory": 0.25, # 25% + "SemanticMemory": 0.25, # 25% + "ProceduralMemory": 0.20, # 20% + "EmotionalMemory": 0.15, # 15% + "MetaMemory": 0.05, # 5% + } + + print( + f"Starting type-balanced sampling, original nodes: {len(nodes)}, target nodes: {target_count}" + ) + print(f"Target type ratios: {type_ratios}") + + # Analyze current node type distribution + current_type_counts = {} + nodes_by_type = {} + + for node in nodes: + memory_type = node.get("metadata", {}).get("memory_type", "Unknown") + current_type_counts[memory_type] = current_type_counts.get(memory_type, 0) + 1 + if memory_type not in nodes_by_type: + nodes_by_type[memory_type] = [] + nodes_by_type[memory_type].append(node) + + print(f"Current type distribution: {current_type_counts}") + + # Calculate target node count for each type + type_targets = {} + remaining_target = target_count + + for memory_type, ratio in type_ratios.items(): + if memory_type in nodes_by_type: + target_for_type = int(target_count * ratio) + # Ensure not exceeding the actual node count for this type + target_for_type = min(target_for_type, len(nodes_by_type[memory_type])) + type_targets[memory_type] = target_for_type + remaining_target -= target_for_type + + # Handle types not in ratio configuration + other_types = set(nodes_by_type.keys()) - set(type_ratios.keys()) + if other_types and remaining_target > 0: + per_other_type = max(1, remaining_target // len(other_types)) + for memory_type in other_types: + allocation = min(per_other_type, len(nodes_by_type[memory_type])) + type_targets[memory_type] = allocation + remaining_target -= allocation + + # If there's still remaining, distribute proportionally to main types + if remaining_target > 0: + main_types = [t for t in type_ratios if t in nodes_by_type] + if main_types: + extra_per_type = remaining_target // len(main_types) + for memory_type in main_types: + additional = min( + extra_per_type, + len(nodes_by_type[memory_type]) - type_targets.get(memory_type, 0), + ) + type_targets[memory_type] = type_targets.get(memory_type, 0) + additional + + print(f"Target node count for each type: {type_targets}") + + # Perform subtree quality sampling for each type + selected_nodes = [] + + for memory_type, target_for_type in type_targets.items(): + if target_for_type <= 0 or memory_type not in nodes_by_type: + continue + + type_nodes = nodes_by_type[memory_type] + print(f"\n--- Processing {memory_type} type: {len(type_nodes)} -> {target_for_type} ---") + + if len(type_nodes) <= target_for_type: + selected_nodes.extend(type_nodes) + print(f" Select all: {len(type_nodes)} nodes") + else: + # Use enhanced subtree quality sampling + type_selected = sample_by_enhanced_subtree_quality(type_nodes, edges, target_for_type) + selected_nodes.extend(type_selected) + print(f" Sampled selection: {len(type_selected)} nodes") + + # Filter edges + selected_node_ids = {node["id"] for node in selected_nodes} + filtered_edges = [ + edge + for edge in edges + if edge["source"] in selected_node_ids and edge["target"] in selected_node_ids + ] + + print(f"\nFinal selected nodes: {len(selected_nodes)}") + print(f"Final edges: {len(filtered_edges)}") + + # Verify final type distribution + final_type_counts = {} + for node in selected_nodes: + memory_type = node.get("metadata", {}).get("memory_type", "Unknown") + final_type_counts[memory_type] = final_type_counts.get(memory_type, 0) + 1 + + print(f"Final type distribution: {final_type_counts}") + for memory_type, count in final_type_counts.items(): + percentage = count / len(selected_nodes) * 100 + target_percentage = type_ratios.get(memory_type, 0) * 100 + print( + f" {memory_type}: {count} nodes ({percentage:.1f}%, target: {target_percentage:.1f}%)" + ) + + return selected_nodes, filtered_edges + + +def sample_by_enhanced_subtree_quality( + nodes: list[dict], edges: list[dict], target_count: int +) -> list[dict]: + """Sample using enhanced subtree quality""" + if len(nodes) <= target_count: + return nodes + + # Analyze subtree structure + subtree_analysis = analyze_tree_structure_enhanced(nodes, edges) + + if not subtree_analysis: + # If no subtree structure, sample by node importance + return sample_nodes_by_importance(nodes, edges, target_count) + + # Sort subtrees by quality score + sorted_subtrees = sorted( + subtree_analysis.items(), key=lambda x: x[1]["quality_score"], reverse=True + ) + + print(" Subtree quality ranking:") + for i, (root_id, analysis) in enumerate(sorted_subtrees[:5]): + print( + f" #{i + 1} Root node {root_id}: Quality={analysis['quality_score']:.2f}, " + f"Depth={analysis['max_depth']}, Branches={analysis['branch_nodes']}, " + f"Leaves={analysis['leaf_count']}, Max Width={analysis['max_width']}" + ) + + # Greedy selection of high-quality subtrees + selected_nodes = [] + selected_node_ids = set() + + for root_id, analysis in sorted_subtrees: + subtree_nodes = analysis["nodes_in_subtree"] + new_nodes = [node_id for node_id in subtree_nodes if node_id not in selected_node_ids] + + if not new_nodes: + continue + + remaining_quota = target_count - len(selected_nodes) + + if len(new_nodes) <= remaining_quota: + # Entire subtree can be added + for node_id in new_nodes: + node = next((n for n in nodes if n["id"] == node_id), None) + if node: + selected_nodes.append(node) + selected_node_ids.add(node_id) + print(f" Select entire subtree {root_id}: +{len(new_nodes)} nodes") + else: + # Subtree too large, need partial selection + if analysis["quality_score"] > 5: # Only partial selection for high-quality subtrees + subtree_node_objects = [n for n in nodes if n["id"] in new_nodes] + partial_selection = select_best_nodes_from_subtree( + subtree_node_objects, edges, remaining_quota, root_id + ) + + selected_nodes.extend(partial_selection) + for node in partial_selection: + selected_node_ids.add(node["id"]) + print( + f" Partial selection of subtree {root_id}: +{len(partial_selection)} nodes" + ) + + if len(selected_nodes) >= target_count: + break + + # If target count not reached, supplement with remaining nodes + if len(selected_nodes) < target_count: + remaining_nodes = [n for n in nodes if n["id"] not in selected_node_ids] + remaining_count = target_count - len(selected_nodes) + additional = sample_nodes_by_importance(remaining_nodes, edges, remaining_count) + selected_nodes.extend(additional) + print(f" Supplementary selection: +{len(additional)} nodes") + + return selected_nodes + + +def select_best_nodes_from_subtree( + subtree_nodes: list[dict], edges: list[dict], max_count: int, root_id: str +) -> list[dict]: + """Select the most important nodes from subtree, prioritizing branch structure""" + if len(subtree_nodes) <= max_count: + return subtree_nodes + + # Build internal connection relationships of subtree + subtree_node_ids = {node["id"] for node in subtree_nodes} + subtree_edges = [ + edge + for edge in edges + if edge["source"] in subtree_node_ids and edge["target"] in subtree_node_ids + ] + + # Calculate importance score for each node + node_scores = [] + + for node in subtree_nodes: + node_id = node["id"] + + # Out-degree and in-degree + out_degree = sum(1 for edge in subtree_edges if edge["source"] == node_id) + in_degree = sum(1 for edge in subtree_edges if edge["target"] == node_id) + + # Content length score + content_score = min(len(node.get("memory", "")), 300) / 15 + + # Branch node bonus + branch_bonus = out_degree * 8 if out_degree > 1 else 0 + + # Root node bonus + root_bonus = 15 if node_id == root_id else 0 + + # Connection importance + connection_score = (out_degree + in_degree) * 3 + + # Leaf node moderate bonus (ensure certain number of leaf nodes) + leaf_bonus = 5 if out_degree == 0 and in_degree > 0 else 0 + + total_score = content_score + connection_score + branch_bonus + root_bonus + leaf_bonus + node_scores.append((node, total_score)) + + # Sort by score and select + node_scores.sort(key=lambda x: x[1], reverse=True) + selected = [node for node, _ in node_scores[:max_count]] + + return selected + + +def sample_nodes_by_importance( + nodes: list[dict], edges: list[dict], target_count: int +) -> list[dict]: + """Sample by node importance (for cases without tree structure)""" + if len(nodes) <= target_count: + return nodes + + node_scores = [] + + for node in nodes: + node_id = node["id"] + out_degree = sum(1 for edge in edges if edge["source"] == node_id) + in_degree = sum(1 for edge in edges if edge["target"] == node_id) + content_score = min(len(node.get("memory", "")), 200) / 10 + connection_score = (out_degree + in_degree) * 5 + random_score = random.random() * 10 + + total_score = content_score + connection_score + random_score + node_scores.append((node, total_score)) + + node_scores.sort(key=lambda x: x[1], reverse=True) + return [node for node, _ in node_scores[:target_count]] + + +# Modified main function to use new sampling strategy +def convert_graph_to_tree_forworkmem( + json_data: dict[str, Any], + target_node_count: int = 150, + type_ratios: dict[str, float] | None = None, +) -> dict[str, Any]: + """ + Enhanced graph-to-tree conversion function, prioritizing branching degree and type balance + """ + original_nodes = json_data.get("nodes", []) + original_edges = json_data.get("edges", []) + + print(f"Original node count: {len(original_nodes)}") + print(f"Target node count: {target_node_count}") + filter_original_edges = [] + for original_edge in original_edges: + if original_edge["type"] == "PARENT": + filter_original_edges.append(original_edge) + node_type_count = {} + for node in original_nodes: + node_type = node.get("metadata", {}).get("memory_type", "Unknown") + node_type_count[node_type] = node_type_count.get(node_type, 0) + 1 + original_edges = filter_original_edges + # Use enhanced type-balanced sampling + if len(original_nodes) > target_node_count: + nodes, edges = sample_nodes_with_type_balance( + original_nodes, original_edges, target_node_count, type_ratios + ) + else: + nodes, edges = original_nodes, original_edges + + # The rest of tree structure building remains unchanged... + # [Original tree building code here] + + # Create node mapping table + node_map = {} + for node in nodes: + memory = node.get("memory", "") + node_name = extract_node_name(memory) + memory_key = node.get("metadata", {}).get("key", node_name) + usage = node.get("metadata", {}).get("usage", []) + frequency = len(usage) + node_map[node["id"]] = { + "id": node["id"], + "value": memory, + "frequency": frequency, + "node_name": memory_key, + "memory_type": node.get("metadata", {}).get("memory_type", "Unknown"), + "children": [], + } + + # Build parent-child relationship mapping + children_map = {} + parent_map = {} + + for edge in edges: + source = edge["source"] + target = edge["target"] + if source not in children_map: + children_map[source] = [] + children_map[source].append(target) + parent_map[target] = source + + # Find root nodes + all_node_ids = set(node_map.keys()) + children_node_ids = set(parent_map.keys()) + root_node_ids = all_node_ids - children_node_ids + + # Separate WorkingMemory and other root nodes + working_memory_roots = [] + other_roots = [] + + for root_id in root_node_ids: + if node_map[root_id]["memory_type"] == "WorkingMemory": + working_memory_roots.append(root_id) + else: + other_roots.append(root_id) + + def build_tree(node_id: str) -> dict[str, Any]: + """Recursively build tree structure""" + if node_id not in node_map: + return None + + children_ids = children_map.get(node_id, []) + children = [] + for child_id in children_ids: + child_tree = build_tree(child_id) + if child_tree: + children.append(child_tree) + + node = { + "id": node_id, + "node_name": node_map[node_id]["node_name"], + "value": node_map[node_id]["value"], + "memory_type": node_map[node_id]["memory_type"], + "frequency": node_map[node_id]["frequency"], + } + + if children: + node["children"] = children + + return node + + # Build root tree list + root_trees = [] + for root_id in other_roots: + tree = build_tree(root_id) + if tree: + root_trees.append(tree) + + # Handle WorkingMemory + if working_memory_roots: + working_memory_children = [] + for wm_root_id in working_memory_roots: + tree = build_tree(wm_root_id) + if tree: + working_memory_children.append(tree) + + working_memory_node = { + "id": "WorkingMemory", + "node_name": "WorkingMemory", + "value": "WorkingMemory", + "memory_type": "WorkingMemory", + "children": working_memory_children, + "frequency": 0, + } + + root_trees.append(working_memory_node) + + # Create total root node + result = { + "id": "root", + "node_name": "root", + "value": "root", + "memory_type": "Root", + "children": root_trees, + "frequency": 0, + } + + return result, node_type_count + + +def print_tree_structure(node: dict[str, Any], level: int = 0, max_level: int = 5): + """Print the first few layers of tree structure for easy viewing""" + if level > max_level: + return + + indent = " " * level + node_id = node.get("id", "unknown") + node_name = node.get("node_name", "") + node_value = node.get("value", "") + memory_type = node.get("memory_type", "Unknown") + + # Determine display method based on whether there are children + children = node.get("children", []) + if children: + # Intermediate node, display name, type and child count + print(f"{indent}- {node_name} [{memory_type}] ({len(children)} children)") + print(f"{indent} ID: {node_id}") + display_value = node_value[:80] + "..." if len(node_value) > 80 else node_value + print(f"{indent} Value: {display_value}") + + if level < max_level: + for child in children: + print_tree_structure(child, level + 1, max_level) + elif level == max_level: + print(f"{indent} ... (expansion limited)") + else: + # Leaf node, display name, type and value + display_value = node_value[:80] + "..." if len(node_value) > 80 else node_value + print(f"{indent}- {node_name} [{memory_type}]: {display_value}") + print(f"{indent} ID: {node_id}") + + +def analyze_final_tree_quality(tree_data: dict[str, Any]) -> dict: + """Analyze final tree quality, including type diversity, branch structure, etc.""" + stats = { + "total_nodes": 0, + "by_type": {}, + "by_depth": {}, + "max_depth": 0, + "total_leaves": 0, + "total_branches": 0, # Number of branch nodes with multiple children + "subtrees": [], + "type_diversity": {}, + "structure_quality": {}, + "chain_analysis": {}, # Single chain structure analysis + } + + def analyze_subtree(node, depth=0, parent_path="", chain_length=0): + stats["total_nodes"] += 1 + stats["max_depth"] = max(stats["max_depth"], depth) + + # Count by type + memory_type = node.get("memory_type", "Unknown") + stats["by_type"][memory_type] = stats["by_type"].get(memory_type, 0) + 1 + + # Count by depth + stats["by_depth"][depth] = stats["by_depth"].get(depth, 0) + 1 + + children = node.get("children", []) + current_path = ( + f"{parent_path}/{node.get('node_name', 'unknown')}" + if parent_path + else node.get("node_name", "root") + ) + + # Analyze node type + if not children: # Leaf node + stats["total_leaves"] += 1 + # Record chain length + if "max_chain_length" not in stats["chain_analysis"]: + stats["chain_analysis"]["max_chain_length"] = 0 + stats["chain_analysis"]["max_chain_length"] = max( + stats["chain_analysis"]["max_chain_length"], chain_length + ) + elif len(children) == 1: # Single child node (chain) + # Continue calculating chain length + for child in children: + analyze_subtree(child, depth + 1, current_path, chain_length + 1) + return # Early return to avoid duplicate processing + else: # Branch node (multiple children) + stats["total_branches"] += 1 + # Reset chain length + chain_length = 0 + + # If it's the root node of a major subtree, analyze its characteristics + if depth <= 2 and children: # Major subtree + subtree_depth = 0 + subtree_leaves = 0 + subtree_nodes = 0 + subtree_branches = 0 + subtree_types = {} + subtree_max_width = 0 + width_per_level = {} + + def count_subtree(subnode, subdepth): + nonlocal \ + subtree_depth, \ + subtree_leaves, \ + subtree_nodes, \ + subtree_branches, \ + subtree_max_width + subtree_nodes += 1 + subtree_depth = max(subtree_depth, subdepth) + + # Count type distribution within subtree + sub_memory_type = subnode.get("memory_type", "Unknown") + subtree_types[sub_memory_type] = subtree_types.get(sub_memory_type, 0) + 1 + + # Count width per level + width_per_level[subdepth] = width_per_level.get(subdepth, 0) + 1 + subtree_max_width = max(subtree_max_width, width_per_level[subdepth]) + + subchildren = subnode.get("children", []) + if not subchildren: + subtree_leaves += 1 + elif len(subchildren) > 1: + subtree_branches += 1 + + for child in subchildren: + count_subtree(child, subdepth + 1) + + count_subtree(node, 0) + + # Calculate subtree quality metrics + branch_density = subtree_branches / subtree_nodes if subtree_nodes > 0 else 0 + leaf_ratio = subtree_leaves / subtree_nodes if subtree_nodes > 0 else 0 + depth_width_ratio = ( + subtree_depth / subtree_max_width if subtree_max_width > 0 else subtree_depth + ) + + stats["subtrees"].append( + { + "root": node.get("node_name", "unknown"), + "type": memory_type, + "depth": subtree_depth, + "leaves": subtree_leaves, + "nodes": subtree_nodes, + "branches": subtree_branches, + "branch_density": branch_density, + "leaf_ratio": leaf_ratio, + "max_width": subtree_max_width, + "depth_width_ratio": depth_width_ratio, + "path": current_path, + "type_distribution": subtree_types, + "quality_score": calculate_enhanced_quality( + subtree_depth, + subtree_leaves, + subtree_nodes, + subtree_branches, + 0, + branch_density, + depth_width_ratio, + subtree_max_width, + ), + } + ) + + # Recursively analyze child nodes + for child in children: + analyze_subtree(child, depth + 1, current_path, 0) # Reset chain length + + analyze_subtree(tree_data) + + # Calculate overall structure quality + if stats["total_nodes"] > 1: + branch_density = stats["total_branches"] / stats["total_nodes"] + leaf_ratio = stats["total_leaves"] / stats["total_nodes"] + + # Calculate average width per level + total_width = sum(stats["by_depth"].values()) + avg_width = total_width / len(stats["by_depth"]) if stats["by_depth"] else 0 + max_width = max(stats["by_depth"].values()) if stats["by_depth"] else 0 + + stats["structure_quality"] = { + "branch_density": branch_density, + "leaf_ratio": leaf_ratio, + "avg_width": avg_width, + "max_width": max_width, + "depth_width_ratio": stats["max_depth"] / max_width + if max_width > 0 + else stats["max_depth"], + "is_well_balanced": 0.2 <= branch_density <= 0.6 and 0.3 <= leaf_ratio <= 0.7, + } + + # Calculate type diversity metrics + total_types = len(stats["by_type"]) + if total_types > 1: + # Calculate uniformity of type distribution (Shannon diversity index) + shannon_diversity = 0 + for count in stats["by_type"].values(): + if count > 0: + p = count / stats["total_nodes"] + shannon_diversity -= p * math.log2(p) + + # Normalize diversity index (0-1 range) + max_diversity = math.log2(total_types) if total_types > 1 else 0 + normalized_diversity = shannon_diversity / max_diversity if max_diversity > 0 else 0 + + stats["type_diversity"] = { + "total_types": total_types, + "shannon_diversity": shannon_diversity, + "normalized_diversity": normalized_diversity, + "distribution_balance": min(stats["by_type"].values()) / max(stats["by_type"].values()) + if max(stats["by_type"].values()) > 0 + else 0, + } + + # Single chain analysis + total_single_child_nodes = sum( + 1 for subtree in stats["subtrees"] if subtree.get("branch_density", 0) < 0.1 + ) + stats["chain_analysis"].update( + { + "single_chain_subtrees": total_single_child_nodes, + "chain_subtree_ratio": total_single_child_nodes / len(stats["subtrees"]) + if stats["subtrees"] + else 0, + } + ) + + return stats + + +def print_tree_analysis(tree_data: dict[str, Any]): + """Print enhanced tree analysis results""" + stats = analyze_final_tree_quality(tree_data) + + print("\n" + "=" * 60) + print("🌳 Enhanced Tree Structure Quality Analysis Report") + print("=" * 60) + + # Basic statistics + print("\n📊 Basic Statistics:") + print(f" Total nodes: {stats['total_nodes']}") + print(f" Max depth: {stats['max_depth']}") + print( + f" Leaf nodes: {stats['total_leaves']} ({stats['total_leaves'] / stats['total_nodes'] * 100:.1f}%)" + ) + print( + f" Branch nodes: {stats['total_branches']} ({stats['total_branches'] / stats['total_nodes'] * 100:.1f}%)" + ) + + # Structure quality assessment + structure = stats.get("structure_quality", {}) + if structure: + print("\n🏗️ Structure Quality Assessment:") + print( + f" Branch density: {structure['branch_density']:.3f} ({'✅ Good' if 0.2 <= structure['branch_density'] <= 0.6 else '⚠️ Needs improvement'})" + ) + print( + f" Leaf ratio: {structure['leaf_ratio']:.3f} ({'✅ Good' if 0.3 <= structure['leaf_ratio'] <= 0.7 else '⚠️ Needs improvement'})" + ) + print(f" Max width: {structure['max_width']}") + print( + f" Depth-width ratio: {structure['depth_width_ratio']:.2f} ({'✅ Good' if structure['depth_width_ratio'] <= 3 else '⚠️ Too thin'})" + ) + print( + f" Overall balance: {'✅ Good' if structure['is_well_balanced'] else '⚠️ Needs improvement'}" + ) + + # Single chain analysis + chain_analysis = stats.get("chain_analysis", {}) + if chain_analysis: + print("\n🔗 Single Chain Structure Analysis:") + print(f" Longest chain: {chain_analysis.get('max_chain_length', 0)} layers") + print(f" Single chain subtrees: {chain_analysis.get('single_chain_subtrees', 0)}") + print( + f" Single chain subtree ratio: {chain_analysis.get('chain_subtree_ratio', 0) * 100:.1f}%" + ) + + if chain_analysis.get("max_chain_length", 0) > 5: + print(" ⚠️ Warning: Overly long single chain structure may affect display") + elif chain_analysis.get("chain_subtree_ratio", 0) > 0.3: + print( + " ⚠️ Warning: Too many single chain subtrees, suggest increasing branch structure" + ) + else: + print(" ✅ Single chain structure well controlled") + + # Type diversity + type_div = stats.get("type_diversity", {}) + if type_div: + print("\n🎨 Type Diversity Analysis:") + print(f" Total types: {type_div['total_types']}") + print(f" Diversity index: {type_div['shannon_diversity']:.3f}") + print(f" Normalized diversity: {type_div['normalized_diversity']:.3f}") + print(f" Distribution balance: {type_div['distribution_balance']:.3f}") + + # Type distribution + print("\n📋 Type Distribution Details:") + for mem_type, count in sorted(stats["by_type"].items(), key=lambda x: x[1], reverse=True): + percentage = count / stats["total_nodes"] * 100 + print(f" {mem_type}: {count} nodes ({percentage:.1f}%)") + + # Depth distribution + print("\n📏 Depth Distribution:") + for depth in sorted(stats["by_depth"].keys()): + count = stats["by_depth"][depth] + print(f" Depth {depth}: {count} nodes") + + # Major subtree analysis + if stats["subtrees"]: + print("\n🌲 Major Subtree Analysis (sorted by quality):") + sorted_subtrees = sorted( + stats["subtrees"], key=lambda x: x.get("quality_score", 0), reverse=True + ) + for i, subtree in enumerate(sorted_subtrees[:8]): # Show first 8 + quality = subtree.get("quality_score", 0) + print(f" #{i + 1} {subtree['root']} [{subtree['type']}]:") + print(f" Quality score: {quality:.2f}") + print( + f" Structure: Depth={subtree['depth']}, Branches={subtree['branches']}, Leaves={subtree['leaves']}" + ) + print( + f" Density: Branch density={subtree.get('branch_density', 0):.3f}, Leaf ratio={subtree.get('leaf_ratio', 0):.3f}" + ) + + if quality > 15: + print(" ✅ High quality subtree") + elif quality > 8: + print(" 🟡 Medium quality subtree") + else: + print(" 🔴 Low quality subtree") + + print("\n" + "=" * 60) + + +def remove_embedding_recursive(memory_info: dict) -> Any: + """remove the embedding from the memory info + Args: + memory_info: product memory info + + Returns: + Any: product memory info without embedding + """ + if isinstance(memory_info, dict): + new_dict = {} + for key, value in memory_info.items(): + if key != "embedding": + new_dict[key] = remove_embedding_recursive(value) + return new_dict + elif isinstance(memory_info, list): + return [remove_embedding_recursive(item) for item in memory_info] + else: + return memory_info + + +def remove_embedding_from_memory_items(memory_items: list[Any]) -> list[dict]: + """Batch remove embedding fields from multiple TextualMemoryItem objects""" + clean_memories = [] + + for item in memory_items: + memory_dict = item.model_dump() + + # Remove embedding from metadata + if "metadata" in memory_dict and "embedding" in memory_dict["metadata"]: + del memory_dict["metadata"]["embedding"] + + clean_memories.append(memory_dict) + + return clean_memories + + +def sort_children_by_memory_type(children: list[dict[str, Any]]) -> list[dict[str, Any]]: + """ + sort the children by the memory_type + Args: + children: the children of the node + Returns: + the sorted children + """ + if not children: + return children + + def get_sort_key(child): + memory_type = child.get("memory_type", "Unknown") + # Sort directly by memory_type string, same types will naturally cluster together + return memory_type + + # Sort by memory_type + sorted_children = sorted(children, key=get_sort_key) + + return sorted_children + + +def extract_all_ids_from_tree(tree_node): + """ + Recursively traverse tree structure to extract all node IDs + + Args: + tree_node: Tree node (dictionary format) + + Returns: + set: Set containing all node IDs + """ + ids = set() + + # Add current node ID (if exists) + if "id" in tree_node: + ids.add(tree_node["id"]) + + # Recursively process child nodes + if tree_node.get("children"): + for child in tree_node["children"]: + ids.update(extract_all_ids_from_tree(child)) + + return ids + + +def filter_nodes_by_tree_ids(tree_data, nodes_data): + """ + Filter nodes list based on IDs used in tree structure + + Args: + tree_data: Tree structure data (dictionary) + nodes_data: Data containing nodes list (dictionary) + + Returns: + dict: Filtered nodes data, maintaining original structure + """ + # Extract all IDs used in the tree + used_ids = extract_all_ids_from_tree(tree_data) + + # Filter nodes list, keeping only nodes with IDs used in the tree + filtered_nodes = [node for node in nodes_data["nodes"] if node["id"] in used_ids] + + # Return result maintaining original structure + return {"nodes": filtered_nodes} + + +def convert_activation_memory_to_serializable( + act_mem_items: list[KVCacheItem], +) -> list[dict[str, Any]]: + """ + Convert activation memory items to a serializable format. + + Args: + act_mem_items: List of KVCacheItem objects + + Returns: + List of dictionaries with serializable data + """ + serializable_items = [] + + for item in act_mem_items: + # Extract basic information that can be serialized + serializable_item = { + "id": item.id, + "metadata": item.metadata, + "memory_info": { + "type": "DynamicCache", + "key_cache_layers": len(item.memory.key_cache) if item.memory else 0, + "value_cache_layers": len(item.memory.value_cache) if item.memory else 0, + "device": str(item.memory.key_cache[0].device) + if item.memory and item.memory.key_cache + else "unknown", + "dtype": str(item.memory.key_cache[0].dtype) + if item.memory and item.memory.key_cache + else "unknown", + }, + } + + # Add tensor shape information if available + if item.memory and item.memory.key_cache: + key_shapes = [] + value_shapes = [] + + for i, key_tensor in enumerate(item.memory.key_cache): + if key_tensor is not None: + key_shapes.append({"layer": i, "shape": list(key_tensor.shape)}) + + if i < len(item.memory.value_cache) and item.memory.value_cache[i] is not None: + value_shapes.append( + {"layer": i, "shape": list(item.memory.value_cache[i].shape)} + ) + + serializable_item["memory_info"]["key_shapes"] = key_shapes + serializable_item["memory_info"]["value_shapes"] = value_shapes + + serializable_items.append(serializable_item) + + return serializable_items + + +def convert_activation_memory_summary(act_mem_items: list[KVCacheItem]) -> dict[str, Any]: + """ + Create a summary of activation memory for API responses. + + Args: + act_mem_items: List of KVCacheItem objects + + Returns: + Dictionary with summary information + """ + if not act_mem_items: + return {"total_items": 0, "summary": "No activation memory items found"} + + total_items = len(act_mem_items) + total_layers = 0 + total_parameters = 0 + + for item in act_mem_items: + if item.memory and item.memory.key_cache: + total_layers += len(item.memory.key_cache) + + # Calculate approximate parameter count + for key_tensor in item.memory.key_cache: + if key_tensor is not None: + total_parameters += key_tensor.numel() + + for value_tensor in item.memory.value_cache: + if value_tensor is not None: + total_parameters += value_tensor.numel() + + return { + "total_items": total_items, + "total_layers": total_layers, + "total_parameters": total_parameters, + "summary": f"Activation memory contains {total_items} items with {total_layers} layers and approximately {total_parameters:,} parameters", + } diff --git a/src/memos/mem_reader/simple_struct.py b/src/memos/mem_reader/simple_struct.py index 8908fca42..1e1786fc0 100644 --- a/src/memos/mem_reader/simple_struct.py +++ b/src/memos/mem_reader/simple_struct.py @@ -1,7 +1,7 @@ import concurrent.futures import copy import json -import re + from abc import ABC from typing import Any @@ -16,8 +16,8 @@ from memos.parsers.factory import ParserFactory from memos.templates.mem_reader_prompts import ( SIMPLE_STRUCT_DOC_READER_PROMPT, - SIMPLE_STRUCT_MEM_READER_PROMPT, SIMPLE_STRUCT_MEM_READER_EXAMPLE, + SIMPLE_STRUCT_MEM_READER_PROMPT, ) @@ -208,15 +208,15 @@ def _process_doc_data(self, scene_data_info, info): for i, chunk_res in enumerate(processed_chunks): if chunk_res: node_i = TextualMemoryItem( - memory=chunk_res["summary"], + memory=chunk_res["value"], metadata=TreeNodeTextualMemoryMetadata( user_id=info.get("user_id"), session_id=info.get("session_id"), memory_type="LongTermMemory", status="activated", tags=chunk_res["tags"], - key="", - embedding=self.embedder.embed([chunk_res["summary"]])[0], + key=chunk_res["key"], + embedding=self.embedder.embed([chunk_res["value"]])[0], usage=[], sources=[f"{scene_data_info['file']}_{i}"], background="", diff --git a/src/memos/mem_scheduler/base_scheduler.py b/src/memos/mem_scheduler/base_scheduler.py index 012cc1d91..4f8c8c805 100644 --- a/src/memos/mem_scheduler/base_scheduler.py +++ b/src/memos/mem_scheduler/base_scheduler.py @@ -2,61 +2,283 @@ import threading import time -from abc import abstractmethod -from queue import Queue +from datetime import datetime +from pathlib import Path -from memos.configs.mem_scheduler import BaseSchedulerConfig +from memos.configs.mem_scheduler import AuthConfig, BaseSchedulerConfig from memos.llms.base import BaseLLM from memos.log import get_logger +from memos.mem_cube.general import GeneralMemCube from memos.mem_scheduler.modules.dispatcher import SchedulerDispatcher +from memos.mem_scheduler.modules.misc import AutoDroppingQueue as Queue +from memos.mem_scheduler.modules.monitor import SchedulerMonitor +from memos.mem_scheduler.modules.rabbitmq_service import RabbitMQSchedulerModule from memos.mem_scheduler.modules.redis_service import RedisSchedulerModule +from memos.mem_scheduler.modules.retriever import SchedulerRetriever from memos.mem_scheduler.modules.schemas import ( + ACTIVATION_MEMORY_TYPE, + ADD_LABEL, + DEFAULT_ACT_MEM_DUMP_PATH, DEFAULT_CONSUME_INTERVAL_SECONDS, DEFAULT_THREAD__POOL_MAX_WORKERS, + LONG_TERM_MEMORY_TYPE, + NOT_INITIALIZED, + PARAMETER_MEMORY_TYPE, + QUERY_LABEL, + TEXT_MEMORY_TYPE, + USER_INPUT_TYPE, + WORKING_MEMORY_TYPE, ScheduleLogForWebItem, ScheduleMessageItem, + TreeTextMemory_SEARCH_METHOD, ) +from memos.mem_scheduler.utils import transform_name_to_key +from memos.memories.activation.kv import KVCacheMemory +from memos.memories.activation.vllmkv import VLLMKVCacheItem, VLLMKVCacheMemory +from memos.memories.textual.tree import TextualMemoryItem, TreeTextMemory +from memos.templates.mem_scheduler_prompts import MEMORY_ASSEMBLY_TEMPLATE logger = get_logger(__name__) -class BaseScheduler(RedisSchedulerModule): +class BaseScheduler(RabbitMQSchedulerModule, RedisSchedulerModule): """Base class for all mem_scheduler.""" def __init__(self, config: BaseSchedulerConfig): """Initialize the scheduler with the given configuration.""" super().__init__() self.config = config + + # hyper-parameters + self.top_k = self.config.get("top_k", 5) + self.context_window_size = self.config.get("context_window_size", 5) + self.enable_act_memory_update = self.config.get("enable_act_memory_update", False) + self.act_mem_dump_path = self.config.get("act_mem_dump_path", DEFAULT_ACT_MEM_DUMP_PATH) + self.search_method = TreeTextMemory_SEARCH_METHOD + + self.enable_parallel_dispatch = self.config.get("enable_parallel_dispatch", False) self.max_workers = self.config.get( "thread_pool_max_workers", DEFAULT_THREAD__POOL_MAX_WORKERS ) - self.retriever = None - self.monitor = None - self.enable_parallel_dispatch = self.config.get("enable_parallel_dispatch", False) + + self.retriever: SchedulerRetriever | None = None + self.monitor: SchedulerMonitor | None = None + self.dispatcher = SchedulerDispatcher( max_workers=self.max_workers, enable_parallel_dispatch=self.enable_parallel_dispatch ) - # message queue - self.memos_message_queue: Queue[ScheduleMessageItem] = Queue() - self._web_log_message_queue: Queue[ScheduleLogForWebItem] = Queue() + # internal message queue + self.max_internal_messae_queue_size = 100 + self.memos_message_queue: Queue[ScheduleMessageItem] = Queue( + maxsize=self.max_internal_messae_queue_size + ) + self._web_log_message_queue: Queue[ScheduleLogForWebItem] = Queue( + maxsize=self.max_internal_messae_queue_size + ) self._consumer_thread = None # Reference to our consumer thread self._running = False self._consume_interval = self.config.get( "consume_interval_seconds", DEFAULT_CONSUME_INTERVAL_SECONDS ) - # others + # other attributes + self._context_lock = threading.Lock() self._current_user_id: str | None = None + self.auth_config_path: str | Path | None = self.config.get("auth_config_path", None) + self.auth_config = None + self.rabbitmq_config = None + + def initialize_modules(self, chat_llm: BaseLLM, process_llm: BaseLLM | None = None): + if process_llm is None: + process_llm = chat_llm + + # initialize submodules + self.chat_llm = chat_llm + self.process_llm = process_llm + self.monitor = SchedulerMonitor(process_llm=self.process_llm, config=self.config) + self.retriever = SchedulerRetriever(process_llm=self.process_llm, config=self.config) + self.retriever.log_working_memory_replacement = self.log_working_memory_replacement + + # initialize with auth_cofig + if self.auth_config_path is not None and Path(self.auth_config_path).exists(): + self.auth_config = AuthConfig.from_local_yaml(config_path=self.auth_config_path) + elif AuthConfig.default_config_exists(): + self.auth_config = AuthConfig.from_local_yaml() + else: + self.auth_config = None + + if self.auth_config is not None: + self.rabbitmq_config = self.auth_config.rabbitmq + self.initialize_rabbitmq(config=self.rabbitmq_config) - @abstractmethod - def initialize_modules(self, chat_llm: BaseLLM) -> None: - """Initialize all necessary modules for the scheduler + logger.debug("GeneralScheduler has been initialized") + + @property + def mem_cube(self) -> GeneralMemCube: + """The memory cube associated with this MemChat.""" + return self._current_mem_cube + + @mem_cube.setter + def mem_cube(self, value: GeneralMemCube) -> None: + """The memory cube associated with this MemChat.""" + self._current_mem_cube = value + self.retriever.mem_cube = value + + def _set_current_context_from_message(self, msg: ScheduleMessageItem) -> None: + """Update current user/cube context from the incoming message (thread-safe).""" + with self._context_lock: + self._current_user_id = msg.user_id + self._current_mem_cube_id = msg.mem_cube_id + self._current_mem_cube = msg.mem_cube + + def _validate_messages(self, messages: list[ScheduleMessageItem], label: str): + """Validate if all messages match the expected label. + + Args: + messages: List of message items to validate. + label: Expected message label (e.g., QUERY_LABEL/ANSWER_LABEL). + + Returns: + bool: True if all messages passed validation, False if any failed. + """ + for message in messages: + if not self._validate_message(message, label): + return False + logger.error("Message batch contains invalid labels, aborting processing") + return True + + def _validate_message(self, message: ScheduleMessageItem, label: str): + """Validate if the message matches the expected label. Args: - chat_llm: The LLM instance to be used for chat interactions + message: Incoming message item to validate. + label: Expected message label (e.g., QUERY_LABEL/ANSWER_LABEL). + + Returns: + bool: True if validation passed, False otherwise. + """ + if message.label != label: + logger.error(f"Handler validation failed: expected={label}, actual={message.label}") + return False + return True + + def update_activation_memory( + self, + new_memories: list[str | TextualMemoryItem], + label: str, + user_id: str, + mem_cube_id: str, + mem_cube: GeneralMemCube, + ) -> None: """ + Update activation memory by extracting KVCacheItems from new_memory (list of str), + add them to a KVCacheMemory instance, and dump to disk. + """ + if len(new_memories) == 0: + logger.error("update_activation_memory: new_memory is empty.") + return + if isinstance(new_memories[0], TextualMemoryItem): + new_text_memories = [mem.memory for mem in new_memories] + elif isinstance(new_memories[0], str): + new_text_memories = new_memories + else: + logger.error("Not Implemented.") + + try: + if isinstance(mem_cube.act_mem, VLLMKVCacheMemory): + act_mem: VLLMKVCacheMemory = mem_cube.act_mem + elif isinstance(mem_cube.act_mem, KVCacheMemory): + act_mem: KVCacheMemory = mem_cube.act_mem + else: + logger.error("Not Implemented.") + return + + text_memory = MEMORY_ASSEMBLY_TEMPLATE.format( + memory_text="".join( + [ + f"{i + 1}. {sentence.strip()}\n" + for i, sentence in enumerate(new_text_memories) + if sentence.strip() # Skip empty strings + ] + ) + ) + + # huggingface or vllm kv cache + original_cache_items: list[VLLMKVCacheItem] = act_mem.get_all() + original_text_memories = [] + if len(original_cache_items) > 0: + pre_cache_item: VLLMKVCacheItem = original_cache_items[-1] + original_text_memories = pre_cache_item.records.text_memories + act_mem.delete_all() + + cache_item = act_mem.extract(text_memory) + cache_item.records.text_memories = new_text_memories + + act_mem.add([cache_item]) + act_mem.dump(self.act_mem_dump_path) + + self.log_activation_memory_update( + original_text_memories=original_text_memories, + new_text_memories=new_text_memories, + label=label, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + ) + + except Exception as e: + logger.warning(f"MOS-based activation memory update failed: {e}", exc_info=True) + + def update_activation_memory_periodically( + self, + interval_seconds: int, + label: str, + user_id: str, + mem_cube_id: str, + mem_cube: GeneralMemCube, + ): + new_activation_memories = [] + + if self.monitor.timed_trigger( + last_time=self.monitor._last_activation_mem_update_time, + interval_seconds=interval_seconds, + ): + logger.info(f"Updating activation memory for user {user_id} and mem_cube {mem_cube_id}") + + self.monitor.update_memory_monitors( + user_id=user_id, mem_cube_id=mem_cube_id, mem_cube=mem_cube + ) + + new_activation_memories = [ + m.memory_text + for m in self.monitor.activation_memory_monitors[user_id][mem_cube_id].memories + ] + + logger.info( + f"Collected {len(new_activation_memories)} new memory entries for processing" + ) + + self.update_activation_memory( + new_memories=new_activation_memories, + label=label, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + ) + + self.monitor._last_activation_mem_update_time = datetime.now() + + logger.debug( + f"Activation memory update completed at {self.monitor._last_activation_mem_update_time}" + ) + else: + logger.info( + f"Skipping update - {interval_seconds} second interval not yet reached. " + f"Last update time is {self.monitor._last_activation_mem_update_time} and now is" + f"{datetime.now()}" + ) def submit_messages(self, messages: ScheduleMessageItem | list[ScheduleMessageItem]): """Submit multiple messages to the message queue.""" @@ -68,15 +290,185 @@ def submit_messages(self, messages: ScheduleMessageItem | list[ScheduleMessageIt logger.info(f"Submitted message: {message.label} - {message.content}") def _submit_web_logs(self, messages: ScheduleLogForWebItem | list[ScheduleLogForWebItem]): + """Submit log messages to the web log queue and optionally to RabbitMQ. + + Args: + messages: Single log message or list of log messages + """ if isinstance(messages, ScheduleLogForWebItem): messages = [messages] # transform single message to list for message in messages: self._web_log_message_queue.put(message) + logger.info(f"Submitted Scheduling log for web: {message.log_content}") + + if self.is_rabbitmq_connected(): + logger.info("Submitted Scheduling log to rabbitmq") + self.rabbitmq_publish_message(message=message.to_dict()) + logger.debug(f"{len(messages)} submitted. {self._web_log_message_queue.qsize()} in queue.") + + def log_activation_memory_update( + self, + original_text_memories: list[str], + new_text_memories: list[str], + label: str, + user_id: str, + mem_cube_id: str, + mem_cube: GeneralMemCube, + ): + """Log changes when activation memory is updated. + + Args: + original_text_memories: List of original memory texts + new_text_memories: List of new memory texts + """ + original_set = set(original_text_memories) + new_set = set(new_text_memories) + + # Identify changes + added_memories = list(new_set - original_set) # Present in new but not original + + # recording messages + for mem in added_memories: + log_message_a = self.create_autofilled_log_item( + log_content=mem, + label=label, + from_memory_type=TEXT_MEMORY_TYPE, + to_memory_type=ACTIVATION_MEMORY_TYPE, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + ) + log_message_b = self.create_autofilled_log_item( + log_content=mem, + label=label, + from_memory_type=ACTIVATION_MEMORY_TYPE, + to_memory_type=PARAMETER_MEMORY_TYPE, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + ) + self._submit_web_logs(messages=[log_message_a, log_message_b]) logger.info( - f"Submitted Scheduling log for web: {message.log_title} - {message.log_content}" + f"{len(added_memories)} {LONG_TERM_MEMORY_TYPE} memorie(s) " + f"transformed to {WORKING_MEMORY_TYPE} memories." ) - logger.debug(f"{len(messages)} submitted. {self._web_log_message_queue.qsize()} in queue.") + + def log_working_memory_replacement( + self, + original_memory: list[TextualMemoryItem], + new_memory: list[TextualMemoryItem], + user_id: str, + mem_cube_id: str, + mem_cube: GeneralMemCube, + ): + """Log changes when working memory is replaced.""" + memory_type_map = { + transform_name_to_key(name=m.memory): m.metadata.memory_type + for m in original_memory + new_memory + } + + original_text_memories = [m.memory for m in original_memory] + new_text_memories = [m.memory for m in new_memory] + + # Convert to sets for efficient difference operations + original_set = set(original_text_memories) + new_set = set(new_text_memories) + + # Identify changes + added_memories = list(new_set - original_set) # Present in new but not original + + # recording messages + for mem in added_memories: + normalized_mem = transform_name_to_key(name=mem) + if normalized_mem not in memory_type_map: + logger.error(f"Memory text not found in type mapping: {mem[:50]}...") + # Get the memory type from the map, default to LONG_TERM_MEMORY_TYPE if not found + mem_type = memory_type_map.get(normalized_mem, LONG_TERM_MEMORY_TYPE) + + if mem_type == WORKING_MEMORY_TYPE: + logger.warning(f"Memory already in working memory: {mem[:50]}...") + continue + + log_message = self.create_autofilled_log_item( + log_content=mem, + label=QUERY_LABEL, + from_memory_type=mem_type, + to_memory_type=WORKING_MEMORY_TYPE, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + ) + self._submit_web_logs(messages=log_message) + logger.info( + f"{len(added_memories)} {LONG_TERM_MEMORY_TYPE} memorie(s) " + f"transformed to {WORKING_MEMORY_TYPE} memories." + ) + + def log_adding_user_inputs( + self, + user_inputs: list[str], + user_id: str, + mem_cube_id: str, + mem_cube: GeneralMemCube, + ): + """Log changes when working memory is replaced.""" + + # recording messages + for input_str in user_inputs: + log_message = self.create_autofilled_log_item( + log_content=input_str, + label=ADD_LABEL, + from_memory_type=USER_INPUT_TYPE, + to_memory_type=TEXT_MEMORY_TYPE, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + ) + self._submit_web_logs(messages=log_message) + logger.info( + f"{len(user_inputs)} {USER_INPUT_TYPE} memorie(s) " + f"transformed to {TEXT_MEMORY_TYPE} memories." + ) + + def create_autofilled_log_item( + self, + log_content: str, + label: str, + from_memory_type: str, + to_memory_type: str, + user_id: str, + mem_cube_id: str, + mem_cube: GeneralMemCube, + ) -> ScheduleLogForWebItem: + text_mem_base: TreeTextMemory = mem_cube.text_mem + current_memory_sizes = text_mem_base.get_current_memory_size() + current_memory_sizes = { + "long_term_memory_size": current_memory_sizes["LongTermMemory"], + "user_memory_size": current_memory_sizes["UserMemory"], + "working_memory_size": current_memory_sizes["WorkingMemory"], + "transformed_act_memory_size": NOT_INITIALIZED, + "parameter_memory_size": NOT_INITIALIZED, + } + memory_capacities = { + "long_term_memory_capacity": text_mem_base.memory_manager.memory_size["LongTermMemory"], + "user_memory_capacity": text_mem_base.memory_manager.memory_size["UserMemory"], + "working_memory_capacity": text_mem_base.memory_manager.memory_size["WorkingMemory"], + "transformed_act_memory_capacity": NOT_INITIALIZED, + "parameter_memory_capacity": NOT_INITIALIZED, + } + + log_message = ScheduleLogForWebItem( + user_id=user_id, + mem_cube_id=mem_cube_id, + label=label, + from_memory_type=from_memory_type, + to_memory_type=to_memory_type, + log_content=log_content, + current_memory_sizes=current_memory_sizes, + memory_capacities=memory_capacities, + ) + return log_message def get_web_log_messages(self) -> list[dict]: """ @@ -87,13 +479,12 @@ def get_web_log_messages(self) -> list[dict]: ready for JSON serialization. The list is ordered from oldest to newest. """ messages = [] - - # Process all items in the queue - while not self._web_log_message_queue.empty(): - item = self._web_log_message_queue.get() - # Convert the ScheduleLogForWebItem to a dictionary and ensure datetime is serialized - item_dict = item.to_dict() - messages.append(item_dict) + while True: + try: + item = self._web_log_message_queue.get_nowait() # 线程安全的 get + messages.append(item.to_dict()) + except queue.Empty: + break return messages def _message_consumer(self) -> None: @@ -133,32 +524,72 @@ def _message_consumer(self) -> None: def start(self) -> None: """ - Start the message consumer thread. + Start the message consumer thread and initialize dispatcher resources. - Initializes and starts a daemon thread that will periodically - check for and process messages from the queue. + Initializes and starts: + 1. Message consumer thread + 2. Dispatcher thread pool (if parallel dispatch enabled) """ - if self._consumer_thread is not None and self._consumer_thread.is_alive(): - logger.warning("Consumer thread is already running") + if self._running: + logger.warning("Memory Scheduler is already running") return + # Initialize dispatcher resources + if self.enable_parallel_dispatch: + logger.info(f"Initializing dispatcher thread pool with {self.max_workers} workers") + + # Start consumer thread self._running = True self._consumer_thread = threading.Thread( target=self._message_consumer, - daemon=True, # Allows program to exit even if thread is running + daemon=True, name="MessageConsumerThread", ) self._consumer_thread.start() logger.info("Message consumer thread started") def stop(self) -> None: - """Stop the consumer thread and clean up resources.""" - if self._consumer_thread is None or not self._running: - logger.warning("Consumer thread is not running") + """Stop all scheduler components gracefully. + + 1. Stops message consumer thread + 2. Shuts down dispatcher thread pool + 3. Cleans up resources + """ + if not self._running: + logger.warning("Memory Scheduler is not running") return + + # Signal consumer thread to stop self._running = False - if self._consumer_thread.is_alive(): - self._consumer_thread.join(timeout=5.0) # Wait up to 5 seconds + + # Wait for consumer thread + if self._consumer_thread and self._consumer_thread.is_alive(): + self._consumer_thread.join(timeout=5.0) if self._consumer_thread.is_alive(): logger.warning("Consumer thread did not stop gracefully") - logger.info("Message consumer thread stopped") + else: + logger.info("Consumer thread stopped") + + # Shutdown dispatcher + if hasattr(self, "dispatcher") and self.dispatcher: + logger.info("Shutting down dispatcher...") + self.dispatcher.shutdown() + + # Clean up queues + self._cleanup_queues() + logger.info("Memory Scheduler stopped completely") + + def _cleanup_queues(self) -> None: + """Ensure all queues are emptied and marked as closed.""" + try: + while not self.memos_message_queue.empty(): + self.memos_message_queue.get_nowait() + self.memos_message_queue.task_done() + except queue.Empty: + pass + + try: + while not self._web_log_message_queue.empty(): + self._web_log_message_queue.get_nowait() + except queue.Empty: + pass diff --git a/src/memos/mem_scheduler/general_scheduler.py b/src/memos/mem_scheduler/general_scheduler.py index 09ef62dc0..42cebc677 100644 --- a/src/memos/mem_scheduler/general_scheduler.py +++ b/src/memos/mem_scheduler/general_scheduler.py @@ -1,27 +1,16 @@ import json -from datetime import datetime, timedelta - from memos.configs.mem_scheduler import GeneralSchedulerConfig -from memos.llms.base import BaseLLM from memos.log import get_logger from memos.mem_cube.general import GeneralMemCube from memos.mem_scheduler.base_scheduler import BaseScheduler -from memos.mem_scheduler.modules.monitor import SchedulerMonitor -from memos.mem_scheduler.modules.retriever import SchedulerRetriever from memos.mem_scheduler.modules.schemas import ( + ADD_LABEL, ANSWER_LABEL, - DEFAULT_ACT_MEM_DUMP_PATH, - DEFAULT_ACTIVATION_MEM_SIZE, - NOT_INITIALIZED, QUERY_LABEL, - ScheduleLogForWebItem, ScheduleMessageItem, - TextMemory_SEARCH_METHOD, - TreeTextMemory_SEARCH_METHOD, ) from memos.memories.textual.tree import TextualMemoryItem, TreeTextMemory -from memos.templates.mem_scheduler_prompts import MEMORY_ASSEMBLY_TEMPLATE logger = get_logger(__name__) @@ -31,275 +20,167 @@ class GeneralScheduler(BaseScheduler): def __init__(self, config: GeneralSchedulerConfig): """Initialize the scheduler with the given configuration.""" super().__init__(config) - self.top_k = self.config.get("top_k", 10) - self.top_n = self.config.get("top_n", 5) - self.act_mem_update_interval = self.config.get("act_mem_update_interval", 300) - self.context_window_size = self.config.get("context_window_size", 5) - self.activation_mem_size = self.config.get( - "activation_mem_size", DEFAULT_ACTIVATION_MEM_SIZE - ) - self.act_mem_dump_path = self.config.get("act_mem_dump_path", DEFAULT_ACT_MEM_DUMP_PATH) - self.search_method = TextMemory_SEARCH_METHOD - self._last_activation_mem_update_time = 0.0 - self.query_list = [] # register handlers handlers = { - QUERY_LABEL: self._query_message_consume, - ANSWER_LABEL: self._answer_message_consume, + QUERY_LABEL: self._query_message_consumer, + ANSWER_LABEL: self._answer_message_consumer, + ADD_LABEL: self._add_message_consumer, } self.dispatcher.register_handlers(handlers) - def initialize_modules(self, chat_llm: BaseLLM): - self.chat_llm = chat_llm - self.monitor = SchedulerMonitor( - chat_llm=self.chat_llm, activation_mem_size=self.activation_mem_size - ) - self.retriever = SchedulerRetriever(chat_llm=self.chat_llm) - logger.debug("GeneralScheduler has been initialized") - - def _answer_message_consume(self, messages: list[ScheduleMessageItem]) -> None: + def _query_message_consumer(self, messages: list[ScheduleMessageItem]) -> None: """ - Process and handle answer trigger messages from the queue. + Process and handle query trigger messages from the queue. Args: - messages: List of answer messages to process + messages: List of query messages to process """ - # TODO: This handler is not ready yet - logger.debug(f"Messages {messages} assigned to {ANSWER_LABEL} handler.") - for msg in messages: - if msg.label is not ANSWER_LABEL: - logger.error(f"_answer_message_consume is not designed for {msg.label}") - continue - answer = msg.content - self._current_user_id = msg.user_id - self._current_mem_cube_id = msg.mem_cube_id - self._current_mem_cube = msg.mem_cube + logger.debug(f"Messages {messages} assigned to {QUERY_LABEL} handler.") - # Get current activation memory items - current_activation_mem = [ - item["memory"] - for item in self.monitor.activation_memory_freq_list - if item["memory"] is not None - ] + # Process the query in a session turn + grouped_messages = self.dispatcher.group_messages_by_user_and_cube(messages=messages) - # Update memory frequencies based on the answer - # TODO: not implemented - self.monitor.activation_memory_freq_list = self.monitor.update_freq( - answer=answer, activation_memory_freq_list=self.monitor.activation_memory_freq_list - ) + self._validate_messages(messages=messages, label=QUERY_LABEL) - # Check if it's time to update activation memory - now = datetime.now() - if (now - self._last_activation_mem_update_time) >= timedelta( - seconds=self.act_mem_update_interval - ): - # TODO: not implemented - self.update_activation_memory(current_activation_mem) - self._last_activation_mem_update_time = now + for user_id in grouped_messages: + for mem_cube_id in grouped_messages[user_id]: + messages = grouped_messages[user_id][mem_cube_id] + if len(messages) == 0: + return - # recording messages - log_message = self.create_autofilled_log_item( - log_title="memos answer triggers scheduling...", - label=ANSWER_LABEL, - log_content="activation_memory has been updated", - ) - self._submit_web_logs(messages=log_message) + # for status update + self._set_current_context_from_message(msg=messages[0]) - def _query_message_consume(self, messages: list[ScheduleMessageItem]) -> None: + self.process_session_turn( + queries=[msg.content for msg in messages], + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=messages[0].mem_cube, + top_k=self.top_k, + ) + + def _answer_message_consumer(self, messages: list[ScheduleMessageItem]) -> None: """ - Process and handle query trigger messages from the queue. + Process and handle answer trigger messages from the queue. Args: - messages: List of query messages to process + messages: List of answer messages to process """ - logger.debug(f"Messages {messages} assigned to {QUERY_LABEL} handler.") - for msg in messages: - if msg.label is not QUERY_LABEL: - logger.error(f"_query_message_consume is not designed for {msg.label}") - continue - # Process the query in a session turn - self._current_user_id = msg.user_id - self._current_mem_cube_id = msg.mem_cube_id - self._current_mem_cube = msg.mem_cube - self.process_session_turn(query=msg.content, top_k=self.top_k, top_n=self.top_n) + logger.debug(f"Messages {messages} assigned to {ANSWER_LABEL} handler.") + # Process the query in a session turn + grouped_messages = self.dispatcher.group_messages_by_user_and_cube(messages=messages) + + self._validate_messages(messages=messages, label=ANSWER_LABEL) + + for user_id in grouped_messages: + for mem_cube_id in grouped_messages[user_id]: + messages = grouped_messages[user_id][mem_cube_id] + if len(messages) == 0: + return + + # for status update + self._set_current_context_from_message(msg=messages[0]) + + # update acivation memories + if self.enable_act_memory_update: + self.update_activation_memory_periodically( + interval_seconds=self.monitor.act_mem_update_interval, + label=ANSWER_LABEL, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=messages[0].mem_cube, + ) + + def _add_message_consumer(self, messages: list[ScheduleMessageItem]) -> None: + logger.debug(f"Messages {messages} assigned to {ADD_LABEL} handler.") + # Process the query in a session turn + grouped_messages = self.dispatcher.group_messages_by_user_and_cube(messages=messages) + + self._validate_messages(messages=messages, label=ADD_LABEL) + + for user_id in grouped_messages: + for mem_cube_id in grouped_messages[user_id]: + messages = grouped_messages[user_id][mem_cube_id] + if len(messages) == 0: + return + + # for status update + self._set_current_context_from_message(msg=messages[0]) + + # submit logs + for msg in messages: + user_inputs = json.loads(msg.content) + self.log_adding_user_inputs( + user_inputs=user_inputs, + user_id=msg.user_id, + mem_cube_id=msg.mem_cube_id, + mem_cube=msg.mem_cube, + ) + + # update acivation memories + if self.enable_act_memory_update: + self.update_activation_memory_periodically( + interval_seconds=self.monitor.act_mem_update_interval, + label=ADD_LABEL, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=messages[0].mem_cube, + ) def process_session_turn( self, - query: str, + queries: str | list[str], + user_id: str, + mem_cube_id: str, + mem_cube: GeneralMemCube, top_k: int = 10, - top_n: int = 5, + query_history: list[str] | None = None, ) -> None: """ Process a dialog turn: - If q_list reaches window size, trigger retrieval; - Immediately switch to the new memory if retrieval is triggered. """ - q_list = [query] - self.query_list.append(query) - text_mem_base = self.mem_cube.text_mem - if isinstance(text_mem_base, TreeTextMemory): - working_memory: list[TextualMemoryItem] = text_mem_base.get_working_memory() + if isinstance(queries, str): + queries = [queries] + + if query_history is None: + query_history = queries else: - logger.error("Not implemented!") + query_history.extend(queries) + + text_mem_base = mem_cube.text_mem + if not isinstance(text_mem_base, TreeTextMemory): + logger.error("Not implemented!", exc_info=True) return + + working_memory: list[TextualMemoryItem] = text_mem_base.get_working_memory() text_working_memory: list[str] = [w_m.memory for w_m in working_memory] intent_result = self.monitor.detect_intent( - q_list=q_list, text_working_memory=text_working_memory + q_list=query_history, text_working_memory=text_working_memory ) + if intent_result["trigger_retrieval"]: - missing_evidence = intent_result["missing_evidence"] - num_evidence = len(missing_evidence) + missing_evidences = intent_result["missing_evidences"] + num_evidence = len(missing_evidences) k_per_evidence = max(1, top_k // max(1, num_evidence)) new_candidates = [] - for item in missing_evidence: - logger.debug(f"missing_evidence: {item}") - results = self.search(query=item, top_k=k_per_evidence, method=self.search_method) - logger.debug(f"search results for {missing_evidence}: {results}") + for item in missing_evidences: + logger.debug(f"missing_evidences: {item}") + results = self.retriever.search( + query=item, mem_cube=mem_cube, top_k=k_per_evidence, method=self.search_method + ) + logger.debug(f"search results for {missing_evidences}: {results}") new_candidates.extend(results) - # recording messages - log_message = self.create_autofilled_log_item( - log_title="user query triggers scheduling...", - label=QUERY_LABEL, - log_content=f"search new candidates for working memory: {len(new_candidates)}", - ) - self._submit_web_logs(messages=log_message) - new_order_working_memory = self.replace_working_memory( - original_memory=working_memory, new_memory=new_candidates, top_k=top_k, top_n=top_n - ) - self.update_activation_memory(new_order_working_memory) - - def create_autofilled_log_item( - self, log_title: str, log_content: str, label: str - ) -> ScheduleLogForWebItem: - # TODO: create the log iterm with real stats - text_mem_base: TreeTextMemory = self.mem_cube.text_mem - current_memory_sizes = { - "long_term_memory_size": NOT_INITIALIZED, - "user_memory_size": NOT_INITIALIZED, - "working_memory_size": NOT_INITIALIZED, - "transformed_act_memory_size": NOT_INITIALIZED, - "parameter_memory_size": NOT_INITIALIZED, - } - memory_capacities = { - "long_term_memory_capacity": text_mem_base.memory_manager.memory_size["LongTermMemory"], - "user_memory_capacity": text_mem_base.memory_manager.memory_size["UserMemory"], - "working_memory_capacity": text_mem_base.memory_manager.memory_size["WorkingMemory"], - "transformed_act_memory_capacity": NOT_INITIALIZED, - "parameter_memory_capacity": NOT_INITIALIZED, - } - - log_message = ScheduleLogForWebItem( - user_id=self._current_user_id, - mem_cube_id=self._current_mem_cube_id, - label=label, - log_title=log_title, - log_content=log_content, - current_memory_sizes=current_memory_sizes, - memory_capacities=memory_capacities, - ) - return log_message - - @property - def mem_cube(self) -> GeneralMemCube: - """The memory cube associated with this MemChat.""" - return self._current_mem_cube - - @mem_cube.setter - def mem_cube(self, value: GeneralMemCube) -> None: - """The memory cube associated with this MemChat.""" - self._current_mem_cube = value - self.retriever.mem_cube = value - - def replace_working_memory( - self, - original_memory: list[TextualMemoryItem], - new_memory: list[TextualMemoryItem], - top_k: int = 10, - top_n: int = 5, - ) -> None | list[TextualMemoryItem]: - new_order_memory = None - text_mem_base = self.mem_cube.text_mem - if isinstance(text_mem_base, TreeTextMemory): - text_mem_base: TreeTextMemory = text_mem_base - combined_text_memory = [new_m.memory for new_m in original_memory] + [ - new_m.memory for new_m in new_memory - ] - combined_memory = original_memory + new_memory - memory_map = {mem_obj.memory: mem_obj for mem_obj in combined_memory} - - unique_memory = list(dict.fromkeys(combined_text_memory)) - prompt = self.build_prompt( - "memory_reranking", query="", current_order=unique_memory, staging_buffer=[] - ) - response = self.chat_llm.generate([{"role": "user", "content": prompt}]) - response = json.loads(response) - new_order_text_memory = response.get("new_order", [])[: top_n + top_k] - - new_order_memory = [] - for text in new_order_text_memory: - if text in memory_map: - new_order_memory.append(memory_map[text]) - else: - logger.warning( - f"Memory text not found in memory map. text: {text}; memory_map: {memory_map}" - ) - - text_mem_base.replace_working_memory(new_order_memory[top_n:]) - new_order_memory = new_order_memory[:top_n] - logger.info( - f"The working memory has been replaced with {len(new_order_memory)} new memories." - ) - else: - logger.error("memory_base is not supported") - - return new_order_memory - - def search(self, query: str, top_k: int, method=TreeTextMemory_SEARCH_METHOD): - text_mem_base = self.mem_cube.text_mem - if isinstance(text_mem_base, TreeTextMemory) and method == TextMemory_SEARCH_METHOD: - results_long_term = text_mem_base.search( - query=query, top_k=top_k, memory_type="LongTermMemory" - ) - results_user = text_mem_base.search(query=query, top_k=top_k, memory_type="UserMemory") - results = results_long_term + results_user - else: - logger.error("Not implemented.") - results = None - return results - - def update_activation_memory(self, new_memory: list[str | TextualMemoryItem]) -> None: - """ - Update activation memory by extracting KVCacheItems from new_memory (list of str), - add them to a KVCacheMemory instance, and dump to disk. - """ - # TODO: The function of update activation memory is waiting to test - if len(new_memory) == 0: - logger.error("update_activation_memory: new_memory is empty.") - return - if isinstance(new_memory[0], TextualMemoryItem): - new_text_memory = [mem.memory for mem in new_memory] - elif isinstance(new_memory[0], str): - new_text_memory = new_memory - else: - logger.error("Not Implemented.") - - try: - act_mem = self.mem_cube.act_mem - - text_memory = MEMORY_ASSEMBLY_TEMPLATE.format( - memory_text="".join( - [ - f"{i + 1}. {sentence.strip()}\n" - for i, sentence in enumerate(new_text_memory) - if sentence.strip() # Skip empty strings - ] - ) + new_order_working_memory = self.retriever.replace_working_memory( + queries=queries, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + original_memory=working_memory, + new_memory=new_candidates, + top_k=top_k, ) - act_mem.delete_all() - cache_item = act_mem.extract(text_memory) - act_mem.add(cache_item) - act_mem.dump(self.act_mem_dump_path) - except Exception as e: - logger.warning(f"MOS-based activation memory update failed: {e}") + logger.debug(f"size of new_order_working_memory: {len(new_order_working_memory)}") diff --git a/src/memos/mem_scheduler/modules/base.py b/src/memos/mem_scheduler/modules/base.py index b81514da0..2bc397166 100644 --- a/src/memos/mem_scheduler/modules/base.py +++ b/src/memos/mem_scheduler/modules/base.py @@ -16,6 +16,7 @@ def __init__(self): self.base_dir = Path(BASE_DIR) self._chat_llm = None + self._process_llm = None self._current_mem_cube_id: str | None = None self._current_mem_cube: GeneralMemCube | None = None self.mem_cubes: dict[str, GeneralMemCube] = {} @@ -63,6 +64,14 @@ def chat_llm(self, value: BaseLLM) -> None: """The memory cube associated with this MemChat.""" self._chat_llm = value + @property + def process_llm(self) -> BaseLLM: + return self._process_llm + + @process_llm.setter + def process_llm(self, value: BaseLLM) -> None: + self._process_llm = value + @property def mem_cube(self) -> GeneralMemCube: """The memory cube associated with this MemChat.""" diff --git a/src/memos/mem_scheduler/modules/dispatcher.py b/src/memos/mem_scheduler/modules/dispatcher.py index c46f99939..945a74ac6 100644 --- a/src/memos/mem_scheduler/modules/dispatcher.py +++ b/src/memos/mem_scheduler/modules/dispatcher.py @@ -73,6 +73,38 @@ def register_handlers( def _default_message_handler(self, messages: list[ScheduleMessageItem]) -> None: logger.debug(f"Using _default_message_handler to deal with messages: {messages}") + def group_messages_by_user_and_cube( + self, messages: list[ScheduleMessageItem] + ) -> dict[str, dict[str, list[ScheduleMessageItem]]]: + """ + Groups messages into a nested dictionary structure first by user_id, then by mem_cube_id. + + Args: + messages: List of ScheduleMessageItem objects to be grouped + + Returns: + A nested dictionary with the structure: + { + "user_id_1": { + "mem_cube_id_1": [msg1, msg2, ...], + "mem_cube_id_2": [msg3, msg4, ...], + ... + }, + "user_id_2": { + ... + }, + ... + } + Where each msg is the original ScheduleMessageItem object + """ + grouped_dict = defaultdict(lambda: defaultdict(list)) + + for msg in messages: + grouped_dict[msg.user_id][msg.mem_cube_id].append(msg) + + # Convert defaultdict to regular dict for cleaner output + return {user_id: dict(cube_groups) for user_id, cube_groups in grouped_dict.items()} + def dispatch(self, msg_list: list[ScheduleMessageItem]): """ Dispatch a list of messages to their respective handlers. @@ -98,6 +130,40 @@ def dispatch(self, msg_list: list[ScheduleMessageItem]): # dispatch to different handler logger.debug(f"Dispatch {len(msgs)} messages to {label} handler.") if self.enable_parallel_dispatch and self.dispatcher_executor is not None: - self.dispatcher_executor.submit(handler, msgs) + # Capture variables in lambda to avoid loop variable issues + # TODO check this + future = self.dispatcher_executor.submit(handler, msgs) + logger.debug(f"Dispatched {len(msgs)} messages as future task") + return future else: - handler(msgs) # Direct serial execution + handler(msgs) + return None + + def join(self, timeout: float | None = None) -> bool: + """Wait for all dispatched tasks to complete. + + Args: + timeout: Maximum time to wait in seconds. None means wait forever. + + Returns: + bool: True if all tasks completed, False if timeout occurred. + """ + if not self.enable_parallel_dispatch or self.dispatcher_executor is None: + return True # 串行模式无需等待 + + self.dispatcher_executor.shutdown(wait=True, timeout=timeout) + return True + + def shutdown(self) -> None: + """Gracefully shutdown the dispatcher.""" + if self.dispatcher_executor is not None: + self.dispatcher_executor.shutdown(wait=True) + self._running = False + logger.info("Dispatcher has been shutdown") + + def __enter__(self): + self._running = True + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + self.shutdown() diff --git a/src/memos/mem_scheduler/modules/misc.py b/src/memos/mem_scheduler/modules/misc.py new file mode 100644 index 000000000..b05552954 --- /dev/null +++ b/src/memos/mem_scheduler/modules/misc.py @@ -0,0 +1,39 @@ +import threading + +from queue import Empty, Full, Queue +from typing import TypeVar + + +T = TypeVar("T") + + +class AutoDroppingQueue(Queue[T]): + """A thread-safe queue that automatically drops the oldest item when full.""" + + def __init__(self, maxsize: int = 0): + super().__init__(maxsize=maxsize) + self._lock = threading.Lock() # Additional lock to prevent race conditions + + def put(self, item: T, block: bool = True, timeout: float | None = None) -> None: + """Put an item into the queue. + + If the queue is full, the oldest item will be automatically removed to make space. + This operation is thread-safe. + + Args: + item: The item to be put into the queue + block: Ignored (kept for compatibility with Queue interface) + timeout: Ignored (kept for compatibility with Queue interface) + """ + with self._lock: # Ensure atomic operation + try: + # First try non-blocking put + super().put(item, block=False) + except Full: + # If queue is full, remove the oldest item + from contextlib import suppress + + with suppress(Empty): + self.get_nowait() # Remove oldest item + # Retry putting the new item + super().put(item, block=False) diff --git a/src/memos/mem_scheduler/modules/monitor.py b/src/memos/mem_scheduler/modules/monitor.py index 7ecd5ce42..366f0c202 100644 --- a/src/memos/mem_scheduler/modules/monitor.py +++ b/src/memos/mem_scheduler/modules/monitor.py @@ -1,45 +1,241 @@ -import json - +from datetime import datetime from typing import Any +from memos.configs.mem_scheduler import BaseSchedulerConfig +from memos.llms.base import BaseLLM from memos.log import get_logger from memos.mem_cube.general import GeneralMemCube from memos.mem_scheduler.modules.base import BaseSchedulerModule +from memos.mem_scheduler.modules.misc import AutoDroppingQueue as Queue +from memos.mem_scheduler.modules.schemas import ( + DEFAULT_ACTIVATION_MEM_MONITOR_SIZE_LIMIT, + DEFAULT_WORKING_MEM_MONITOR_SIZE_LIMIT, + MONITOR_ACTIVATION_MEMORY_TYPE, + MONITOR_WORKING_MEMORY_TYPE, + MemCubeID, + MemoryMonitorManager, + UserID, +) from memos.mem_scheduler.utils import extract_json_dict -from memos.memories.textual.tree import TreeTextMemory +from memos.memories.textual.tree import TextualMemoryItem, TreeTextMemory logger = get_logger(__name__) class SchedulerMonitor(BaseSchedulerModule): - def __init__(self, chat_llm, activation_mem_size=5): + """Monitors and manages scheduling operations with LLM integration.""" + + def __init__(self, process_llm: BaseLLM, config: BaseSchedulerConfig): super().__init__() - self.statistics = {} - self.intent_history: list[str] = [] - self.activation_mem_size = activation_mem_size - self.activation_memory_freq_list = [ - {"memory": None, "count": 0} for _ in range(self.activation_mem_size) - ] - - self._chat_llm = chat_llm - - def update_stats(self, mem_cube): - self.statistics["activation_mem_size"] = self.activation_mem_size - mem_cube_info = self.get_mem_cube_info(mem_cube) - self.statistics.update(mem_cube_info) - - def get_mem_cube_info(self, mem_cube: GeneralMemCube): - mem_cube_info = {} - - text_mem = mem_cube.text_mem - if isinstance(text_mem, TreeTextMemory): - memory_size_dict = text_mem.memory_manager.memory_size - mem_cube_info["text_mem"] = memory_size_dict + + # hyper-parameters + self.config: BaseSchedulerConfig = config + self.act_mem_update_interval = self.config.get("act_mem_update_interval", 300) + + # Partial Retention Strategy + self.partial_retention_number = 2 + self.working_mem_monitor_capacity = DEFAULT_WORKING_MEM_MONITOR_SIZE_LIMIT + self.activation_mem_monitor_capacity = DEFAULT_ACTIVATION_MEM_MONITOR_SIZE_LIMIT + + # attributes + self.query_history = Queue(maxsize=self.config.context_window_size) + self.intent_history = Queue(maxsize=self.config.context_window_size) + self.working_memory_monitors: dict[UserID, dict[MemCubeID, MemoryMonitorManager]] = {} + self.activation_memory_monitors: dict[UserID, dict[MemCubeID, MemoryMonitorManager]] = {} + + # Lifecycle monitor + self._last_activation_mem_update_time = datetime.min + + self._process_llm = process_llm + + def register_memory_manager_if_not_exists( + self, + user_id: str, + mem_cube_id: str, + memory_monitors: dict[UserID, dict[MemCubeID, MemoryMonitorManager]], + max_capacity: int, + ) -> None: + """ + Register a new MemoryMonitorManager for the given user and memory cube if it doesn't exist. + + Checks if a MemoryMonitorManager already exists for the specified user_id and mem_cube_id. + If not, creates a new MemoryMonitorManager with appropriate capacity settings and registers it. + + Args: + user_id: The ID of the user to associate with the memory manager + mem_cube_id: The ID of the memory cube to monitor + + Note: + This function will update the loose_max_working_memory_capacity based on the current + WorkingMemory size plus partial retention number before creating a new manager. + """ + # Check if a MemoryMonitorManager already exists for the current user_id and mem_cube_id + # If doesn't exist, create and register a new one + if (user_id not in memory_monitors) or (mem_cube_id not in memory_monitors[user_id]): + # Initialize MemoryMonitorManager with user ID, memory cube ID, and max capacity + monitor_manager = MemoryMonitorManager( + user_id=user_id, mem_cube_id=mem_cube_id, max_capacity=max_capacity + ) + + # Safely register the new manager in the nested dictionary structure + memory_monitors.setdefault(user_id, {})[mem_cube_id] = monitor_manager + logger.info( + f"Registered new MemoryMonitorManager for user_id={user_id}," + f" mem_cube_id={mem_cube_id} with max_capacity={max_capacity}" + ) else: + logger.info( + f"MemoryMonitorManager already exists for user_id={user_id}, " + f"mem_cube_id={mem_cube_id} in the provided memory_monitors dictionary" + ) + + def update_memory_monitors(self, user_id: str, mem_cube_id: str, mem_cube: GeneralMemCube): + text_mem_base: TreeTextMemory = mem_cube.text_mem + + if not isinstance(text_mem_base, TreeTextMemory): logger.error("Not Implemented") + return + + self.working_mem_monitor_capacity = min( + DEFAULT_WORKING_MEM_MONITOR_SIZE_LIMIT, + ( + text_mem_base.memory_manager.memory_size["WorkingMemory"] + + self.partial_retention_number + ), + ) + + self.update_working_memory_monitors( + user_id=user_id, mem_cube_id=mem_cube_id, mem_cube=mem_cube + ) - return mem_cube_info + self.update_activation_memory_monitors( + user_id=user_id, mem_cube_id=mem_cube_id, mem_cube=mem_cube + ) + + def update_working_memory_monitors( + self, user_id: str, mem_cube_id: str, mem_cube: GeneralMemCube + ): + # register monitors + self.register_memory_manager_if_not_exists( + user_id=user_id, + mem_cube_id=mem_cube_id, + memory_monitors=self.working_memory_monitors, + max_capacity=self.working_mem_monitor_capacity, + ) + + # === update working memory monitors === + # Retrieve current working memory content + text_mem_base: TreeTextMemory = mem_cube.text_mem + working_memory: list[TextualMemoryItem] = text_mem_base.get_working_memory() + text_working_memory: list[str] = [w_m.memory for w_m in working_memory] + + self.working_memory_monitors[user_id][mem_cube_id].update_memories( + text_working_memories=text_working_memory, + partial_retention_number=self.partial_retention_number, + ) + + def update_activation_memory_monitors( + self, user_id: str, mem_cube_id: str, mem_cube: GeneralMemCube + ): + self.register_memory_manager_if_not_exists( + user_id=user_id, + mem_cube_id=mem_cube_id, + memory_monitors=self.activation_memory_monitors, + max_capacity=self.activation_mem_monitor_capacity, + ) + + # === update activation memory monitors === + # Sort by importance_score in descending order and take top k + top_k_memories = sorted( + self.working_memory_monitors[user_id][mem_cube_id].memories, + key=lambda m: m.get_score(), + reverse=True, + )[: self.activation_mem_monitor_capacity] + + # Extract just the text from these memories + text_top_k_memories = [m.memory_text for m in top_k_memories] + + # Update the activation memory monitors with these important memories + self.activation_memory_monitors[user_id][mem_cube_id].update_memories( + text_working_memories=text_top_k_memories, + partial_retention_number=self.partial_retention_number, + ) + + def timed_trigger(self, last_time: datetime, interval_seconds: float) -> bool: + now = datetime.now() + elapsed = (now - last_time).total_seconds() + if elapsed >= interval_seconds: + return True + logger.debug(f"Time trigger not ready, {elapsed:.1f}s elapsed (needs {interval_seconds}s)") + return False + + def get_monitor_memories( + self, + user_id: str, + mem_cube_id: str, + memory_type: str = MONITOR_WORKING_MEMORY_TYPE, + top_k: int = 10, + ) -> list[str]: + """Retrieves memory items managed by the scheduler, sorted by recording count. + + Args: + user_id: Unique identifier of the user + mem_cube_id: Unique identifier of the memory cube + memory_type: Type of memory to retrieve (MONITOR_WORKING_MEMORY_TYPE or + MONITOR_ACTIVATION_MEMORY_TYPE) + top_k: Maximum number of memory items to return (default: 10) + + Returns: + List of memory texts, sorted by recording count in descending order. + Returns empty list if no MemoryMonitorManager exists for the given parameters. + """ + # Select the appropriate monitor dictionary based on memory_type + if memory_type == MONITOR_WORKING_MEMORY_TYPE: + monitor_dict = self.working_memory_monitors + elif memory_type == MONITOR_ACTIVATION_MEMORY_TYPE: + monitor_dict = self.activation_memory_monitors + else: + logger.warning(f"Invalid memory type: {memory_type}") + return [] + + if user_id not in monitor_dict or mem_cube_id not in monitor_dict[user_id]: + logger.warning( + f"MemoryMonitorManager not found for user {user_id}, " + f"mem_cube {mem_cube_id}, type {memory_type}" + ) + return [] + + manager = monitor_dict[user_id][mem_cube_id] + # Sort memories by recording_count in descending order and return top_k items + sorted_memories = sorted(manager.memories, key=lambda m: m.recording_count, reverse=True) + sorted_text_memories = [m.memory_text for m in sorted_memories[:top_k]] + return sorted_text_memories + + def get_monitors_info(self, user_id: str, mem_cube_id: str) -> dict[str, Any]: + """Retrieves monitoring information for a specific memory cube.""" + if ( + user_id not in self.working_memory_monitors + or mem_cube_id not in self.working_memory_monitors[user_id] + ): + logger.warning( + f"MemoryMonitorManager not found for user {user_id}, mem_cube {mem_cube_id}" + ) + return {} + + info_dict = {} + for manager in [ + self.working_memory_monitors[user_id][mem_cube_id], + self.activation_memory_monitors[user_id][mem_cube_id], + ]: + info_dict[str(type(manager))] = { + "user_id": user_id, + "mem_cube_id": mem_cube_id, + "memory_count": manager.memory_size, + "max_capacity": manager.max_capacity, + "top_memories": self.get_scheduler_working_memories(user_id, mem_cube_id, top_k=1), + } + return info_dict def detect_intent( self, @@ -55,28 +251,11 @@ def detect_intent( q_list=q_list, working_memory_list=text_working_memory, ) - response = self._chat_llm.generate([{"role": "user", "content": prompt}]) - response = extract_json_dict(response) - return response - - def update_freq( - self, - answer: str, - activation_memory_freq_list: list[dict], - prompt_name="freq_detecting", - ) -> list[dict]: - """ - Use LLM to detect which memories in activation_memory_freq_list appear in the answer, - increment their count by 1, and return the updated list. - """ - prompt = self.build_prompt( - template_name=prompt_name, - answer=answer, - activation_memory_freq_list=activation_memory_freq_list, - ) - response = self._chat_llm.generate([{"role": "user", "content": prompt}]) + response = self._process_llm.generate([{"role": "user", "content": prompt}]) try: - result = json.loads(response) + response = extract_json_dict(response) + assert ("trigger_retrieval" in response) and ("missing_evidences" in response) except Exception: - result = activation_memory_freq_list - return result + logger.error(f"Fail to extract json dict from response: {response}") + response = {"trigger_retrieval": False, "missing_evidences": q_list} + return response diff --git a/src/memos/mem_scheduler/modules/rabbitmq_service.py b/src/memos/mem_scheduler/modules/rabbitmq_service.py new file mode 100644 index 000000000..c60bc6c21 --- /dev/null +++ b/src/memos/mem_scheduler/modules/rabbitmq_service.py @@ -0,0 +1,317 @@ +import json +import ssl +import threading +import time + +from pathlib import Path +from queue import Queue + +from memos.configs.mem_scheduler import AuthConfig, RabbitMQConfig +from memos.dependency import require_python_package +from memos.log import get_logger +from memos.mem_scheduler.modules.base import BaseSchedulerModule +from memos.mem_scheduler.modules.schemas import DIRECT_EXCHANGE_TYPE, FANOUT_EXCHANGE_TYPE + + +logger = get_logger(__name__) + + +class RabbitMQSchedulerModule(BaseSchedulerModule): + @require_python_package( + import_name="pika", + install_command="pip install pika", + install_link="https://pika.readthedocs.io/en/stable/index.html", + ) + def __init__(self): + """ + Initialize RabbitMQ connection settings. + """ + super().__init__() + + # RabbitMQ settings + self.rabbitmq_config: RabbitMQConfig | None = None + self.rabbit_queue_name = "memos-scheduler" + self.rabbitmq_exchange_name = "memos-fanout" + self.rabbitmq_exchange_type = FANOUT_EXCHANGE_TYPE + self.rabbitmq_connection = None + self.rabbitmq_channel = None + + # fixed params + self.rabbitmq_message_cache_max_size = 10 # Max 10 messages + self.rabbitmq_message_cache = Queue(maxsize=self.rabbitmq_message_cache_max_size) + self.rabbitmq_connection_attempts = 3 # Max retry attempts on connection failure + self.rabbitmq_retry_delay = 5 # Delay (seconds) between retries + self.rabbitmq_heartbeat = 60 # Heartbeat interval (seconds) for connectio + self.rabbitmq_conn_max_waiting_seconds = 30 + self.rabbitmq_conn_sleep_seconds = 1 + + # Thread management + self._rabbitmq_io_loop_thread = None # For IOLoop execution + self._rabbitmq_stop_flag = False # Graceful shutdown flag + self._rabbitmq_lock = threading.Lock() # Ensure thread safety + + def is_rabbitmq_connected(self) -> bool: + """Check if RabbitMQ connection is alive""" + return ( + self.rabbitmq_connection + and self.rabbitmq_connection.is_open + and self.rabbitmq_channel + and self.rabbitmq_channel.is_open + ) + + def initialize_rabbitmq( + self, config: dict | None | RabbitMQConfig = None, config_path: str | Path | None = None + ): + """ + Establish connection to RabbitMQ using pika. + """ + from pika.adapters.select_connection import SelectConnection + + if config is None: + if config_path is None and AuthConfig.default_config_exists(): + auth_config = AuthConfig.from_local_yaml() + elif Path(config_path).exists(): + auth_config = AuthConfig.from_local_yaml(config_path=config_path) + else: + logger.error("Fail to initialize auth_config") + return + self.rabbitmq_config = auth_config.rabbitmq + elif isinstance(config, RabbitMQConfig): + self.rabbitmq_config = config + elif isinstance(config, dict): + self.rabbitmq_config = AuthConfig.from_dict(config).rabbitmq + else: + logger.error("Not implemented") + + # Start connection process + parameters = self.get_rabbitmq_connection_param() + self.rabbitmq_connection = SelectConnection( + parameters, + on_open_callback=self.on_rabbitmq_connection_open, + on_open_error_callback=self.on_rabbitmq_connection_error, + on_close_callback=self.on_rabbitmq_connection_closed, + ) + + # Start IOLoop in dedicated thread + self._io_loop_thread = threading.Thread( + target=self.rabbitmq_connection.ioloop.start, daemon=True + ) + self._io_loop_thread.start() + logger.info("RabbitMQ connection process started") + + def get_rabbitmq_queue_size(self) -> int: + """Get the current number of messages in the queue. + + Returns: + int: Number of messages in the queue. + Returns -1 if there's an error or no active connection. + """ + if self.rabbitmq_exchange_type != DIRECT_EXCHANGE_TYPE: + logger.warning("Queue size can only be checked for direct exchanges") + return None + + with self._rabbitmq_lock: + if not self.is_rabbitmq_connected(): + logger.warning("No active connection to check queue size") + return -1 + + # Declare queue passively (only checks existence, doesn't create) + # Using passive=True prevents accidental queue creation + result = self.rabbitmq_channel.queue_declare( + queue=self.rabbit_queue_name, + durable=True, # Match the original queue durability setting + passive=True, # Only check queue existence, don't create + ) + + if result is None: + return 0 + # Return the message count from the queue declaration result + return result.method.message_count + + def get_rabbitmq_connection_param(self): + import pika + + credentials = pika.PlainCredentials( + username=self.rabbitmq_config.user_name, + password=self.rabbitmq_config.password, + erase_on_connect=self.rabbitmq_config.erase_on_connect, + ) + if self.rabbitmq_config.port == 5671: + context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + context.check_hostname = False + context.verify_mode = False + return pika.ConnectionParameters( + host=self.rabbitmq_config.host_name, + port=self.rabbitmq_config.port, + virtual_host=self.rabbitmq_config.virtual_host, + credentials=credentials, + ssl_options=pika.SSLOptions(context), + connection_attempts=self.rabbitmq_connection_attempts, + retry_delay=self.rabbitmq_retry_delay, + heartbeat=self.rabbitmq_heartbeat, + ) + else: + return pika.ConnectionParameters( + host=self.rabbitmq_config.host_name, + port=self.rabbitmq_config.port, + virtual_host=self.rabbitmq_config.virtual_host, + credentials=credentials, + connection_attempts=self.rabbitmq_connection_attempts, + retry_delay=self.rabbitmq_retry_delay, + heartbeat=self.rabbitmq_heartbeat, + ) + + # Connection lifecycle callbacks + def on_rabbitmq_connection_open(self, connection): + """Called when connection is established.""" + logger.debug("Connection opened") + connection.channel(on_open_callback=self.on_rabbitmq_channel_open) + + def on_rabbitmq_connection_error(self, connection, error): + """Called if connection fails to open.""" + logger.error(f"Connection failed: {error}") + self.rabbit_reconnect() + + def on_rabbitmq_connection_closed(self, connection, reason): + """Called when connection closes.""" + logger.warning(f"Connection closed: {reason}") + if not self._rabbitmq_stop_flag: + self.rabbit_reconnect() + + # Channel lifecycle callbacks + def on_rabbitmq_channel_open(self, channel): + """Called when channel is ready.""" + self.rabbitmq_channel = channel + logger.debug("Channel opened") + + # Setup exchange and queue + channel.exchange_declare( + exchange=self.rabbitmq_exchange_name, + exchange_type=self.rabbitmq_exchange_type, + durable=True, + callback=self.on_rabbitmq_exchange_declared, + ) + + def on_rabbitmq_exchange_declared(self, frame): + """Called when exchange is ready.""" + self.rabbitmq_channel.queue_declare( + queue=self.rabbit_queue_name, durable=True, callback=self.on_rabbitmq_queue_declared + ) + + def on_rabbitmq_queue_declared(self, frame): + """Called when queue is ready.""" + self.rabbitmq_channel.queue_bind( + exchange=self.rabbitmq_exchange_name, + queue=self.rabbit_queue_name, + routing_key=self.rabbit_queue_name, + callback=self.on_rabbitmq_bind_ok, + ) + + def on_rabbitmq_bind_ok(self, frame): + """Final setup step when bind is complete.""" + logger.info("RabbitMQ setup completed") + + def on_rabbitmq_message(self, channel, method, properties, body): + """Handle incoming messages. Only for test.""" + try: + print(f"Received message: {body.decode()}") + self.rabbitmq_message_cache.put_nowait({"properties": properties, "body": body}) + print(f"message delivery_tag: {method.delivery_tag}") + channel.basic_ack(delivery_tag=method.delivery_tag) + except Exception as e: + logger.error(f"Message handling failed: {e}") + + def wait_for_connection_ready(self): + start_time = time.time() + while not self.is_rabbitmq_connected(): + delta_time = time.time() - start_time + if delta_time > self.rabbitmq_conn_max_waiting_seconds: + logger.error("Failed to start consuming: Connection timeout") + return False + self.rabbit_reconnect() + time.sleep(self.rabbitmq_conn_sleep_seconds) # Reduced frequency of checks + + # Message handling + def rabbitmq_start_consuming(self): + """Start consuming messages asynchronously.""" + self.wait_for_connection_ready() + + self.rabbitmq_channel.basic_consume( + queue=self.rabbit_queue_name, + on_message_callback=self.on_rabbitmq_message, + auto_ack=False, + ) + logger.info("Started rabbitmq consuming messages") + + def rabbitmq_publish_message(self, message: dict): + """ + Publish a message to RabbitMQ. + """ + import pika + + with self._rabbitmq_lock: + if not self.is_rabbitmq_connected(): + logger.error("Cannot publish - no active connection") + return False + + try: + self.rabbitmq_channel.basic_publish( + exchange=self.rabbitmq_exchange_name, + routing_key=self.rabbit_queue_name, + body=json.dumps(message), + properties=pika.BasicProperties( + delivery_mode=2, # Persistent + ), + mandatory=True, + ) + logger.debug(f"Published message: {message}") + return True + except Exception as e: + logger.error(f"Failed to publish message: {e}") + self.rabbit_reconnect() + return False + + # Connection management + def rabbit_reconnect(self): + """Schedule reconnection attempt.""" + logger.info("Attempting to reconnect...") + if self.rabbitmq_connection and not self.rabbitmq_connection.is_closed: + self.rabbitmq_connection.ioloop.stop() + + # Reset connection state + self.rabbitmq_channel = None + self.initialize_rabbitmq() + + def rabbitmq_close(self): + """Gracefully shutdown connection.""" + with self._rabbitmq_lock: + self._rabbitmq_stop_flag = True + + # Close channel if open + if self.rabbitmq_channel and self.rabbitmq_channel.is_open: + try: + self.rabbitmq_channel.close() + except Exception as e: + logger.warning(f"Error closing channel: {e}") + + # Close connection if open + if self.rabbitmq_connection: + if self.rabbitmq_connection.is_open: + try: + self.rabbitmq_connection.close() + except Exception as e: + logger.warning(f"Error closing connection: {e}") + + # Stop IOLoop if running + try: + self.rabbitmq_connection.ioloop.stop() + except Exception as e: + logger.warning(f"Error stopping IOLoop: {e}") + + # Wait for IOLoop thread to finish + if self._io_loop_thread and self._io_loop_thread.is_alive(): + self._io_loop_thread.join(timeout=5) + if self._io_loop_thread.is_alive(): + logger.warning("IOLoop thread did not terminate cleanly") + + logger.info("RabbitMQ connection closed") diff --git a/src/memos/mem_scheduler/modules/redis_service.py b/src/memos/mem_scheduler/modules/redis_service.py index 8e57cdde7..4a6ad35bd 100644 --- a/src/memos/mem_scheduler/modules/redis_service.py +++ b/src/memos/mem_scheduler/modules/redis_service.py @@ -2,11 +2,9 @@ import threading from collections.abc import Callable +from typing import Any -import redis - -from redis import Redis - +from memos.dependency import require_python_package from memos.log import get_logger from memos.mem_scheduler.modules.base import BaseSchedulerModule @@ -15,6 +13,11 @@ class RedisSchedulerModule(BaseSchedulerModule): + @require_python_package( + import_name="redis", + install_command="pip install redis", + install_link="https://redis.readthedocs.io/en/stable/", + ) def __init__(self): """ intent_detector: Object used for intent recognition (such as the above IntentDetector) @@ -35,23 +38,25 @@ def __init__(self): self._redis_listener_loop: asyncio.AbstractEventLoop | None = None @property - def redis(self) -> Redis: + def redis(self) -> Any: return self._redis_conn @redis.setter - def redis(self, value: Redis) -> None: + def redis(self, value: Any) -> None: self._redis_conn = value def initialize_redis( self, redis_host: str = "localhost", redis_port: int = 6379, redis_db: int = 0 ): + import redis + self.redis_host = redis_host self.redis_port = redis_port self.redis_db = redis_db try: logger.debug(f"Connecting to Redis at {redis_host}:{redis_port}/{redis_db}") - self._redis_conn = Redis( + self._redis_conn = redis.Redis( host=self.redis_host, port=self.redis_port, db=self.redis_db, decode_responses=True ) # test conn @@ -63,21 +68,21 @@ def initialize_redis( self._redis_conn.xtrim("user:queries:stream", self.query_list_capacity) return self._redis_conn - async def add_message_stream(self, message: dict): + async def redis_add_message_stream(self, message: dict): logger.debug(f"add_message_stream: {message}") return self._redis_conn.xadd("user:queries:stream", message) - async def consume_message_stream(self, message: dict): + async def redis_consume_message_stream(self, message: dict): logger.debug(f"consume_message_stream: {message}") - def _run_listener_async(self, handler: Callable): + def _redis_run_listener_async(self, handler: Callable): """Run the async listener in a separate thread""" self._redis_listener_loop = asyncio.new_event_loop() asyncio.set_event_loop(self._redis_listener_loop) async def listener_wrapper(): try: - await self._listen_query_stream(handler) + await self.__redis_listen_query_stream(handler) except Exception as e: logger.error(f"Listener thread error: {e}") finally: @@ -85,8 +90,12 @@ async def listener_wrapper(): self._redis_listener_loop.run_until_complete(listener_wrapper()) - async def _listen_query_stream(self, handler=None, last_id: str = "$", block_time: int = 2000): + async def __redis_listen_query_stream( + self, handler=None, last_id: str = "$", block_time: int = 2000 + ): """Internal async stream listener""" + import redis + self._redis_listener_running = True while self._redis_listener_running: try: @@ -99,6 +108,7 @@ async def _listen_query_stream(self, handler=None, last_id: str = "$", block_tim for _, stream_messages in messages: for message_id, message_data in stream_messages: try: + print(f"deal with message_data {message_data}") await handler(message_data) last_id = message_id except Exception as e: @@ -112,17 +122,17 @@ async def _listen_query_stream(self, handler=None, last_id: str = "$", block_tim logger.error(f"Unexpected error: {e}") await asyncio.sleep(1) - def start_listening(self, handler: Callable | None = None): + def redis_start_listening(self, handler: Callable | None = None): """Start the Redis stream listener in a background thread""" if self._redis_listener_thread and self._redis_listener_thread.is_alive(): logger.warning("Listener is already running") return if handler is None: - handler = self.consume_message_stream + handler = self.redis_consume_message_stream self._redis_listener_thread = threading.Thread( - target=self._run_listener_async, + target=self._redis_run_listener_async, args=(handler,), daemon=True, name="RedisListenerThread", @@ -130,13 +140,7 @@ def start_listening(self, handler: Callable | None = None): self._redis_listener_thread.start() logger.info("Started Redis stream listener thread") - def close(self): - """Close Redis connection""" - if self._redis_conn is not None: - self._redis_conn.close() - self._redis_conn = None - - def stop_listening(self): + def redis_stop_listening(self): """Stop the listener thread gracefully""" self._redis_listener_running = False if self._redis_listener_thread and self._redis_listener_thread.is_alive(): @@ -144,3 +148,9 @@ def stop_listening(self): if self._redis_listener_thread.is_alive(): logger.warning("Listener thread did not stop gracefully") logger.info("Redis stream listener stopped") + + def redis_close(self): + """Close Redis connection""" + if self._redis_conn is not None: + self._redis_conn.close() + self._redis_conn = None diff --git a/src/memos/mem_scheduler/modules/retriever.py b/src/memos/mem_scheduler/modules/retriever.py index 95d00aa87..219863d43 100644 --- a/src/memos/mem_scheduler/modules/retriever.py +++ b/src/memos/mem_scheduler/modules/retriever.py @@ -1,41 +1,268 @@ +import logging + +from memos.configs.mem_scheduler import BaseSchedulerConfig +from memos.dependency import require_python_package +from memos.llms.base import BaseLLM from memos.log import get_logger +from memos.mem_cube.general import GeneralMemCube from memos.mem_scheduler.modules.base import BaseSchedulerModule +from memos.mem_scheduler.modules.schemas import ( + TreeTextMemory_SEARCH_METHOD, +) +from memos.mem_scheduler.utils import ( + extract_json_dict, + is_all_chinese, + is_all_english, + transform_name_to_key, +) +from memos.memories.textual.tree import TextualMemoryItem, TreeTextMemory logger = get_logger(__name__) class SchedulerRetriever(BaseSchedulerModule): - def __init__(self, chat_llm, context_window_size=5): + def __init__(self, process_llm: BaseLLM, config: BaseSchedulerConfig): + super().__init__() + + self.config: BaseSchedulerConfig = config + self.process_llm = process_llm + + # hyper-parameters + self.filter_similarity_threshold = 0.75 + self.filter_min_length_threshold = 6 + + # log function callbacks + self.log_working_memory_replacement = None + + def search( + self, query: str, mem_cube: GeneralMemCube, top_k: int, method=TreeTextMemory_SEARCH_METHOD + ): + """Search in text memory with the given query. + + Args: + query: The search query string + top_k: Number of top results to return + method: Search method to use + + Returns: + Search results or None if not implemented + """ + text_mem_base = mem_cube.text_mem + try: + if method == TreeTextMemory_SEARCH_METHOD: + assert isinstance(text_mem_base, TreeTextMemory) + results_long_term = text_mem_base.search( + query=query, top_k=top_k, memory_type="LongTermMemory" + ) + results_user = text_mem_base.search( + query=query, top_k=top_k, memory_type="UserMemory" + ) + results = results_long_term + results_user + else: + raise NotImplementedError(str(type(text_mem_base))) + except Exception as e: + logger.error(f"Fail to search. The exeption is {e}.", exc_info=True) + results = [] + return results + + @require_python_package( + import_name="sklearn", + install_command="pip install scikit-learn", + install_link="https://scikit-learn.org/stable/install.html", + ) + def filter_similar_memories( + self, text_memories: list[str], similarity_threshold: float = 0.75 + ) -> list[str]: """ - monitor: Object used to acquire monitoring information - mem_cube: Object/interface for querying the underlying database - context_window_size: Size of the context window for conversation history + Filters out low-quality or duplicate memories based on text similarity. + + Args: + text_memories: List of text memories to filter + similarity_threshold: Threshold for considering memories duplicates (0.0-1.0) + Higher values mean stricter filtering + + Returns: + List of filtered memories with duplicates removed """ - super().__init__() + from sklearn.feature_extraction.text import TfidfVectorizer + from sklearn.metrics.pairwise import cosine_similarity + + if not text_memories: + logging.warning("Received empty memories list - nothing to filter") + return [] + + for idx in range(len(text_memories)): + if not isinstance(text_memories[idx], str): + logger.error( + f"{text_memories[idx]} in memories is not a string," + f" and now has been transformed to be a string." + ) + text_memories[idx] = str(text_memories[idx]) + + try: + # Step 1: Vectorize texts using TF-IDF + vectorizer = TfidfVectorizer() + tfidf_matrix = vectorizer.fit_transform(text_memories) + + # Step 2: Calculate pairwise similarity matrix + similarity_matrix = cosine_similarity(tfidf_matrix) + + # Step 3: Identify duplicates + to_keep = [] + removal_reasons = {} - self.monitors = {} - self.context_window_size = context_window_size + for current_idx in range(len(text_memories)): + is_duplicate = False - self._chat_llm = chat_llm - self._current_mem_cube = None + # Compare with already kept memories + for kept_idx in to_keep: + similarity_score = similarity_matrix[current_idx, kept_idx] - @property - def memory_texts(self) -> list[str]: - """The memory cube associated with this MemChat.""" - return self._memory_text_list + if similarity_score > similarity_threshold: + is_duplicate = True + # Generate removal reason with sample text + removal_reasons[current_idx] = ( + f"Memory too similar (score: {similarity_score:.2f}) to kept memory #{kept_idx}. " + f"Kept: '{text_memories[kept_idx][:100]}...' | " + f"Removed: '{text_memories[current_idx][:100]}...'" + ) + logger.info(removal_reasons) + break - @memory_texts.setter - def memory_texts(self, value: list[str]) -> None: - """The memory cube associated with this MemChat.""" - self._memory_text_list = value + if not is_duplicate: + to_keep.append(current_idx) - def fetch_context(self): + # Return filtered memories + return [text_memories[i] for i in sorted(to_keep)] + + except Exception as e: + logging.error(f"Error filtering memories: {e!s}") + return text_memories # Return original list if error occurs + + def filter_too_short_memories( + self, text_memories: list[str], min_length_threshold: int = 20 + ) -> list[str]: """ - Extract the context window from the current conversation - conversation_history: a list (in chronological order) + Filters out text memories that fall below the minimum length requirement. + Handles both English (word count) and Chinese (character count) differently. + + Args: + text_memories: List of text memories to be filtered + min_length_threshold: Minimum length required to keep a memory. + For English: word count, for Chinese: character count. + + Returns: + List of filtered memories meeting the length requirement """ - return self._memory_text_list[-self.context_window_size :] + if not text_memories: + logging.debug("Empty memories list received in short memory filter") + return [] + + filtered_memories = [] + removed_count = 0 + + for memory in text_memories: + stripped_memory = memory.strip() + if not stripped_memory: # Skip empty/whitespace memories + removed_count += 1 + continue + + # Determine measurement method based on language + if is_all_english(stripped_memory): + length = len(stripped_memory.split()) # Word count for English + elif is_all_chinese(stripped_memory): + length = len(stripped_memory) # Character count for Chinese + else: + logger.debug( + f"Mixed-language memory, using character count: {stripped_memory[:50]}..." + ) + length = len(stripped_memory) # Default to character count + + if length >= min_length_threshold: + filtered_memories.append(memory) + else: + removed_count += 1 + + if removed_count > 0: + logger.info( + f"Filtered out {removed_count} short memories " + f"(below {min_length_threshold} units). " + f"Total remaining: {len(filtered_memories)}" + ) + + return filtered_memories + + def replace_working_memory( + self, + queries: list[str], + user_id: str, + mem_cube_id: str, + mem_cube: GeneralMemCube, + original_memory: list[TextualMemoryItem], + new_memory: list[TextualMemoryItem], + top_k: int = 10, + ) -> None | list[TextualMemoryItem]: + """Replace working memory with new memories after reranking.""" + memories_with_new_order = None + text_mem_base = mem_cube.text_mem + if isinstance(text_mem_base, TreeTextMemory): + text_mem_base: TreeTextMemory = text_mem_base + combined_memory = original_memory + new_memory + memory_map = { + transform_name_to_key(name=mem_obj.memory): mem_obj for mem_obj in combined_memory + } + combined_text_memory = [transform_name_to_key(name=m.memory) for m in combined_memory] + + # apply filters + filtered_combined_text_memory = self.filter_similar_memories( + text_memories=combined_text_memory, + similarity_threshold=self.filter_similarity_threshold, + ) + + filtered_combined_text_memory = self.filter_too_short_memories( + text_memories=filtered_combined_text_memory, + min_length_threshold=self.filter_min_length_threshold, + ) + + unique_memory = list(dict.fromkeys(filtered_combined_text_memory)) + + try: + prompt = self.build_prompt( + "memory_reranking", + queries=queries, + current_order=unique_memory, + staging_buffer=[], + ) + response = self.process_llm.generate([{"role": "user", "content": prompt}]) + response = extract_json_dict(response) + text_memories_with_new_order = response.get("new_order", [])[:top_k] + except Exception as e: + logger.error(f"Fail to rerank with LLM, Exeption: {e}.", exc_info=True) + text_memories_with_new_order = unique_memory[:top_k] + + memories_with_new_order = [] + for text in text_memories_with_new_order: + normalized_text = transform_name_to_key(name=text) + if text in memory_map: + memories_with_new_order.append(memory_map[normalized_text]) + else: + logger.warning( + f"Memory text not found in memory map. text: {text}; keys of memory_map: {memory_map.keys()}" + ) + + text_mem_base.replace_working_memory(memories_with_new_order) + logger.info( + f"The working memory has been replaced with {len(memories_with_new_order)} new memories." + ) + self.log_working_memory_replacement( + original_memory=original_memory, + new_memory=memories_with_new_order, + user_id=user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + ) + else: + logger.error("memory_base is not supported") - def retrieve(self, query: str, memory_texts: list[str], top_k: int = 5) -> list[str]: - return None + return memories_with_new_order diff --git a/src/memos/mem_scheduler/modules/schemas.py b/src/memos/mem_scheduler/modules/schemas.py index 0c39c9b73..51e13d01e 100644 --- a/src/memos/mem_scheduler/modules/schemas.py +++ b/src/memos/mem_scheduler/modules/schemas.py @@ -1,35 +1,65 @@ +import json + from datetime import datetime from pathlib import Path -from typing import ClassVar, TypeVar +from typing import ClassVar, NewType, TypeVar from uuid import uuid4 -from pydantic import BaseModel, Field +from pydantic import BaseModel, Field, computed_field from typing_extensions import TypedDict +from memos.log import get_logger from memos.mem_cube.general import GeneralMemCube +logger = get_logger(__name__) + + FILE_PATH = Path(__file__).absolute() BASE_DIR = FILE_PATH.parent.parent.parent.parent.parent QUERY_LABEL = "query" ANSWER_LABEL = "answer" +ADD_LABEL = "add" TreeTextMemory_SEARCH_METHOD = "tree_text_memory_search" TextMemory_SEARCH_METHOD = "text_memory_search" -DEFAULT_ACTIVATION_MEM_SIZE = 5 +DIRECT_EXCHANGE_TYPE = "direct" +FANOUT_EXCHANGE_TYPE = "fanout" +DEFAULT_WORKING_MEM_MONITOR_SIZE_LIMIT = 20 +DEFAULT_ACTIVATION_MEM_MONITOR_SIZE_LIMIT = 5 DEFAULT_ACT_MEM_DUMP_PATH = f"{BASE_DIR}/outputs/mem_scheduler/mem_cube_scheduler_test.kv_cache" DEFAULT_THREAD__POOL_MAX_WORKERS = 5 DEFAULT_CONSUME_INTERVAL_SECONDS = 3 NOT_INITIALIZED = -1 BaseModelType = TypeVar("T", bound="BaseModel") +# web log +LONG_TERM_MEMORY_TYPE = "LongTermMemory" +USER_MEMORY_TYPE = "UserMemory" +WORKING_MEMORY_TYPE = "WorkingMemory" +TEXT_MEMORY_TYPE = "TextMemory" +ACTIVATION_MEMORY_TYPE = "ActivationMemory" +PARAMETER_MEMORY_TYPE = "ParameterMemory" +USER_INPUT_TYPE = "UserInput" +NOT_APPLICABLE_TYPE = "NotApplicable" + +# monitors +MONITOR_WORKING_MEMORY_TYPE = "MonitorWorkingMemoryType" +MONITOR_ACTIVATION_MEMORY_TYPE = "MonitorActivationMemoryType" + + +# new types +UserID = NewType("UserID", str) +MemCubeID = NewType("CubeID", str) + +# ************************* Public ************************* class DictConversionMixin: def to_dict(self) -> dict: """Convert the instance to a dictionary.""" return { - **self.dict(), + **self.model_dump(), # 替换 self.dict() "timestamp": self.timestamp.isoformat() if hasattr(self, "timestamp") else None, } @@ -40,10 +70,25 @@ def from_dict(cls: type[BaseModelType], data: dict) -> BaseModelType: data["timestamp"] = datetime.fromisoformat(data["timestamp"]) return cls(**data) + def __str__(self) -> str: + """Convert the instance to a JSON string with indentation of 4 spaces. + This will be used when str() or print() is called on the instance. + + Returns: + str: A JSON string representation of the instance with 4-space indentation. + """ + return json.dumps( + self.to_dict(), + indent=4, + ensure_ascii=False, + default=str, # 处理无法序列化的对象 + ) + class Config: json_encoders: ClassVar[dict[type, object]] = {datetime: lambda v: v.isoformat()} +# ************************* Messages ************************* class ScheduleMessageItem(BaseModel, DictConversionMixin): item_id: str = Field(description="uuid", default_factory=lambda: str(uuid4())) user_id: str = Field(..., description="user id") @@ -68,7 +113,6 @@ def to_dict(self) -> dict: "item_id": self.item_id, "user_id": self.user_id, "cube_id": self.mem_cube_id, - "message_id": self.message_id, "label": self.label, "cube": "Not Applicable", # Custom cube serialization "content": self.content, @@ -82,7 +126,6 @@ def from_dict(cls, data: dict) -> "ScheduleMessageItem": item_id=data.get("item_id", str(uuid4())), user_id=data["user_id"], cube_id=data["cube_id"], - message_id=data.get("message_id", str(uuid4())), label=data["label"], cube="Not Applicable", # Custom cube deserialization content=data["content"], @@ -130,7 +173,8 @@ class ScheduleLogForWebItem(BaseModel, DictConversionMixin): ..., description="Identifier for the memcube associated with this log entry" ) label: str = Field(..., description="Label categorizing the type of log") - log_title: str = Field(..., description="Title or brief summary of the log content") + from_memory_type: str = Field(..., description="Source memory type") + to_memory_type: str = Field(..., description="Destination memory type") log_content: str = Field(..., description="Detailed content of the log entry") current_memory_sizes: MemorySizes = Field( default_factory=lambda: dict(DEFAULT_MEMORY_SIZES), @@ -144,3 +188,141 @@ class ScheduleLogForWebItem(BaseModel, DictConversionMixin): default_factory=datetime.now, description="Timestamp indicating when the log entry was created", ) + + +# ************************* Monitor ************************* +class MemoryMonitorItem(BaseModel, DictConversionMixin): + item_id: str = Field( + description="Unique identifier for the memory item", default_factory=lambda: str(uuid4()) + ) + memory_text: str = Field( + ..., + description="The actual content of the memory", + min_length=1, + max_length=10000, # Prevent excessively large memory texts + ) + importance_score: float = Field( + default=NOT_INITIALIZED, + description="Numerical score representing the memory's importance", + ge=NOT_INITIALIZED, # Minimum value of 0 + ) + recording_count: int = Field( + default=1, + description="How many times this memory has been recorded", + ge=1, # Greater than or equal to 1 + ) + + def get_score(self) -> float: + """ + Calculate the effective score for the memory item. + + Returns: + float: The importance_score if it has been initialized (>=0), + otherwise the recording_count converted to float. + + Note: + This method provides a unified way to retrieve a comparable score + for memory items, regardless of whether their importance has been explicitly set. + """ + if self.importance_score == NOT_INITIALIZED: + # Return recording_count as float when importance_score is not initialized + return float(self.recording_count) + else: + # Return the initialized importance_score + return self.importance_score + + +class MemoryMonitorManager(BaseModel, DictConversionMixin): + user_id: str = Field(..., description="Required user identifier", min_length=1) + mem_cube_id: str = Field(..., description="Required memory cube identifier", min_length=1) + memories: list[MemoryMonitorItem] = Field( + default_factory=list, description="Collection of memory items" + ) + max_capacity: int | None = Field( + default=None, description="Maximum number of memories allowed (None for unlimited)", ge=1 + ) + + @computed_field + @property + def memory_size(self) -> int: + """Automatically calculated count of memory items.""" + return len(self.memories) + + def update_memories( + self, text_working_memories: list[str], partial_retention_number: int + ) -> MemoryMonitorItem: + """ + Update memories based on text_working_memories. + + Args: + text_working_memories: List of memory texts to update + partial_retention_number: Number of top memories to keep by recording count + + Returns: + List of added or updated MemoryMonitorItem instances + """ + + # Validate partial_retention_number + if partial_retention_number < 0: + raise ValueError("partial_retention_number must be non-negative") + + # Create text lookup set + working_memory_set = set(text_working_memories) + + # Step 1: Update existing memories or add new ones + added_or_updated = [] + memory_text_map = {item.memory_text: item for item in self.memories} + + for text in text_working_memories: + if text in memory_text_map: + # Update existing memory + memory = memory_text_map[text] + memory.recording_count += 1 + added_or_updated.append(memory) + else: + # Add new memory + new_memory = MemoryMonitorItem(memory_text=text, recording_count=1) + self.memories.append(new_memory) + added_or_updated.append(new_memory) + + # Step 2: Identify memories to remove + # Sort memories by recording_count in descending order + sorted_memories = sorted(self.memories, key=lambda item: item.recording_count, reverse=True) + + # Keep the top N memories by recording_count + records_to_keep = { + memory.memory_text for memory in sorted_memories[:partial_retention_number] + } + + # Collect memories to remove: not in current working memory and not in top N + memories_to_remove = [ + memory + for memory in self.memories + if memory.memory_text not in working_memory_set + and memory.memory_text not in records_to_keep + ] + + # Step 3: Remove identified memories + for memory in memories_to_remove: + self.memories.remove(memory) + + # Step 4: Enforce max_capacity if set + if self.max_capacity is not None and len(self.memories) > self.max_capacity: + # Sort by importance and then recording count + sorted_memories = sorted( + self.memories, + key=lambda item: (item.importance_score, item.recording_count), + reverse=True, + ) + # Keep only the top max_capacity memories + self.memories = sorted_memories[: self.max_capacity] + + # Log the update result + logger.info( + f"Updated monitor manager for user {self.user_id}, mem_cube {self.mem_cube_id}: " + f"Total memories: {len(self.memories)}, " + f"Added/Updated: {len(added_or_updated)}, " + f"Removed: {len(memories_to_remove)} (excluding top {partial_retention_number} by recording_count)" + ) + + return added_or_updated diff --git a/src/memos/mem_scheduler/mos_for_test_scheduler.py b/src/memos/mem_scheduler/mos_for_test_scheduler.py new file mode 100644 index 000000000..600d6cf5e --- /dev/null +++ b/src/memos/mem_scheduler/mos_for_test_scheduler.py @@ -0,0 +1,143 @@ +from datetime import datetime + +from memos.configs.mem_os import MOSConfig +from memos.log import get_logger +from memos.mem_os.main import MOS +from memos.mem_scheduler.modules.schemas import ( + ANSWER_LABEL, + MONITOR_WORKING_MEMORY_TYPE, + QUERY_LABEL, + ScheduleMessageItem, +) + + +logger = get_logger(__name__) + + +class MOSForTestScheduler(MOS): + """This class is only to test abilities of mem scheduler""" + + def __init__(self, config: MOSConfig): + super().__init__(config) + + def _str_memories(self, memories: list[str]) -> str: + """Format memories for display.""" + if not memories: + return "No memories." + return "\n".join(f"{i + 1}. {memory}" for i, memory in enumerate(memories)) + + def chat(self, query: str, user_id: str | None = None) -> str: + """ + Chat with the MOS. + + Args: + query (str): The user's query. + + Returns: + str: The response from the MOS. + """ + target_user_id = user_id if user_id is not None else self.user_id + accessible_cubes = self.user_manager.get_user_cubes(target_user_id) + user_cube_ids = [cube.cube_id for cube in accessible_cubes] + if target_user_id not in self.chat_history_manager: + self._register_chat_history(target_user_id) + + chat_history = self.chat_history_manager[target_user_id] + + topk_for_scheduler = 2 + + if self.config.enable_textual_memory and self.mem_cubes: + memories_all = [] + for mem_cube_id, mem_cube in self.mem_cubes.items(): + if mem_cube_id not in user_cube_ids: + continue + if not mem_cube.text_mem: + continue + + # submit message to scheduler + if self.enable_mem_scheduler and self.mem_scheduler is not None: + message_item = ScheduleMessageItem( + user_id=target_user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + label=QUERY_LABEL, + content=query, + timestamp=datetime.now(), + ) + self.mem_scheduler.submit_messages(messages=[message_item]) + + self.mem_scheduler.monitor.register_memory_manager_if_not_exists( + user_id=user_id, + mem_cube_id=mem_cube_id, + memory_monitors=self.mem_scheduler.monitor.working_memory_monitors, + max_capacity=self.mem_scheduler.monitor.working_mem_monitor_capacity, + ) + + # from scheduler + scheduler_memories = self.mem_scheduler.monitor.get_monitor_memories( + user_id=target_user_id, + mem_cube_id=mem_cube_id, + memory_type=MONITOR_WORKING_MEMORY_TYPE, + top_k=topk_for_scheduler, + ) + memories_all.extend(scheduler_memories) + + # from mem_cube + memories = mem_cube.text_mem.search( + query, top_k=self.config.top_k - topk_for_scheduler + ) + text_memories = [m.memory for m in memories] + memories_all.extend(text_memories) + + memories_all = list(set(memories_all)) + + logger.info(f"🧠 [Memory] Searched memories:\n{self._str_memories(memories_all)}\n") + system_prompt = self._build_system_prompt(memories_all) + else: + system_prompt = self._build_system_prompt() + current_messages = [ + {"role": "system", "content": system_prompt}, + *chat_history.chat_history, + {"role": "user", "content": query}, + ] + past_key_values = None + + if self.config.enable_activation_memory: + assert self.config.chat_model.backend == "huggingface", ( + "Activation memory only used for huggingface backend." + ) + # TODO this only one cubes + for mem_cube_id, mem_cube in self.mem_cubes.items(): + if mem_cube_id not in user_cube_ids: + continue + if mem_cube.act_mem: + kv_cache = next(iter(mem_cube.act_mem.get_all()), None) + past_key_values = ( + kv_cache.memory if (kv_cache and hasattr(kv_cache, "memory")) else None + ) + break + # Generate response + response = self.chat_llm.generate(current_messages, past_key_values=past_key_values) + else: + response = self.chat_llm.generate(current_messages) + logger.info(f"🤖 [Assistant] {response}\n") + chat_history.chat_history.append({"role": "user", "content": query}) + chat_history.chat_history.append({"role": "assistant", "content": response}) + self.chat_history_manager[user_id] = chat_history + + # submit message to scheduler + for accessible_mem_cube in accessible_cubes: + mem_cube_id = accessible_mem_cube.cube_id + mem_cube = self.mem_cubes[mem_cube_id] + if self.enable_mem_scheduler and self.mem_scheduler is not None: + message_item = ScheduleMessageItem( + user_id=target_user_id, + mem_cube_id=mem_cube_id, + mem_cube=mem_cube, + label=ANSWER_LABEL, + content=response, + timestamp=datetime.now(), + ) + self.mem_scheduler.submit_messages(messages=[message_item]) + + return response diff --git a/src/memos/mem_scheduler/utils.py b/src/memos/mem_scheduler/utils.py index 91fe09447..3675b8553 100644 --- a/src/memos/mem_scheduler/utils.py +++ b/src/memos/mem_scheduler/utils.py @@ -1,4 +1,5 @@ import json +import re from pathlib import Path @@ -7,13 +8,41 @@ def extract_json_dict(text: str): text = text.strip() - patterns_to_remove = ["json'''", "latex'''", "'''"] + patterns_to_remove = ["json```", "```json", "latex```", "```latex", "```"] for pattern in patterns_to_remove: text = text.replace(pattern, "") - res = json.loads(text) + res = json.loads(text.strip()) return res +def transform_name_to_key(name): + """ + Normalize text by removing all punctuation marks, keeping only letters, numbers, and word characters. + + Args: + name (str): Input text to be processed + + Returns: + str: Processed text with all punctuation removed + """ + # Match all characters that are NOT: + # \w - word characters (letters, digits, underscore) + # \u4e00-\u9fff - Chinese/Japanese/Korean characters + # \s - whitespace + pattern = r"[^\w\u4e00-\u9fff\s]" + + # Substitute all matched punctuation marks with empty string + # re.UNICODE flag ensures proper handling of Unicode characters + normalized = re.sub(pattern, "", name, flags=re.UNICODE) + + # Optional: Collapse multiple whitespaces into single space + normalized = "_".join(normalized.split()) + + normalized = normalized.lower() + + return normalized + + def parse_yaml(yaml_file): yaml_path = Path(yaml_file) yaml_path = Path(yaml_file) @@ -24,3 +53,23 @@ def parse_yaml(yaml_file): data = yaml.safe_load(fr) return data + + +def is_all_english(input_string: str) -> bool: + """Determine if the string consists entirely of English characters (including spaces)""" + return all(char.isascii() or char.isspace() for char in input_string) + + +def is_all_chinese(input_string: str) -> bool: + """Determine if the string consists entirely of Chinese characters (including Chinese punctuation and spaces)""" + return all( + ("\u4e00" <= char <= "\u9fff") # Basic Chinese characters + or ("\u3400" <= char <= "\u4dbf") # Extension A + or ("\u20000" <= char <= "\u2a6df") # Extension B + or ("\u2a700" <= char <= "\u2b73f") # Extension C + or ("\u2b740" <= char <= "\u2b81f") # Extension D + or ("\u2b820" <= char <= "\u2ceaf") # Extension E + or ("\u2f800" <= char <= "\u2fa1f") # Extension F + or char.isspace() # Spaces + for char in input_string + ) diff --git a/src/memos/mem_user/persistent_user_manager.py b/src/memos/mem_user/persistent_user_manager.py new file mode 100644 index 000000000..e3c476262 --- /dev/null +++ b/src/memos/mem_user/persistent_user_manager.py @@ -0,0 +1,260 @@ +"""Persistent user management system for MemOS with configuration storage. + +This module extends the base UserManager to provide persistent storage +for user configurations and MOS instances. +""" + +import json + +from datetime import datetime +from typing import Any + +from sqlalchemy import Column, String, Text + +from memos.configs.mem_os import MOSConfig +from memos.log import get_logger +from memos.mem_user.user_manager import Base, UserManager + + +logger = get_logger(__name__) + + +class UserConfig(Base): + """User configuration model for the database.""" + + __tablename__ = "user_configs" + + user_id = Column(String, primary_key=True) + config_data = Column(Text, nullable=False) # JSON string of MOSConfig + created_at = Column(String, nullable=False) # ISO format timestamp + updated_at = Column(String, nullable=False) # ISO format timestamp + + def __repr__(self): + return f"" + + +class PersistentUserManager(UserManager): + """Extended UserManager with configuration persistence.""" + + def __init__(self, db_path: str | None = None, user_id: str = "root"): + """Initialize the persistent user manager. + + Args: + db_path (str, optional): Path to the SQLite database file. + If None, uses default path in MEMOS_DIR. + user_id (str, optional): User ID. If None, uses default user ID. + """ + super().__init__(db_path, user_id) + + # Create user_configs table + Base.metadata.create_all(bind=self.engine) + logger.info("PersistentUserManager initialized with configuration storage") + + def _convert_datetime_strings(self, obj: Any) -> Any: + """Recursively convert datetime strings back to datetime objects in config dict. + + Args: + obj: The object to process (dict, list, or primitive type) + + Returns: + The object with datetime strings converted to datetime objects + """ + if isinstance(obj, dict): + result = {} + for key, value in obj.items(): + if key == "created_at" and isinstance(value, str): + try: + result[key] = datetime.fromisoformat(value) + except ValueError: + # If parsing fails, keep the original string + result[key] = value + else: + result[key] = self._convert_datetime_strings(value) + return result + elif isinstance(obj, list): + return [self._convert_datetime_strings(item) for item in obj] + else: + return obj + + def save_user_config(self, user_id: str, config: MOSConfig) -> bool: + """Save user configuration to database. + + Args: + user_id (str): The user ID. + config (MOSConfig): The user's MOS configuration. + + Returns: + bool: True if successful, False otherwise. + """ + session = self._get_session() + try: + # Convert config to JSON string with proper datetime handling + config_dict = config.model_dump(mode="json") + config_json = json.dumps(config_dict, indent=2) + + from datetime import datetime + + now = datetime.now().isoformat() + + # Check if config already exists + existing_config = ( + session.query(UserConfig).filter(UserConfig.user_id == user_id).first() + ) + + if existing_config: + # Update existing config + existing_config.config_data = config_json + existing_config.updated_at = now + logger.info(f"Updated configuration for user {user_id}") + else: + # Create new config + user_config = UserConfig( + user_id=user_id, config_data=config_json, created_at=now, updated_at=now + ) + session.add(user_config) + logger.info(f"Saved new configuration for user {user_id}") + + session.commit() + return True + + except Exception as e: + session.rollback() + logger.error(f"Error saving user config for {user_id}: {e}") + return False + finally: + session.close() + + def get_user_config(self, user_id: str) -> MOSConfig | None: + """Get user configuration from database. + + Args: + user_id (str): The user ID. + + Returns: + MOSConfig | None: The user's configuration or None if not found. + """ + session = self._get_session() + try: + user_config = session.query(UserConfig).filter(UserConfig.user_id == user_id).first() + + if user_config: + config_dict = json.loads(user_config.config_data) + # Convert datetime strings back to datetime objects + config_dict = self._convert_datetime_strings(config_dict) + return MOSConfig(**config_dict) + return None + + except Exception as e: + logger.error(f"Error loading user config for {user_id}: {e}") + return None + finally: + session.close() + + def delete_user_config(self, user_id: str) -> bool: + """Delete user configuration from database. + + Args: + user_id (str): The user ID. + + Returns: + bool: True if successful, False otherwise. + """ + session = self._get_session() + try: + user_config = session.query(UserConfig).filter(UserConfig.user_id == user_id).first() + + if user_config: + session.delete(user_config) + session.commit() + logger.info(f"Deleted configuration for user {user_id}") + return True + return False + + except Exception as e: + session.rollback() + logger.error(f"Error deleting user config for {user_id}: {e}") + return False + finally: + session.close() + + def list_user_configs(self) -> dict[str, MOSConfig]: + """List all user configurations. + + Returns: + Dict[str, MOSConfig]: Dictionary mapping user_id to MOSConfig. + """ + session = self._get_session() + try: + user_configs = session.query(UserConfig).all() + result = {} + + for user_config in user_configs: + try: + config_dict = json.loads(user_config.config_data) + # Convert datetime strings back to datetime objects + config_dict = self._convert_datetime_strings(config_dict) + result[user_config.user_id] = MOSConfig(**config_dict) + except Exception as e: + logger.error(f"Error parsing config for user {user_config.user_id}: {e}") + continue + + return result + + except Exception as e: + logger.error(f"Error listing user configs: {e}") + return {} + finally: + session.close() + + def create_user_with_config( + self, user_name: str, config: MOSConfig, role=None, user_id: str | None = None + ) -> str: + """Create a new user with configuration. + + Args: + user_name (str): Name of the user. + config (MOSConfig): The user's configuration. + role: User role (optional, uses default from UserManager). + user_id (str, optional): Custom user ID. + + Returns: + str: The created user ID. + + Raises: + ValueError: If user_name already exists. + """ + # Create user using parent method + created_user_id = self.create_user(user_name, role, user_id) + + # Save configuration + if not self.save_user_config(created_user_id, config): + logger.error(f"Failed to save configuration for user {created_user_id}") + + return created_user_id + + def delete_user(self, user_id: str) -> bool: + """Delete a user and their configuration. + + Args: + user_id (str): The user ID. + + Returns: + bool: True if successful, False otherwise. + """ + # Delete configuration first + self.delete_user_config(user_id) + + # Delete user using parent method + return super().delete_user(user_id) + + def get_user_cube_access(self, user_id: str) -> list[str]: + """Get list of cube IDs that a user has access to. + + Args: + user_id (str): The user ID. + + Returns: + list[str]: List of cube IDs the user can access. + """ + cubes = self.get_user_cubes(user_id) + return [cube.cube_id for cube in cubes] diff --git a/src/memos/memories/activation/item.py b/src/memos/memories/activation/item.py index 0b66dbc36..fd8c1ce66 100644 --- a/src/memos/memories/activation/item.py +++ b/src/memos/memories/activation/item.py @@ -1,5 +1,6 @@ import uuid +from datetime import datetime from typing import Any from pydantic import BaseModel, ConfigDict, Field @@ -12,6 +13,16 @@ class ActivationMemoryItem(BaseModel): metadata: dict = {} +class KVCacheRecords(BaseModel): + text_memories: list[str] = Field( + default=[], + description="The list of text memories transformed to the activation memory.", + ) + timestamp: datetime = Field( + default_factory=datetime.now, description="submit time for schedule_messages" + ) + + class KVCacheItem(ActivationMemoryItem): id: str = Field(default_factory=lambda: str(uuid.uuid4())) memory: DynamicCache = Field( @@ -23,3 +34,17 @@ class KVCacheItem(ActivationMemoryItem): ) model_config = ConfigDict(arbitrary_types_allowed=True) # To allow DynamicCache as a field type + records: KVCacheRecords = KVCacheRecords() + + +class VLLMKVCacheItem(KVCacheItem): + """ + VLLM KV Cache Item that stores prompt strings instead of DynamicCache objects. + This is because vLLM handles KV cache on the server side via preloading. + """ + + # Override memory field to store prompt string instead of DynamicCache + memory: str = Field( + default="", + description="Prompt string used to preload KV cache in vLLM server", + ) diff --git a/src/memos/memories/activation/kv.py b/src/memos/memories/activation/kv.py index 76b399f1a..2355b7751 100644 --- a/src/memos/memories/activation/kv.py +++ b/src/memos/memories/activation/kv.py @@ -3,11 +3,10 @@ from datetime import datetime -import torch - from transformers import DynamicCache from memos.configs.memory import KVCacheMemoryConfig +from memos.dependency import require_python_package from memos.llms.factory import LLMFactory from memos.memories.activation.base import BaseActMemory from memos.memories.activation.item import KVCacheItem @@ -20,6 +19,10 @@ class KVCacheMemory(BaseActMemory): This memory type is designed to store and retrieve key-value caches. """ + @require_python_package( + import_name="torch", + install_link="https://pytorch.org/get-started/locally/", + ) def __init__(self, config: KVCacheMemoryConfig) -> None: """Initialize the KV Cache Memory with a configuration.""" self.config = config @@ -139,6 +142,8 @@ def load(self, dir: str) -> None: Args: dir (str): The directory containing the memory files. """ + import torch + file_path = os.path.join(dir, self.config.memory_filename) if not os.path.exists(file_path): @@ -197,6 +202,8 @@ def _concat_caches(self, caches: list[DynamicCache]) -> DynamicCache: Faster concat merge: for each layer, gather all caches' tensors and do a single torch.cat per layer. """ + import torch + assert caches, "Need at least one cache" if len(caches) == 1: return caches[0] @@ -215,7 +222,7 @@ def _concat_caches(self, caches: list[DynamicCache]) -> DynamicCache: return merged -def move_dynamic_cache_htod(dynamic_cache: DynamicCache, device: torch.device) -> DynamicCache: +def move_dynamic_cache_htod(dynamic_cache: DynamicCache, device: str) -> DynamicCache: """ In SimpleMemChat.run(), if self.config.enable_activation_memory is enabled, we load serialized kv cache from a [class KVCacheMemory] object, which has a kv_cache_memories on CPU. diff --git a/src/memos/memories/activation/vllmkv.py b/src/memos/memories/activation/vllmkv.py new file mode 100644 index 000000000..4b74115f4 --- /dev/null +++ b/src/memos/memories/activation/vllmkv.py @@ -0,0 +1,219 @@ +import os +import pickle + +from datetime import datetime + +from memos.configs.memory import KVCacheMemoryConfig +from memos.dependency import require_python_package +from memos.llms.factory import LLMFactory +from memos.memories.activation.base import BaseActMemory +from memos.memories.activation.item import VLLMKVCacheItem +from memos.memories.textual.item import TextualMemoryItem + + +class VLLMKVCacheMemory(BaseActMemory): + """ + VLLM Key-Value Cache Memory for activation memories. + This memory type is designed to store and retrieve prompt strings for vLLM KV cache preloading. + Unlike traditional KV cache that stores DynamicCache objects, vLLM handles cache on server side. + """ + + @require_python_package( + import_name="torch", + install_link="https://pytorch.org/get-started/locally/", + ) + def __init__(self, config: KVCacheMemoryConfig) -> None: + """Initialize the VLLM KV Cache Memory with a configuration.""" + self.config = config + self.llm = LLMFactory.from_config(config.extractor_llm) + self.kv_cache_memories: dict[str, VLLMKVCacheItem] = {} + + def extract(self, text: str) -> VLLMKVCacheItem: + """Extract memory based on the text. + + Uses the LLM to build vLLM KV cache from the provided text. + For vLLM, this means preloading the KV cache on the server side. + + Args: + text: Input text to extract memory from + + Returns: + Extracted VLLM KV cache item with prompt string + """ + # Build vLLM KV cache from the text using the LLM + # This preloads the cache on the vLLM server and returns the prompt + prompt = self.llm.build_vllm_kv_cache(text) + + # Create a VLLMKVCacheItem with the extracted prompt + cache_item = VLLMKVCacheItem( + memory=prompt, + metadata={"source_text": text, "extracted_at": datetime.now().isoformat()}, + ) + + return cache_item + + def add(self, memories: list[VLLMKVCacheItem]) -> None: + """Add memories to the VLLM KV cache memory. + + Args: + memories: List of VLLMKVCacheItem to add + """ + for memory in memories: + self.kv_cache_memories[memory.id] = memory + + def get_cache(self, cache_ids: list[str]) -> str | None: + """Get the prompt string for the most recent cache. + + Since vLLM handles KV cache on server side, we return the prompt string + that can be used for generation. For multiple caches, we return the most recent one. + + Args: + cache_ids: List of cache IDs to consider + + Returns: + Prompt string for the most recent cache or None if no caches found + """ + if not cache_ids: + return None + + # For vLLM, we typically want the most recent cache + # Return the prompt from the last cache ID in the list + latest_cache_id = cache_ids[-1] + cache_item = self.kv_cache_memories.get(latest_cache_id) + + if cache_item and cache_item.memory: + return cache_item.memory + + return None + + def get(self, memory_id: str) -> VLLMKVCacheItem | None: + """Get a memory by its ID. + + Args: + memory_id: ID of the memory to retrieve + + Returns: + VLLMKVCacheItem or None if not found + """ + return self.kv_cache_memories.get(memory_id) + + def get_by_ids(self, memory_ids: list[str]) -> list[VLLMKVCacheItem | None]: + """Get memories by their IDs. + + Args: + memory_ids: List of memory IDs to retrieve + + Returns: + List of VLLMKVCacheItem or None for missing ones + """ + results = [] + for memory_id in memory_ids: + memory = self.get(memory_id) + results.append(memory) + return results + + def get_all(self) -> list[VLLMKVCacheItem]: + """Get all memories. + + Returns: + List of all VLLMKVCacheItems in the memory + """ + return list(self.kv_cache_memories.values()) + + def delete(self, memory_ids: list[str]) -> None: + """Delete memories by their IDs. + + Args: + memory_ids: List of memory IDs to delete + """ + for memory_id in memory_ids: + self.kv_cache_memories.pop(memory_id, None) + + def delete_all(self) -> None: + """Delete all memories.""" + self.kv_cache_memories.clear() + + def from_textual_memory(self, mem: TextualMemoryItem) -> VLLMKVCacheItem: + """ + Convert a TextualMemoryItem to a VLLMKVCacheItem. + This method extracts the prompt string from the textual memory. + """ + # Build vLLM KV cache from the textual memory content + prompt = self.llm.build_vllm_kv_cache(mem.memory) + return VLLMKVCacheItem(memory=prompt, metadata=mem.metadata.model_dump()) + + def load(self, dir: str) -> None: + """Load memories from os.path.join(dir, self.config.memory_filename) + + Args: + dir (str): The directory containing the memory files. + """ + file_path = os.path.join(dir, self.config.memory_filename) + + if not os.path.exists(file_path): + # If file doesn't exist, start with empty memories + return + + try: + # Allow loading VLLMKVCacheItem types + import torch + + torch.serialization.add_safe_globals([VLLMKVCacheItem]) + + with open(file_path, "rb") as f: + data = pickle.load(f) + + if isinstance(data, dict): + # Load memories, handle both old and new formats + if "kv_cache_memories" in data: + memories = data["kv_cache_memories"] + if isinstance(memories, list): + # Convert list to dict format + self.kv_cache_memories = {item.id: item for item in memories} + else: + self.kv_cache_memories = memories + else: + # Reset to empty if no memories in data + self.kv_cache_memories = {} + elif isinstance(data, list): + # Backward compatibility: convert list to dict + self.kv_cache_memories = {item.id: item for item in data} + else: + # Reset to empty if data format is unexpected + self.kv_cache_memories = {} + + except (EOFError, pickle.UnpicklingError, Exception): + # If loading fails, start with empty memories + self.kv_cache_memories = {} + + def dump(self, dir: str) -> None: + """Dump memories to os.path.join(dir, self.config.memory_filename) + + Args: + dir (str): The directory where the memory files will be saved. + """ + file_path = os.path.join(dir, self.config.memory_filename) + + # Create directory if it doesn't exist + os.makedirs(dir, exist_ok=True) + + # Prepare data to save (only memories) + data = {"kv_cache_memories": self.kv_cache_memories} + + with open(file_path, "wb") as f: + pickle.dump(data, f, protocol=pickle.HIGHEST_PROTOCOL) + + def preload_kv_cache(self, cache_ids: list[str]) -> None: + """ + Preload KV cache on vLLM server for the given cache IDs. + This method calls build_vllm_kv_cache for each cache to ensure + the KV cache is loaded on the server side. + + Args: + cache_ids: List of cache IDs to preload + """ + for cache_id in cache_ids: + cache_item = self.kv_cache_memories.get(cache_id) + if cache_item and cache_item.memory: + # Re-preload the KV cache on the server + self.llm.build_vllm_kv_cache(cache_item.memory) diff --git a/src/memos/memories/factory.py b/src/memos/memories/factory.py index 9abb4d82f..9fdc67c53 100644 --- a/src/memos/memories/factory.py +++ b/src/memos/memories/factory.py @@ -3,6 +3,7 @@ from memos.configs.memory import MemoryConfigFactory from memos.memories.activation.base import BaseActMemory from memos.memories.activation.kv import KVCacheMemory +from memos.memories.activation.vllmkv import VLLMKVCacheMemory from memos.memories.base import BaseMemory from memos.memories.parametric.base import BaseParaMemory from memos.memories.parametric.lora import LoRAMemory @@ -20,6 +21,7 @@ class MemoryFactory(BaseMemory): "general_text": GeneralTextMemory, "tree_text": TreeTextMemory, "kv_cache": KVCacheMemory, + "vllm_kv_cache": VLLMKVCacheMemory, "lora": LoRAMemory, } diff --git a/src/memos/memories/textual/general.py b/src/memos/memories/textual/general.py index 75d4e1add..2f15ab730 100644 --- a/src/memos/memories/textual/general.py +++ b/src/memos/memories/textual/general.py @@ -7,8 +7,8 @@ from tenacity import retry, retry_if_exception_type, stop_after_attempt from memos.configs.memory import GeneralTextMemoryConfig -from memos.embedders.factory import EmbedderFactory, OllamaEmbedder -from memos.llms.factory import LLMFactory, OllamaLLM, OpenAILLM +from memos.embedders.factory import ArkEmbedder, EmbedderFactory, OllamaEmbedder +from memos.llms.factory import AzureLLM, LLMFactory, OllamaLLM, OpenAILLM from memos.log import get_logger from memos.memories.textual.base import BaseTextMemory from memos.memories.textual.item import TextualMemoryItem @@ -26,9 +26,11 @@ class GeneralTextMemory(BaseTextMemory): def __init__(self, config: GeneralTextMemoryConfig): """Initialize memory with the given configuration.""" self.config: GeneralTextMemoryConfig = config - self.extractor_llm: OpenAILLM | OllamaLLM = LLMFactory.from_config(config.extractor_llm) + self.extractor_llm: OpenAILLM | OllamaLLM | AzureLLM = LLMFactory.from_config( + config.extractor_llm + ) self.vector_db: QdrantVecDB = VecDBFactory.from_config(config.vector_db) - self.embedder: OllamaEmbedder = EmbedderFactory.from_config(config.embedder) + self.embedder: OllamaEmbedder | ArkEmbedder = EmbedderFactory.from_config(config.embedder) @retry( stop=stop_after_attempt(3), @@ -202,7 +204,7 @@ def drop( def _embed_one_sentence(self, sentence: str) -> list[float]: """Embed a single sentence.""" - return self.embedder.embed(sentence)[0] + return self.embedder.embed([sentence])[0] EXTRACTION_PROMPT_PART_1 = f"""You are a memory extractor. Your task is to extract memories from the given messages. diff --git a/src/memos/memories/textual/tree.py b/src/memos/memories/textual/tree.py index 112d7c161..3b50001d8 100644 --- a/src/memos/memories/textual/tree.py +++ b/src/memos/memories/textual/tree.py @@ -10,7 +10,7 @@ from memos.configs.memory import TreeTextMemoryConfig from memos.embedders.factory import EmbedderFactory, OllamaEmbedder from memos.graph_dbs.factory import GraphStoreFactory, Neo4jGraphDB -from memos.llms.factory import LLMFactory, OllamaLLM, OpenAILLM +from memos.llms.factory import AzureLLM, LLMFactory, OllamaLLM, OpenAILLM from memos.log import get_logger from memos.memories.textual.base import BaseTextMemory from memos.memories.textual.item import TextualMemoryItem, TreeNodeTextualMemoryMetadata @@ -31,8 +31,12 @@ class TreeTextMemory(BaseTextMemory): def __init__(self, config: TreeTextMemoryConfig): """Initialize memory with the given configuration.""" self.config: TreeTextMemoryConfig = config - self.extractor_llm: OpenAILLM | OllamaLLM = LLMFactory.from_config(config.extractor_llm) - self.dispatcher_llm: OpenAILLM | OllamaLLM = LLMFactory.from_config(config.dispatcher_llm) + self.extractor_llm: OpenAILLM | OllamaLLM | AzureLLM = LLMFactory.from_config( + config.extractor_llm + ) + self.dispatcher_llm: OpenAILLM | OllamaLLM | AzureLLM = LLMFactory.from_config( + config.dispatcher_llm + ) self.embedder: OllamaEmbedder = EmbedderFactory.from_config(config.embedder) self.graph_store: Neo4jGraphDB = GraphStoreFactory.from_config(config.graph_db) self.is_reorganize = config.reorganize @@ -53,7 +57,7 @@ def __init__(self, config: TreeTextMemoryConfig): else: logger.info("No internet retriever configured") - def add(self, memories: list[TextualMemoryItem | dict[str, Any]]) -> None: + def add(self, memories: list[TextualMemoryItem | dict[str, Any]]) -> list[str]: """Add memories. Args: memories: List of TextualMemoryItem objects or dictionaries to add. @@ -63,7 +67,7 @@ def add(self, memories: list[TextualMemoryItem | dict[str, Any]]) -> None: plan = plan_memory_operations(memory_items, metadata, self.graph_store) execute_plan(memory_items, metadata, plan, self.graph_store) """ - self.memory_manager.add(memories) + return self.memory_manager.add(memories) def replace_working_memory(self, memories: list[TextualMemoryItem]) -> None: self.memory_manager.replace_working_memory(memories) diff --git a/src/memos/memories/textual/tree_text_memory/organize/conflict.py b/src/memos/memories/textual/tree_text_memory/organize/conflict.py index 0ecf1847e..48620f73c 100644 --- a/src/memos/memories/textual/tree_text_memory/organize/conflict.py +++ b/src/memos/memories/textual/tree_text_memory/organize/conflict.py @@ -167,10 +167,12 @@ def _resolve_in_graph( if not self.graph_store.edge_exists(new_from, new_to, edge["type"], direction="ANY"): self.graph_store.add_edge(new_from, new_to, edge["type"]) - self.graph_store.delete_node(conflict_a.id) - self.graph_store.delete_node(conflict_b.id) + self.graph_store.update_node(conflict_a.id, {"status": "archived"}) + self.graph_store.update_node(conflict_b.id, {"status": "archived"}) + self.graph_store.add_edge(conflict_a.id, merged.id, type="MERGED_TO") + self.graph_store.add_edge(conflict_b.id, merged.id, type="MERGED_TO") logger.debug( - f"Remove {conflict_a.id} and {conflict_b.id}, and inherit their edges to {merged.id}." + f"Archive {conflict_a.id} and {conflict_b.id}, and inherit their edges to {merged.id}." ) def _merge_metadata( diff --git a/src/memos/memories/textual/tree_text_memory/organize/manager.py b/src/memos/memories/textual/tree_text_memory/organize/manager.py index 471bd3659..85f000e61 100644 --- a/src/memos/memories/textual/tree_text_memory/organize/manager.py +++ b/src/memos/memories/textual/tree_text_memory/organize/manager.py @@ -5,7 +5,7 @@ from memos.embedders.factory import OllamaEmbedder from memos.graph_dbs.neo4j import Neo4jGraphDB -from memos.llms.factory import OllamaLLM, OpenAILLM +from memos.llms.factory import AzureLLM, OllamaLLM, OpenAILLM from memos.log import get_logger from memos.memories.textual.item import TextualMemoryItem, TreeNodeTextualMemoryMetadata from memos.memories.textual.tree_text_memory.organize.reorganizer import ( @@ -22,7 +22,7 @@ def __init__( self, graph_store: Neo4jGraphDB, embedder: OllamaEmbedder, - llm: OpenAILLM | OllamaLLM, + llm: OpenAILLM | OllamaLLM | AzureLLM, memory_size: dict | None = None, threshold: float | None = 0.80, merged_threshold: float | None = 0.92, @@ -49,15 +49,18 @@ def __init__( ) self._merged_threshold = merged_threshold - def add(self, memories: list[TextualMemoryItem]) -> None: + def add(self, memories: list[TextualMemoryItem]) -> list[str]: """ Add new memories in parallel to different memory types (WorkingMemory, LongTermMemory, UserMemory). """ + added_ids: list[str] = [] + with ThreadPoolExecutor(max_workers=8) as executor: - futures = [executor.submit(self._process_memory, memory) for memory in memories] + futures = {executor.submit(self._process_memory, m): m for m in memories} for future in as_completed(futures): try: - future.result() + ids = future.result() + added_ids.extend(ids) except Exception as e: logger.exception("Memory processing error: ", exc_info=e) @@ -72,6 +75,7 @@ def add(self, memories: list[TextualMemoryItem]) -> None: ) self._refresh_memory_size() + return added_ids def replace_working_memory(self, memories: list[TextualMemoryItem]) -> None: """ @@ -113,17 +117,23 @@ def _process_memory(self, memory: TextualMemoryItem): Process and add memory to different memory types (WorkingMemory, LongTermMemory, UserMemory). This method runs asynchronously to process each memory item. """ + ids = [] + # Add to WorkingMemory - self._add_memory_to_db(memory, "WorkingMemory") + working_id = self._add_memory_to_db(memory, "WorkingMemory") + ids.append(working_id) # Add to LongTermMemory and UserMemory if memory.metadata.memory_type in ["LongTermMemory", "UserMemory"]: - self._add_to_graph_memory( + added_id = self._add_to_graph_memory( memory=memory, memory_type=memory.metadata.memory_type, ) + ids.append(added_id) - def _add_memory_to_db(self, memory: TextualMemoryItem, memory_type: str): + return ids + + def _add_memory_to_db(self, memory: TextualMemoryItem, memory_type: str) -> str: """ Add a single memory item to the graph store, with FIFO logic for WorkingMemory. """ @@ -135,6 +145,7 @@ def _add_memory_to_db(self, memory: TextualMemoryItem, memory_type: str): # Insert node into graph self.graph_store.add_node(working_memory.id, working_memory.memory, metadata) + return working_memory.id def _add_to_graph_memory(self, memory: TextualMemoryItem, memory_type: str): """ @@ -159,7 +170,7 @@ def _add_to_graph_memory(self, memory: TextualMemoryItem, memory_type: str): ) if similar_nodes and similar_nodes[0]["score"] > self._merged_threshold: - self._merge(memory, similar_nodes) + return self._merge(memory, similar_nodes) else: node_id = str(uuid.uuid4()) # Step 2: Add new node to graph @@ -172,8 +183,9 @@ def _add_to_graph_memory(self, memory: TextualMemoryItem, memory_type: str): after_node=[node_id], ) ) + return node_id - def _merge(self, source_node: TextualMemoryItem, similar_nodes: list[dict]) -> None: + def _merge(self, source_node: TextualMemoryItem, similar_nodes: list[dict]) -> str: """ TODO: Add node traceability support by optionally preserving source nodes and linking them with MERGED_FROM edges. @@ -200,7 +212,9 @@ def _merge(self, source_node: TextualMemoryItem, similar_nodes: list[dict]) -> N merged_background = f"{original_meta.background}\n⟵MERGED⟶\n{source_meta.background}" merged_embedding = self.embedder.embed([merged_text])[0] - merged_confidence = float((original_meta.confidence + source_meta.confidence) / 2) + original_conf = original_meta.confidence or 0.0 + source_conf = source_meta.confidence or 0.0 + merged_confidence = float((original_conf + source_conf) / 2) merged_usage = list(set((original_meta.usage or []) + (source_meta.usage or []))) # Create new merged node @@ -232,13 +246,6 @@ def _merge(self, source_node: TextualMemoryItem, similar_nodes: list[dict]) -> N # After creating merged node and tracing lineage self._inherit_edges(original_id, merged_id) - # Relate other similar nodes to merged if needed - for related_node in similar_nodes[1:]: - if not self.graph_store.edge_exists( - merged_id, related_node["id"], type="ANY", direction="ANY" - ): - self.graph_store.add_edge(merged_id, related_node["id"], type="RELATE") - # log to reorganizer before updating the graph self.reorganizer.add_message( QueueMessage( @@ -250,6 +257,7 @@ def _merge(self, source_node: TextualMemoryItem, similar_nodes: list[dict]) -> N after_node=[merged_id], ) ) + return merged_id def _inherit_edges(self, from_id: str, to_id: str) -> None: """ diff --git a/src/memos/memories/textual/tree_text_memory/organize/redundancy.py b/src/memos/memories/textual/tree_text_memory/organize/redundancy.py index 7b56a8107..7beae7347 100644 --- a/src/memos/memories/textual/tree_text_memory/organize/redundancy.py +++ b/src/memos/memories/textual/tree_text_memory/organize/redundancy.py @@ -30,7 +30,7 @@ def detect( self, memory: TextualMemoryItem, top_k: int = 5, scope: str | None = None ) -> list[tuple[TextualMemoryItem, TextualMemoryItem]]: """ - Detect redundancy by finding the most similar items in the graph database based on embedding, then use LLM to judge conflict. + Detect redundancy by finding the most similar items in the graph database based on embedding, then use LLM to judge redundancy. Args: memory: The memory item (should have an embedding attribute or field). top_k: Number of top similar nodes to retrieve. @@ -49,7 +49,7 @@ def detect( for info in embedding_candidates_info if info["score"] >= self.EMBEDDING_THRESHOLD and info["id"] != memory.id ] - # 3. Judge conflicts using LLM + # 3. Judge redundancys using LLM embedding_candidates = self.graph_store.get_nodes(embedding_candidates_ids) redundant_pairs = [] for embedding_candidate in embedding_candidates: @@ -57,7 +57,7 @@ def detect( prompt = [ { "role": "system", - "content": "You are a conflict detector for memory items.", + "content": "You are a redundancy detector for memory items.", }, { "role": "user", @@ -71,12 +71,12 @@ def detect( if "yes" in result.lower(): redundant_pairs.append([memory, embedding_candidate]) if len(redundant_pairs): - conflict_text = "\n".join( + redundant_text = "\n".join( f'"{pair[0].memory!s}" <==REDUNDANCY==> "{pair[1].memory!s}"' for pair in redundant_pairs ) logger.warning( - f"Detected {len(redundant_pairs)} redundancies for memory {memory.id}\n {conflict_text}" + f"Detected {len(redundant_pairs)} redundancies for memory {memory.id}\n {redundant_text}" ) return redundant_pairs @@ -84,12 +84,12 @@ def resolve_two_nodes(self, memory_a: TextualMemoryItem, memory_b: TextualMemory """ Resolve detected redundancies between two memory items using LLM fusion. Args: - memory_a: The first conflicting memory item. - memory_b: The second conflicting memory item. + memory_a: The first redundant memory item. + memory_b: The second redundant memory item. Returns: A fused TextualMemoryItem representing the resolved memory. """ - + return # waiting for implementation # ———————————— 1. LLM generate fused memory ———————————— metadata_for_resolve = ["key", "background", "confidence", "updated_at"] metadata_1 = memory_a.metadata.model_dump_json(include=metadata_for_resolve) @@ -115,18 +115,10 @@ def resolve_two_nodes(self, memory_a: TextualMemoryItem, memory_b: TextualMemory try: answer = re.search(r"(.*?)", response, re.DOTALL) answer = answer.group(1).strip() - # —————— 2.1 Can't resolve conflict, hard update by comparing timestamp ———— - if len(answer) <= 10 and "no" in answer.lower(): - logger.warning( - f"Conflict between {memory_a.id} and {memory_b.id} could not be resolved. " - ) - self._hard_update(memory_a, memory_b) - # —————— 2.2 Conflict resolved, update metadata and memory ———— - else: - fixed_metadata = self._merge_metadata(answer, memory_a.metadata, memory_b.metadata) - merged_memory = TextualMemoryItem(memory=answer, metadata=fixed_metadata) - logger.info(f"Resolved result: {merged_memory}") - self._resolve_in_graph(memory_a, memory_b, merged_memory) + fixed_metadata = self._merge_metadata(answer, memory_a.metadata, memory_b.metadata) + merged_memory = TextualMemoryItem(memory=answer, metadata=fixed_metadata) + logger.info(f"Resolved result: {merged_memory}") + self._resolve_in_graph(memory_a, memory_b, merged_memory) except json.decoder.JSONDecodeError: logger.error(f"Failed to parse LLM response: {response}") @@ -145,29 +137,14 @@ def resolve_one_node(self, memory: TextualMemoryItem) -> None: ) logger.debug(f"Merged memory: {memory.memory}") - def _hard_update(self, memory_a: TextualMemoryItem, memory_b: TextualMemoryItem): - """ - Hard update: compare updated_at, keep the newer one, overwrite the older one's metadata. - """ - time_a = datetime.fromisoformat(memory_a.metadata.updated_at) - time_b = datetime.fromisoformat(memory_b.metadata.updated_at) - - newer_mem = memory_a if time_a >= time_b else memory_b - older_mem = memory_b if time_a >= time_b else memory_a - - self.graph_store.delete_node(older_mem.id) - logger.warning( - f"Delete older memory {older_mem.id}: <{older_mem.memory}> due to conflict with {newer_mem.id}: <{newer_mem.memory}>" - ) - def _resolve_in_graph( self, - conflict_a: TextualMemoryItem, - conflict_b: TextualMemoryItem, + redundant_a: TextualMemoryItem, + redundant_b: TextualMemoryItem, merged: TextualMemoryItem, ): - edges_a = self.graph_store.get_edges(conflict_a.id, type="ANY", direction="ANY") - edges_b = self.graph_store.get_edges(conflict_b.id, type="ANY", direction="ANY") + edges_a = self.graph_store.get_edges(redundant_a.id, type="ANY", direction="ANY") + edges_b = self.graph_store.get_edges(redundant_b.id, type="ANY", direction="ANY") all_edges = edges_a + edges_b self.graph_store.add_node( @@ -175,18 +152,22 @@ def _resolve_in_graph( ) for edge in all_edges: - new_from = merged.id if edge["from"] in (conflict_a.id, conflict_b.id) else edge["from"] - new_to = merged.id if edge["to"] in (conflict_a.id, conflict_b.id) else edge["to"] + new_from = ( + merged.id if edge["from"] in (redundant_a.id, redundant_b.id) else edge["from"] + ) + new_to = merged.id if edge["to"] in (redundant_a.id, redundant_b.id) else edge["to"] if new_from == new_to: continue # Check if the edge already exists before adding if not self.graph_store.edge_exists(new_from, new_to, edge["type"], direction="ANY"): self.graph_store.add_edge(new_from, new_to, edge["type"]) - self.graph_store.delete_node(conflict_a.id) - self.graph_store.delete_node(conflict_b.id) + self.graph_store.update_node(redundant_a.id, {"status": "archived"}) + self.graph_store.update_node(redundant_b.id, {"status": "archived"}) + self.graph_store.add_edge(redundant_a.id, merged.id, type="MERGED_TO") + self.graph_store.add_edge(redundant_b.id, merged.id, type="MERGED_TO") logger.debug( - f"Remove {conflict_a.id} and {conflict_b.id}, and inherit their edges to {merged.id}." + f"Archive {redundant_a.id} and {redundant_b.id}, and inherit their edges to {merged.id}." ) def _merge_metadata( diff --git a/src/memos/memories/textual/tree_text_memory/organize/relation_reason_detector.py b/src/memos/memories/textual/tree_text_memory/organize/relation_reason_detector.py index 803a73bcb..cc755d6dd 100644 --- a/src/memos/memories/textual/tree_text_memory/organize/relation_reason_detector.py +++ b/src/memos/memories/textual/tree_text_memory/organize/relation_reason_detector.py @@ -54,20 +54,26 @@ def process_node(self, node: GraphDBNode, exclude_ids: list[str], top_k: int = 5 ) nearest = [GraphDBNode(**cand_data) for cand_data in nearest] + """ # 1) Pairwise relations (including CAUSE/CONDITION/CONFLICT) pairwise = self._detect_pairwise_causal_condition_relations(node, nearest) results["relations"].extend(pairwise["relations"]) + """ + """ # 2) Inferred nodes (from causal/condition) inferred = self._infer_fact_nodes_from_relations(pairwise) results["inferred_nodes"].extend(inferred) + """ - # 3) Sequence (optional, if you have timestamps) + """ + 3) Sequence (optional, if you have timestamps) seq = self._detect_sequence_links(node, nearest) results["sequence_links"].extend(seq) + """ # 4) Aggregate - agg = self._detect_aggregate_node_for_group(node, nearest, min_group_size=3) + agg = self._detect_aggregate_node_for_group(node, nearest, min_group_size=5) if agg: results["aggregate_nodes"].append(agg) @@ -80,7 +86,7 @@ def _detect_pairwise_causal_condition_relations( Vector/tag search ➜ For each candidate, use LLM to decide: - CAUSE - CONDITION - - RELATE_TO + - RELATE - CONFLICT """ results = {"relations": []} @@ -168,7 +174,7 @@ def _detect_aggregate_node_for_group( combined_nodes = [node, *nearest_nodes] joined = "\n".join(f"- {n.memory}" for n in combined_nodes) - prompt = AGGREGATE_PROMPT.format(joined=joined) + prompt = AGGREGATE_PROMPT.replace("{joined}", joined) response_text = self._call_llm(prompt) response_json = self._parse_json_result(response_text) if not response_json: @@ -205,14 +211,6 @@ def _call_llm(self, prompt: str) -> str: logger.warning(f"[LLM Error] {e}") return "" - def _parse_relation_result(self, response_text: str) -> str: - relation = response_text.strip().upper() - valid = {"CAUSE", "CONDITION", "RELATE_TO", "CONFLICT", "NONE"} - if relation not in valid: - logger.warning(f"[RelationDetector] Unexpected relation: {relation}. Fallback NONE.") - return "NONE" - return relation - def _parse_json_result(self, response_text): try: response_text = response_text.replace("```", "").replace("json", "") @@ -226,7 +224,7 @@ def _parse_relation_result(self, response_text: str) -> str: Normalize and validate the LLM relation type output. """ relation = response_text.strip().upper() - valid = {"CAUSE", "CONDITION", "RELATE_TO", "CONFLICT", "NONE"} + valid = {"CAUSE", "CONDITION", "RELATE", "CONFLICT", "NONE"} if relation not in valid: logger.warning( f"[RelationDetector] Unexpected relation type: {relation}. Fallback to NONE." diff --git a/src/memos/memories/textual/tree_text_memory/organize/reorganizer.py b/src/memos/memories/textual/tree_text_memory/organize/reorganizer.py index 3bd81da49..43b0217be 100644 --- a/src/memos/memories/textual/tree_text_memory/organize/reorganizer.py +++ b/src/memos/memories/textual/tree_text_memory/organize/reorganizer.py @@ -3,15 +3,14 @@ import time import traceback +from collections import Counter, defaultdict from concurrent.futures import ThreadPoolExecutor, as_completed from queue import PriorityQueue from typing import Literal import numpy as np -import schedule - -from sklearn.cluster import MiniBatchKMeans +from memos.dependency import require_python_package from memos.embedders.factory import OllamaEmbedder from memos.graph_dbs.item import GraphDBEdge, GraphDBNode from memos.graph_dbs.neo4j import Neo4jGraphDB @@ -32,7 +31,7 @@ class QueueMessage: def __init__( self, - op: Literal["add", "remove", "merge", "update"], + op: Literal["add", "remove", "merge", "update", "end"], # `str` for node and edge IDs, `GraphDBNode` and `GraphDBEdge` for actual objects before_node: list[str] | list[GraphDBNode] | None = None, before_edge: list[str] | list[GraphDBEdge] | None = None, @@ -49,7 +48,7 @@ def __str__(self) -> str: return f"QueueMessage(op={self.op}, before_node={self.before_node if self.before_node is None else len(self.before_node)}, after_node={self.after_node if self.after_node is None else len(self.after_node)})" def __lt__(self, other: "QueueMessage") -> bool: - op_priority = {"add": 2, "remove": 2, "merge": 1} + op_priority = {"add": 2, "remove": 2, "merge": 1, "end": 0} return op_priority[self.op] < op_priority[other.op] @@ -104,7 +103,7 @@ def wait_until_current_task_done(self): def _run_message_consumer_loop(self): while True: message = self.queue.get() - if message is None: + if message.op == "end": break try: @@ -114,11 +113,18 @@ def _run_message_consumer_loop(self): logger.error(traceback.format_exc()) self.queue.task_done() + @require_python_package( + import_name="schedule", + install_command="pip install schedule", + install_link="https://schedule.readthedocs.io/en/stable/installation.html", + ) def _run_structure_organizer_loop(self): """ Use schedule library to periodically trigger structure optimization. This runs until the stop flag is set. """ + import schedule + schedule.every(20).seconds.do(self.optimize_structure, scope="LongTermMemory") schedule.every(20).seconds.do(self.optimize_structure, scope="UserMemory") @@ -134,7 +140,7 @@ def stop(self): if not self.is_reorganize: return - self.add_message(None) + self.add_message(QueueMessage(op="end")) self.thread.join() logger.info("Reorganize thread stopped.") self._stop_scheduler = True @@ -152,9 +158,6 @@ def handle_message(self, message: QueueMessage): def handle_add(self, message: QueueMessage): logger.debug(f"Handling add operation: {str(message)[:500]}") - assert message.before_node is None and message.before_edge is None, ( - "Before node and edge should be None for `add` operation." - ) # ———————— 1. check for conflicts ———————— added_node = message.after_node[0] conflicts = self.conflict.detect(added_node, scope=added_node.metadata.memory_type) @@ -164,9 +167,9 @@ def handle_add(self, message: QueueMessage): logger.info(f"Resolved conflict between {added_node.id} and {existing_node.id}.") # ———————— 2. check for redundancy ———————— - redundancy = self.redundancy.detect(added_node, scope=added_node.metadata.memory_type) - if redundancy: - for added_node, existing_node in redundancy: + redundancies = self.redundancy.detect(added_node, scope=added_node.metadata.memory_type) + if redundancies: + for added_node, existing_node in redundancies: self.redundancy.resolve_two_nodes(added_node, existing_node) logger.info(f"Resolved redundancy between {added_node.id} and {existing_node.id}.") @@ -176,14 +179,14 @@ def handle_remove(self, message: QueueMessage): def handle_merge(self, message: QueueMessage): after_node = message.after_node[0] logger.debug(f"Handling merge operation: <{after_node.memory}>") - self.redundancy_resolver.resolve_one_node(after_node) + self.redundancy.resolve_one_node(after_node) def optimize_structure( self, scope: str = "LongTermMemory", local_tree_threshold: int = 10, min_cluster_size: int = 3, - min_group_size: int = 10, + min_group_size: int = 5, ): """ Periodically reorganize the graph: @@ -358,7 +361,7 @@ def _local_subcluster(self, cluster_nodes: list[GraphDBNode]) -> list[list[Graph scene_lines.append(line) joined_scene = "\n".join(scene_lines) - prompt = LOCAL_SUBCLUSTER_PROMPT.format(joined_scene=joined_scene) + prompt = LOCAL_SUBCLUSTER_PROMPT.replace("{joined_scene}", joined_scene) messages = [{"role": "user", "content": prompt}] response_text = self.llm.generate(messages) @@ -378,9 +381,12 @@ def _local_subcluster(self, cluster_nodes: list[GraphDBNode]) -> list[list[Graph return result_subclusters - def _partition( - self, nodes: list[GraphDBNode], min_cluster_size: int = 3 - ) -> list[list[GraphDBNode]]: + @require_python_package( + import_name="sklearn", + install_command="pip install scikit-learn", + install_link="https://scikit-learn.org/stable/install.html", + ) + def _partition(self, nodes, min_cluster_size: int = 3, max_cluster_size: int = 20): """ Partition nodes by: 1) Frequent tags (top N & above threshold) @@ -394,7 +400,7 @@ def _partition( Returns: List of clusters, each as a list of GraphDBNode """ - from collections import Counter, defaultdict + from sklearn.cluster import MiniBatchKMeans # 1) Count all tags tag_counter = Counter() @@ -407,7 +413,7 @@ def _partition( threshold_tags = {tag for tag, count in tag_counter.items() if count >= 50} frequent_tags = top_n_tags | threshold_tags - # Group nodes by tags, ensure each group is unique internally + # Group nodes by tags tag_groups = defaultdict(list) for node in nodes: @@ -420,48 +426,67 @@ def _partition( assigned_ids = set() for tag, group in tag_groups.items(): if len(group) >= min_cluster_size: - filtered_tag_clusters.append(group) - assigned_ids.update(n.id for n in group) + # Split large groups into chunks of at most max_cluster_size + for i in range(0, len(group), max_cluster_size): + sub_group = group[i : i + max_cluster_size] + filtered_tag_clusters.append(sub_group) + assigned_ids.update(n.id for n in sub_group) else: - logger.info(f"... dropped {tag} ...") + logger.info(f"... dropped tag {tag} due to low size ...") logger.info( f"[MixedPartition] Created {len(filtered_tag_clusters)} clusters from tags. " f"Nodes grouped by tags: {len(assigned_ids)} / {len(nodes)}" ) - # 5) Remaining nodes -> embedding clustering + # Remaining nodes -> embedding clustering remaining_nodes = [n for n in nodes if n.id not in assigned_ids] logger.info( f"[MixedPartition] Remaining nodes for embedding clustering: {len(remaining_nodes)}" ) embedding_clusters = [] - if remaining_nodes: - x = np.array([n.metadata.embedding for n in remaining_nodes if n.metadata.embedding]) - k = max(1, min(len(remaining_nodes) // min_cluster_size, 20)) - if len(x) < k: - k = len(x) - if 1 < k <= len(x): + def recursive_clustering(nodes_list): + """Recursively split clusters until each is <= max_cluster_size.""" + if len(nodes_list) <= max_cluster_size: + return [nodes_list] + + # Try kmeans with k = ceil(len(nodes) / max_cluster_size) + x = np.array([n.metadata.embedding for n in nodes_list if n.metadata.embedding]) + if len(x) < 2: + return [nodes_list] + + k = min(len(x), (len(nodes_list) + max_cluster_size - 1) // max_cluster_size) + k = max(1, min(k, len(x))) + + try: kmeans = MiniBatchKMeans(n_clusters=k, batch_size=256, random_state=42) labels = kmeans.fit_predict(x) label_groups = defaultdict(list) - for node, label in zip(remaining_nodes, labels, strict=False): + for node, label in zip(nodes_list, labels, strict=False): label_groups[label].append(node) - embedding_clusters = list(label_groups.values()) - logger.info( - f"[MixedPartition] Created {len(embedding_clusters)} clusters from embedding." - ) - else: - embedding_clusters = [remaining_nodes] + result = [] + for sub_group in label_groups.values(): + result.extend(recursive_clustering(sub_group)) + return result + except Exception as e: + logger.warning(f"Clustering failed: {e}, falling back to single cluster.") + return [nodes_list] + + if remaining_nodes: + clusters = recursive_clustering(remaining_nodes) + embedding_clusters.extend(clusters) + logger.info( + f"[MixedPartition] Created {len(embedding_clusters)} clusters from embeddings." + ) - # Merge all & handle small clusters + # Merge all clusters all_clusters = filtered_tag_clusters + embedding_clusters - # Optional: merge tiny clusters + # Handle small clusters (< min_cluster_size) final_clusters = [] small_nodes = [] for group in all_clusters: @@ -484,18 +509,15 @@ def _summarize_cluster(self, cluster_nodes: list[GraphDBNode], scope: str) -> Gr if not cluster_nodes: raise ValueError("Cluster nodes cannot be empty.") - joined_keys = "\n".join(f"- {n.metadata.key}" for n in cluster_nodes if n.metadata.key) - joined_values = "\n".join(f"- {n.memory}" for n in cluster_nodes) - joined_backgrounds = "\n".join( - f"- {n.metadata.background}" for n in cluster_nodes if n.metadata.background + memories_items_text = "\n\n".join( + [ + f"{i}. key: {n.metadata.key}\nvalue: {n.memory}\nsummary:{n.metadata.background}" + for i, n in enumerate(cluster_nodes) + ] ) # Build prompt - prompt = REORGANIZE_PROMPT.format( - joined_keys=joined_keys, - joined_values=joined_values, - joined_backgrounds=joined_backgrounds, - ) + prompt = REORGANIZE_PROMPT.replace("{memory_items_text}", memories_items_text) messages = [{"role": "user", "content": prompt}] response_text = self.llm.generate(messages) @@ -505,7 +527,7 @@ def _summarize_cluster(self, cluster_nodes: list[GraphDBNode], scope: str) -> Gr parent_key = response_json.get("key", "").strip() parent_value = response_json.get("value", "").strip() parent_tags = response_json.get("tags", []) - parent_background = response_json.get("background", "").strip() + parent_background = response_json.get("summary", "").strip() embedding = self.embedder.embed([parent_value])[0] @@ -561,7 +583,7 @@ def _link_cluster_nodes(self, parent_node: GraphDBNode, child_nodes: list[GraphD def _preprocess_message(self, message: QueueMessage) -> bool: message = self._convert_id_to_node(message) - if None in message.after_node: + if message.after_node is None or None in message.after_node: logger.debug( f"Found non-existent node in after_node in message: {message}, skip this message." ) diff --git a/src/memos/memories/textual/tree_text_memory/retrieve/recall.py b/src/memos/memories/textual/tree_text_memory/retrieve/recall.py index 8a3fc4b5c..36a0b5fee 100644 --- a/src/memos/memories/textual/tree_text_memory/retrieve/recall.py +++ b/src/memos/memories/textual/tree_text_memory/retrieve/recall.py @@ -56,7 +56,6 @@ def retrieve( # Step 3: Merge and deduplicate results combined = {item.id: item for item in graph_results + vector_results} - # Debug: 打印在 graph_results 中但不在 combined 中的 id graph_ids = {item.id for item in graph_results} combined_ids = set(combined.keys()) lost_ids = graph_ids - combined_ids diff --git a/src/memos/memories/textual/tree_text_memory/retrieve/reranker.py b/src/memos/memories/textual/tree_text_memory/retrieve/reranker.py index fedd2e7fe..2a09ff752 100644 --- a/src/memos/memories/textual/tree_text_memory/retrieve/reranker.py +++ b/src/memos/memories/textual/tree_text_memory/retrieve/reranker.py @@ -1,7 +1,7 @@ import numpy as np from memos.embedders.factory import OllamaEmbedder -from memos.llms.factory import OllamaLLM, OpenAILLM +from memos.llms.factory import AzureLLM, OllamaLLM, OpenAILLM from memos.memories.textual.item import TextualMemoryItem from memos.memories.textual.tree_text_memory.retrieve.retrieval_mid_structs import ParsedTaskGoal @@ -41,7 +41,7 @@ class MemoryReranker: Rank retrieved memory cards by structural priority and contextual similarity. """ - def __init__(self, llm: OpenAILLM | OllamaLLM, embedder: OllamaEmbedder): + def __init__(self, llm: OpenAILLM | OllamaLLM | AzureLLM, embedder: OllamaEmbedder): self.llm = llm self.embedder = embedder diff --git a/src/memos/memories/textual/tree_text_memory/retrieve/searcher.py b/src/memos/memories/textual/tree_text_memory/retrieve/searcher.py index 9cb434b5c..40bd01a4d 100644 --- a/src/memos/memories/textual/tree_text_memory/retrieve/searcher.py +++ b/src/memos/memories/textual/tree_text_memory/retrieve/searcher.py @@ -5,7 +5,7 @@ from memos.embedders.factory import OllamaEmbedder from memos.graph_dbs.factory import Neo4jGraphDB -from memos.llms.factory import OllamaLLM, OpenAILLM +from memos.llms.factory import AzureLLM, OllamaLLM, OpenAILLM from memos.memories.textual.item import SearchedTreeNodeTextualMemoryMetadata, TextualMemoryItem from .internet_retriever_factory import InternetRetrieverFactory @@ -18,7 +18,7 @@ class Searcher: def __init__( self, - dispatcher_llm: OpenAILLM | OllamaLLM, + dispatcher_llm: OpenAILLM | OllamaLLM | AzureLLM, graph_store: Neo4jGraphDB, embedder: OllamaEmbedder, internet_retriever: InternetRetrieverFactory | None = None, @@ -176,9 +176,10 @@ def retrieve_from_internet(): for item, score in sorted(deduped_result.values(), key=lambda pair: pair[1], reverse=True)[ :top_k ]: - new_meta = SearchedTreeNodeTextualMemoryMetadata( - **item.metadata.model_dump(), relativity=score - ) + meta_data = item.metadata.model_dump() + if "relativity" not in meta_data: + meta_data["relativity"] = score + new_meta = SearchedTreeNodeTextualMemoryMetadata(**meta_data) searched_res.append( TextualMemoryItem(id=item.id, memory=item.memory, metadata=new_meta) ) diff --git a/src/memos/parsers/markitdown.py b/src/memos/parsers/markitdown.py index c466fa902..02e75355f 100644 --- a/src/memos/parsers/markitdown.py +++ b/src/memos/parsers/markitdown.py @@ -1,6 +1,5 @@ -from markitdown import MarkItDown - from memos.configs.parser import MarkItDownParserConfig +from memos.dependency import require_python_package from memos.log import get_logger from memos.parsers.base import BaseParser @@ -14,7 +13,14 @@ class MarkItDownParser(BaseParser): def __init__(self, config: MarkItDownParserConfig): self.config = config + @require_python_package( + import_name="markitdown", + install_command="pip install markitdown[all]", + install_link="https://github.com/microsoft/markitdown", + ) def parse(self, file_path: str) -> str: + from markitdown import MarkItDown + """Parse the file at the given path and return its content as a MarkDown string.""" md = MarkItDown(enable_plugins=False) result = md.convert(file_path) diff --git a/src/memos/templates/mem_reader_prompts.py b/src/memos/templates/mem_reader_prompts.py index 8e6bf3c46..baa49a7e6 100644 --- a/src/memos/templates/mem_reader_prompts.py +++ b/src/memos/templates/mem_reader_prompts.py @@ -1,7 +1,5 @@ SIMPLE_STRUCT_MEM_READER_PROMPT = """You are a memory extraction expert. - Your task is to extract memories from the perspective of user, based on a conversation between user and assistant. This means identifying what user would plausibly remember — including their own experiences, thoughts, plans, or relevant statements and actions made by others (such as assistant) that impacted or were acknowledged by user. - Please perform: 1. Identify information that reflects user's experiences, beliefs, concerns, decisions, plans, or reactions — including meaningful input from assistant that user acknowledged or responded to. 2. Resolve all time, person, and event references clearly: @@ -35,7 +33,7 @@ } Language rules: -- The `key`, `value`, `tags`, `summary` fields must match the language of the input conversation. +- The `key`, `value`, `tags`, `summary` fields must match the mostly used language of the input conversation. **如果输入是中文，请输出中文** - Keep `memory_type` in English. Example: @@ -66,33 +64,63 @@ "summary": "Tom is currently focused on managing a new project with a tight schedule. After a team meeting on June 25, 2025, he realized the original deadline of December 15 might not be feasible due to backend delays. Concerned about insufficient testing time, he welcomed Jerry’s suggestion of proposing an extension. Tom plans to raise the idea of shifting the deadline to January 5, 2026 in the next morning’s meeting. His actions reflect both stress about timelines and a proactive, team-oriented problem-solving approach." } +Another Example in Chinese (注意: 当user的语言为中文时，你就需要也输出中文)： +{ + "memory list": [ + { + "key": "项目会议", + "memory_type": "LongTermMemory", + "value": "在2025年6月25日下午3点，Tom与团队开会讨论了新项目，涉及时间表，并提出了对12月15日截止日期可行性的担忧。", + "tags": ["项目", "时间表", "会议", "截止日期"] + }, + ... + ], + "summary": "Tom 目前专注于管理一个进度紧张的新项目..." +} + +Always respond in the same language as the conversation. + Conversation: ${conversation} Your Output:""" -SIMPLE_STRUCT_DOC_READER_PROMPT = """ -You are an expert text analyst for a search and retrieval system. Your task is to process a document chunk and generate a single, structured JSON object. -The input is a single piece of text: `[DOCUMENT_CHUNK]`. -You must generate a single JSON object with two top-level keys: `summary` and `tags`. -1. `summary`: - - A dense, searchable summary of the ENTIRE `[DOCUMENT_CHUNK]`. - - The purpose is for semantic search embedding. - - A clear and accurate sentence that comprehensively summarizes the main points, arguments, and information within the `[DOCUMENT_CHUNK]`. - - The goal is to create a standalone overview that allows a reader to fully understand the essence of the chunk without reading the original text. - - The summary should be **no more than 50 words**. -2. `tags`: - - A concise list of **3 to 5 high-level, summative tags**. - - **Each tag itself should be a short phrase, ideally 2 to 4 words long.** - - These tags must represent the core abstract themes of the text, suitable for broad categorization. - - **Crucially, prioritize abstract concepts** over specific entities or phrases mentioned in the text. For example, prefer "Supply Chain Resilience" over "Reshoring Strategies". - -Here is the document chunk to process: -`[DOCUMENT_CHUNK]` +SIMPLE_STRUCT_DOC_READER_PROMPT = """You are an expert text analyst for a search and retrieval system. +Your task is to process a document chunk and generate a single, structured JSON object. + +Please perform: +1. Identify key information that reflects factual content, insights, decisions, or implications from the documents — including any notable themes, conclusions, or data points. Allow a reader to fully understand the essence of the chunk without reading the original text. +2. Resolve all time, person, location, and event references clearly: + - Convert relative time expressions (e.g., “last year,” “next quarter”) into absolute dates if context allows. + - Clearly distinguish between event time and document time. + - If uncertainty exists, state it explicitly (e.g., “around 2024,” “exact date unclear”). + - Include specific locations if mentioned. + - Resolve all pronouns, aliases, and ambiguous references into full names or identities. + - Disambiguate entities with the same name if applicable. +3. Always write from a third-person perspective, referring to the subject or content clearly rather than using first-person ("I", "me", "my"). +4. Do not omit any information that is likely to be important or memorable from the document summaries. + - Include all key facts, insights, emotional tones, and plans — even if they seem minor. + - Prioritize completeness and fidelity over conciseness. + - Do not generalize or skip details that could be contextually meaningful. + +Return a single valid JSON object with the following structure: + +Return valid JSON: +{ + "key": , + "memory_type": "LongTermMemory", + "value": , + "tags": +} + +Language rules: +- The `key`, `value`, `tags`, `summary` fields must match the mostly used language of the input document summaries. **如果输入是中文，请输出中文** +- Keep `memory_type` in English. + +Document chunk: {chunk_text} -Produce ONLY the JSON object as your response. -""" +Your Output:""" SIMPLE_STRUCT_MEM_READER_EXAMPLE = """Example: Conversation: @@ -122,4 +150,18 @@ "summary": "Tom is currently focused on managing a new project with a tight schedule. After a team meeting on June 25, 2025, he realized the original deadline of December 15 might not be feasible due to backend delays. Concerned about insufficient testing time, he welcomed Jerry’s suggestion of proposing an extension. Tom plans to raise the idea of shifting the deadline to January 5, 2026 in the next morning’s meeting. His actions reflect both stress about timelines and a proactive, team-oriented problem-solving approach." } +Another Example in Chinese (注意: 你的输出必须和输入的user语言一致)： +{ + "memory list": [ + { + "key": "项目会议", + "memory_type": "LongTermMemory", + "value": "在2025年6月25日下午3点，Tom与团队开会讨论了新项目，涉及时间表，并提出了对12月15日截止日期可行性的担忧。", + "tags": ["项目", "时间表", "会议", "截止日期"] + }, + ... + ], + "summary": "Tom 目前专注于管理一个进度紧张的新项目..." +} + """ diff --git a/src/memos/templates/mem_scheduler_prompts.py b/src/memos/templates/mem_scheduler_prompts.py index f60ab35c0..83b984bf8 100644 --- a/src/memos/templates/mem_scheduler_prompts.py +++ b/src/memos/templates/mem_scheduler_prompts.py @@ -1,65 +1,114 @@ -INTENT_RECOGNIZING_PROMPT = """You are a user intent recognizer, and your task is to determine whether the user's current question has been satisfactorily answered. - -You will receive the following information: - -The user’s current question list (q_list), arranged in chronological order (currently contains only one question); -The memory information currently present in the system’s workspace (working_memory_list), i.e., the currently known contextual clues. -Your tasks are: - -Determine whether the user is satisfied with the existing answer; - -If the user is satisfied, explain the reason and return: - -"trigger_retrieval": false -If the user is not satisfied, meaning the system's answer did not meet their actual needs, please return: - -"trigger_retrieval": true -"missing_evidence": ["Information you infer is missing and needs to be supplemented, such as specific experiences of someone, health records, etc."] -Please return strictly according to the following JSON format: - +INTENT_RECOGNIZING_PROMPT = """ +# User Intent Recognition Task + +## Role +You are an advanced intent analysis system that evaluates answer satisfaction and identifies information gaps. + +## Input Analysis +You will receive: +1. User's question list (chronological order) +2. Current system knowledge (working memory) + +## Evaluation Criteria +Consider these satisfaction factors: +1. Answer completeness (covers all aspects of the question) +2. Evidence relevance (directly supports the answer) +3. Detail specificity (contains necessary granularity) +4. Personalization (tailored to user's context) + +## Decision Framework +1. Mark as satisfied ONLY if: + - All question aspects are addressed + - Supporting evidence exists in working memory + - No apparent gaps in information + +2. Mark as unsatisfied if: + - Any question aspect remains unanswered + - Evidence is generic/non-specific + - Personal context is missing + +## Output Specification +Return JSON with: +- "trigger_retrieval": Boolean (true if more evidence needed) +- "missing_evidences": List of specific evidence types required + +## Response Format {{ - "trigger_retrieval": true or false, - "missing_evidence": ["The missing evidence needed for the next step of retrieval and completion"] + "trigger_retrieval": , + "missing_evidences": [ + "", + "" + ] }} -The user's question list is: + +## Evidence Type Examples +- Personal medical history +- Recent activity logs +- Specific measurement data +- Contextual details about [topic] +- Temporal information (when something occurred) + +## Current Task +User Questions: {q_list} -The memory information currently present in the system’s workspace is: +Working Memory Contents: {working_memory_list} -""" -MEMORY_RERANKEING_PROMPT = """You are a memory sorter. Your task is to reorder the evidence according to the user's question, placing the evidence that best supports the user's query as close to the front as possible. - -Please return the newly reordered memory sequence according to the query in the following format, which must be in JSON: +## Required Output +Please provide your analysis in the specified JSON format: +""" +MEMORY_RERANKING_PROMPT = """ +# Memory Reranking Task + +## Role +You are an intelligent memory reorganization system. Your primary function is to analyze and optimize the ordering of memory evidence based on relevance to recent user queries. + +## Task Description +Reorganize the provided memory evidence list by: +1. Analyzing the semantic relationship between each evidence item and the user's queries +2. Calculating relevance scores +3. Sorting evidence in descending order of relevance +4. Maintaining all original items (no additions or deletions) + +## Input Format +- Queries: Recent user questions/requests (list) +- Current Order: Existing memory sequence (list) + +## Output Requirements +Return a JSON object with: +- "new_order": The reordered list (maintaining all original items) +- "reasoning": Brief explanation of your ranking logic (1-2 sentences) + +## Processing Guidelines +1. Prioritize evidence that: + - Directly answers query questions + - Contains exact keyword matches + - Provides contextual support + - Shows temporal relevance (newer > older) +2. For ambiguous cases, maintain original relative ordering + +## Example +Input queries: ["python threading best practices"] +Input order: ["basic python syntax", "thread safety patterns", "data structures"] + +Output: {{ -"new_order": [...] + "new_order": ["thread safety patterns", "data structures", "basic python syntax"], + "reasoning": "Prioritized threading-related content while maintaining general python references" }} -Now the user's question is: -{query} - -The current order is: -{current_order}""" - -FREQ_DETECTING_PROMPT = """You are a memory frequency monitor. Your task is to check which memories in the activation memory list appear in the given answer, and increment their count by 1 for each occurrence. - -Please return strictly according to the following JSON format: - -[ - {{"memory": ..., "count": ...}}, {{"memory": ..., "count": ...}}, ... -] -The answer is: -{answer} +## Current Task +Queries: {queries} +Current order: {current_order} -The activation memory list is: -{activation_memory_freq_list} +Please provide your reorganization: """ PROMPT_MAPPING = { "intent_recognizing": INTENT_RECOGNIZING_PROMPT, - "memory_reranking": MEMORY_RERANKEING_PROMPT, - "freq_detecting": FREQ_DETECTING_PROMPT, + "memory_reranking": MEMORY_RERANKING_PROMPT, } MEMORY_ASSEMBLY_TEMPLATE = """The retrieved memories are listed as follows:\n\n {memory_text}""" diff --git a/src/memos/templates/tree_reorganize_prompts.py b/src/memos/templates/tree_reorganize_prompts.py index 8b62a2547..5b83db474 100644 --- a/src/memos/templates/tree_reorganize_prompts.py +++ b/src/memos/templates/tree_reorganize_prompts.py @@ -2,32 +2,80 @@ Given the following child memory items: -Keys: -{joined_keys} +{memory_items_text} + +Please perform: +1. Identify information that reflects user's experiences, beliefs, concerns, decisions, plans, or reactions — including meaningful input from assistant that user acknowledged or responded to. +2. Resolve all time, person, and event references clearly: + - Convert relative time expressions (e.g., “yesterday,” “next Friday”) into absolute dates using the message timestamp if possible. + - Clearly distinguish between event time and message time. + - If uncertainty exists, state it explicitly (e.g., “around June 2025,” “exact date unclear”). + - Include specific locations if mentioned. + - Resolve all pronouns, aliases, and ambiguous references into full names or identities. + - Disambiguate people with the same name if applicable. +3. Always write from a third-person perspective, referring to user as +"The user" or by name if name mentioned, rather than using first-person ("I", "me", "my"). +For example, write "The user felt exhausted..." instead of "I felt exhausted...". +4. Do not omit any information that user is likely to remember. + - Include all key experiences, thoughts, emotional responses, and plans — even if they seem minor. + - Prioritize completeness and fidelity over conciseness. + - Do not generalize or skip details that could be personally meaningful to user. +5. Summarize all child memory items into one memory item. + +Language rules: +- The `key`, `value`, `tags`, `summary` fields must match the mostly used language of the input memory items. **如果输入是中文，请输出中文** +- Keep `memory_type` in English. -Values: -{joined_values} +Return valid JSON: +{ + "key": , + "memory_type": , + "value": , + "tags": , + "summary": +} -Backgrounds: -{joined_backgrounds} +""" -Your task: -- Generate a single clear English `key` (5–10 words max). -- Write a detailed `value` that merges the key points into a single, complete, well-structured text. This must stand alone and convey what the user should remember. -- Provide a list of 5–10 relevant English `tags`. -- Write a short `background` note (50–100 words) covering any extra context, sources, or traceability info. +DOC_REORGANIZE_PROMPT = """You are a document summarization and knowledge extraction expert. + +Given the following summarized document items: + +{memory_items_text} + +Please perform: +1. Identify key information that reflects factual content, insights, decisions, or implications from the documents — including any notable themes, conclusions, or data points. +2. Resolve all time, person, location, and event references clearly: + - Convert relative time expressions (e.g., “last year,” “next quarter”) into absolute dates if context allows. + - Clearly distinguish between event time and document time. + - If uncertainty exists, state it explicitly (e.g., “around 2024,” “exact date unclear”). + - Include specific locations if mentioned. + - Resolve all pronouns, aliases, and ambiguous references into full names or identities. + - Disambiguate entities with the same name if applicable. +3. Always write from a third-person perspective, referring to the subject or content clearly rather than using first-person ("I", "me", "my"). +4. Do not omit any information that is likely to be important or memorable from the document summaries. + - Include all key facts, insights, emotional tones, and plans — even if they seem minor. + - Prioritize completeness and fidelity over conciseness. + - Do not generalize or skip details that could be contextually meaningful. +5. Summarize all document summaries into one integrated memory item. + +Language rules: +- The `key`, `value`, `tags`, `summary` fields must match the mostly used language of the input document summaries. **如果输入是中文，请输出中文** +- Keep `memory_type` in English. Return valid JSON: -{{ - "key": "", - "value": "", - "tags": ["tag1", "tag2", ...], - "background": "" -}} +{ + "key": , + "memory_type": "LongTermMemory", + "value": , + "tags": , + "summary": +} + """ -LOCAL_SUBCLUSTER_PROMPT = """ -You are a memory organization expert. + +LOCAL_SUBCLUSTER_PROMPT = """You are a memory organization expert. You are given a cluster of memory items, each with an ID and content. Your task is to divide these into smaller, semantically meaningful sub-clusters. @@ -36,21 +84,25 @@ - Identify natural topics by analyzing common time, place, people, and event elements. - Each sub-cluster must reflect a coherent theme that helps retrieval. - Each sub-cluster should have 2–10 items. Discard singletons. -- Each item ID must appear in exactly one sub-cluster. +- Each item ID must appear in exactly one sub-cluster or be discarded. No duplicates are allowed. +- All IDs in the output must be from the provided Memory items. - Return strictly valid JSON only. Example: If you have items about a project across multiple phases, group them by milestone, team, or event. +Language rules: +- The `key` fields must match the mostly used language of the clustered memories. **如果输入是中文，请输出中文** + Return valid JSON: -{{ +{ "clusters": [ - {{ - "ids": ["id1", "id2", ...], - "theme": "" - }}, + { + "ids": ["", "", ...], + "key": "" + }, ... ] -}} +} Memory items: {joined_scene} @@ -70,7 +122,7 @@ Valid options: - CAUSE: One clearly leads to the other. - CONDITION: One happens only if the other condition holds. -- RELATE_TO: They are semantically related by shared people, time, place, or event, but neither causes the other. +- RELATE: They are semantically related by shared people, time, place, or event, but neither causes the other. - CONFLICT: They logically contradict each other. - NONE: No clear useful connection. @@ -84,7 +136,7 @@ - Node 2: "The venue was booked for a wedding in August." Answer: CONFLICT -Always respond with ONE word: [CAUSE | CONDITION | RELATE_TO | CONFLICT | NONE] +Always respond with ONE word, no matter what language is for the input nodes: [CAUSE | CONDITION | RELATE | CONFLICT | NONE] """ INFER_FACT_PROMPT = """ @@ -125,13 +177,16 @@ - "Mary organized the 2023 sustainability summit in Berlin." - "Mary presented a keynote on renewable energy at the same summit." +Language rules: +- The `key`, `value`, `tags`, `background` fields must match the language of the input. + Good Aggregate: -{{ +{ "key": "Mary's Sustainability Summit Role", "value": "Mary organized and spoke at the 2023 sustainability summit in Berlin, highlighting renewable energy initiatives.", "tags": ["Mary", "summit", "Berlin", "2023"], "background": "Combined from multiple memories about Mary's activities at the summit." -}} +} If you find NO useful higher-level concept, reply exactly: "None". """ diff --git a/src/memos/vec_dbs/base.py b/src/memos/vec_dbs/base.py index 2ffa3b539..ee1bfb3ca 100644 --- a/src/memos/vec_dbs/base.py +++ b/src/memos/vec_dbs/base.py @@ -55,6 +55,10 @@ def search( def get_by_id(self, id: str) -> VecDBItem | None: """Get an item from the vector database.""" + @abstractmethod + def get_by_ids(self, ids: list[str]) -> list[VecDBItem]: + """Get multiple items by their IDs.""" + @abstractmethod def get_by_filter(self, filter: dict[str, Any]) -> list[VecDBItem]: """ @@ -103,3 +107,11 @@ def upsert(self, data: list[VecDBItem | dict[str, Any]]) -> None: @abstractmethod def delete(self, ids: list[str]) -> None: """Delete items from the vector database.""" + + @abstractmethod + def ensure_payload_indexes(self, fields: list[str]) -> None: + """ + Create payload indexes for specified fields in the collection. + Args: + fields (list[str]): List of field names to index (as keyword). + """ diff --git a/src/memos/vec_dbs/qdrant.py b/src/memos/vec_dbs/qdrant.py index f569918f1..a0ebf1d80 100644 --- a/src/memos/vec_dbs/qdrant.py +++ b/src/memos/vec_dbs/qdrant.py @@ -1,17 +1,7 @@ from typing import Any -from qdrant_client import QdrantClient -from qdrant_client.http import models -from qdrant_client.http.models import ( - Distance, - FieldCondition, - Filter, - MatchValue, - PointStruct, - VectorParams, -) - from memos.configs.vec_db import QdrantVecDBConfig +from memos.dependency import require_python_package from memos.log import get_logger from memos.vec_dbs.base import BaseVecDB from memos.vec_dbs.item import VecDBItem @@ -23,8 +13,15 @@ class QdrantVecDB(BaseVecDB): """Qdrant vector database implementation.""" + @require_python_package( + import_name="qdrant_client", + install_command="pip install qdrant-client", + install_link="https://python-client.qdrant.tech/", + ) def __init__(self, config: QdrantVecDBConfig): """Initialize the Qdrant vector database and the collection.""" + from qdrant_client import QdrantClient + self.config = config # If both host and port are None, we are running in local mode @@ -43,6 +40,7 @@ def __init__(self, config: QdrantVecDBConfig): def create_collection(self) -> None: """Create a new collection with specified parameters.""" + from qdrant_client.http import models if self.collection_exists(self.config.collection_name): collection_info = self.client.get_collection(self.config.collection_name) @@ -54,14 +52,14 @@ def create_collection(self) -> None: # Map string distance metric to Qdrant Distance enum distance_map = { - "cosine": Distance.COSINE, - "euclidean": Distance.EUCLID, - "dot": Distance.DOT, + "cosine": models.Distance.COSINE, + "euclidean": models.Distance.EUCLID, + "dot": models.Distance.DOT, } self.client.create_collection( collection_name=self.config.collection_name, - vectors_config=VectorParams( + vectors_config=models.VectorParams( size=self.config.vector_dimension, distance=distance_map[self.config.distance_metric], ), @@ -122,16 +120,20 @@ def search( for point in response ] - def _dict_to_filter(self, filter_dict: dict[str, Any]) -> Filter: + def _dict_to_filter(self, filter_dict: dict[str, Any]) -> Any: + from qdrant_client.http import models + """Convert a dictionary filter to a Qdrant Filter object.""" conditions = [] for field, value in filter_dict.items(): # Simple exact match for now # TODO: Extend this to support more complex conditions - conditions.append(FieldCondition(key=field, match=MatchValue(value=value))) + conditions.append( + models.FieldCondition(key=field, match=models.MatchValue(value=value)) + ) - return Filter(must=conditions) + return models.Filter(must=conditions) def get_by_id(self, id: str) -> VecDBItem | None: """Get a single item by ID.""" @@ -235,6 +237,8 @@ def count(self, filter: dict[str, Any] | None = None) -> int: return response.count def add(self, data: list[VecDBItem | dict[str, Any]]) -> None: + from qdrant_client.http import models + """ Add data to the vector database. @@ -249,13 +253,14 @@ def add(self, data: list[VecDBItem | dict[str, Any]]) -> None: if isinstance(item, dict): item = item.copy() item = VecDBItem.from_dict(item) - point = PointStruct(id=item.id, vector=item.vector, payload=item.payload) + point = models.PointStruct(id=item.id, vector=item.vector, payload=item.payload) points.append(point) self.client.upsert(collection_name=self.config.collection_name, points=points) def update(self, id: str, data: VecDBItem | dict[str, Any]) -> None: """Update an item in the vector database.""" + from qdrant_client.http import models if isinstance(data, dict): data = data.copy() @@ -265,7 +270,7 @@ def update(self, id: str, data: VecDBItem | dict[str, Any]) -> None: # For vector updates (with or without payload), use upsert with the same ID self.client.upsert( collection_name=self.config.collection_name, - points=[PointStruct(id=id, vector=data.vector, payload=data.payload)], + points=[models.PointStruct(id=id, vector=data.vector, payload=data.payload)], ) else: # For payload-only updates @@ -273,6 +278,25 @@ def update(self, id: str, data: VecDBItem | dict[str, Any]) -> None: collection_name=self.config.collection_name, payload=data.payload, points=[id] ) + def ensure_payload_indexes(self, fields: list[str]) -> None: + """ + Create payload indexes for specified fields in the collection. + This is idempotent: it will skip if index already exists. + + Args: + fields (list[str]): List of field names to index (as keyword). + """ + for field in fields: + try: + self.client.create_payload_index( + collection_name=self.config.collection_name, + field_name=field, + field_schema="keyword", # Could be extended in future + ) + logger.debug(f"Qdrant payload index on '{field}' ensured.") + except Exception as e: + logger.warning(f"Failed to create payload index on '{field}': {e}") + def upsert(self, data: list[VecDBItem | dict[str, Any]]) -> None: """ Add or update data in the vector database. @@ -284,6 +308,8 @@ def upsert(self, data: list[VecDBItem | dict[str, Any]]) -> None: self.add(data) def delete(self, ids: list[str]) -> None: + from qdrant_client.http import models + """Delete items from the vector database.""" point_ids: list[str | int] = ids self.client.delete( diff --git a/tests/chunkers/test_sentence_chunker.py b/tests/chunkers/test_sentence_chunker.py index 3fcc256cf..28aaeabb9 100644 --- a/tests/chunkers/test_sentence_chunker.py +++ b/tests/chunkers/test_sentence_chunker.py @@ -9,8 +9,8 @@ class TestSentenceChunker(unittest.TestCase): def test_sentence_chunker(self): """Test SentenceChunker functionality with mocked backend.""" - with patch("memos.chunkers.sentence_chunker.ChonkieSentenceChunker") as mock_chunker_cls: - # Set up the mock for ChonkieSentenceChunker + with patch("chonkie.SentenceChunker") as mock_chunker_cls: + # Set up the mock for SentenceChunker mock_chunker = MagicMock() mock_chunks = [ MagicMock( diff --git a/tests/embedders/test_ark.py b/tests/embedders/test_ark.py new file mode 100644 index 000000000..08c23aae6 --- /dev/null +++ b/tests/embedders/test_ark.py @@ -0,0 +1,64 @@ +import unittest + +from unittest.mock import patch + +from memos.configs.embedder import EmbedderConfigFactory +from memos.embedders.factory import ArkEmbedder, EmbedderFactory + + +class TestEmbedderFactory(unittest.TestCase): + @patch.object(ArkEmbedder, "embed") + def test_embed_single_text(self, mock_embed): + """Test embedding a single text.""" + mock_embed.return_value = [[0.1, 0.2, 0.3, 0.4, 0.5, 0.6]] + + config = EmbedderConfigFactory.model_validate( + { + "backend": "ark", + "config": { + "model_name_or_path": "doubao-embedding-vision-250615", + "embedding_dims": 2048, + "api_key": "your-api-key", + "api_base": "https://ark.cn-beijing.volces.com/api/v3", + }, + } + ) + embedder = EmbedderFactory.from_config(config) + text = "This is a sample text for embedding generation." + result = embedder.embed([text]) + + mock_embed.assert_called_once_with([text]) + self.assertEqual(len(result[0]), 6) + + @patch.object(ArkEmbedder, "embed") + def test_embed_batch_text(self, mock_embed): + """Test embedding multiple texts at once.""" + mock_embed.return_value = [ + [0.1, 0.2, 0.3, 0.4, 0.5, 0.6], + [0.6, 0.5, 0.4, 0.3, 0.2, 0.1], + [0.3, 0.4, 0.5, 0.6, 0.1, 0.2], + ] + + config = EmbedderConfigFactory.model_validate( + { + "backend": "ark", + "config": { + "model_name_or_path": "doubao-embedding-vision-250615", + "embedding_dims": 2048, + "api_key": "your-api-key", + "api_base": "https://ark.cn-beijing.volces.com/api/v3", + }, + } + ) + embedder = EmbedderFactory.from_config(config) + texts = [ + "First sample text for batch embedding.", + "Second sample text for batch embedding.", + "Third sample text for batch embedding.", + ] + + result = embedder.embed(texts) + + mock_embed.assert_called_once_with(texts) + self.assertEqual(len(result), 3) + self.assertEqual(len(result[0]), 6) diff --git a/tests/embedders/test_universal_api.py b/tests/embedders/test_universal_api.py new file mode 100644 index 000000000..e4ebb7019 --- /dev/null +++ b/tests/embedders/test_universal_api.py @@ -0,0 +1,76 @@ +import unittest + +from unittest.mock import MagicMock, patch + +from memos.configs.embedder import UniversalAPIEmbedderConfig +from memos.embedders.universal_api import UniversalAPIEmbedder + + +class TestUniversalAPIEmbedder(unittest.TestCase): + @patch("memos.embedders.universal_api.OpenAIClient") + def test_embed_single_text(self, mock_openai_client): + """Test embedding a single text with OpenAI provider.""" + # Mock the embeddings.create return value + mock_response = MagicMock() + mock_response.data = [MagicMock(embedding=[0.1, 0.2, 0.3, 0.4])] + mock_openai_client.return_value.embeddings.create.return_value = mock_response + + config = UniversalAPIEmbedderConfig( + provider="openai", + api_key="fake-api-key", + base_url="https://api.openai.com/v1", + model_name_or_path="text-embedding-3-large", + ) + + embedder = UniversalAPIEmbedder(config) + text = ["Test input for embedding."] + result = embedder.embed(text) + + # Assert OpenAIClient was created with proper args + mock_openai_client.assert_called_once_with( + api_key="fake-api-key", + base_url="https://api.openai.com/v1", + ) + + # Assert embeddings.create called with correct params + embedder.client.embeddings.create.assert_called_once_with( + model="text-embedding-3-large", + input=text, + ) + + self.assertEqual(len(result[0]), 4) + + @patch("memos.embedders.universal_api.OpenAIClient") + def test_embed_batch_text(self, mock_openai_client): + """Test embedding multiple texts at once with OpenAI provider.""" + # Mock response for multiple texts + mock_response = MagicMock() + mock_response.data = [ + MagicMock(embedding=[0.1, 0.2]), + MagicMock(embedding=[0.3, 0.4]), + MagicMock(embedding=[0.5, 0.6]), + ] + mock_openai_client.return_value.embeddings.create.return_value = mock_response + + config = UniversalAPIEmbedderConfig( + provider="openai", + api_key="fake-api-key", + base_url="https://api.openai.com/v1", + model_name_or_path="text-embedding-3-large", + ) + + embedder = UniversalAPIEmbedder(config) + texts = ["First text.", "Second text.", "Third text."] + result = embedder.embed(texts) + + embedder.client.embeddings.create.assert_called_once_with( + model="text-embedding-3-large", + input=texts, + ) + + self.assertEqual(len(result), 3) + self.assertEqual(result[0], [0.1, 0.2]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/llms/test_deepseek.py b/tests/llms/test_deepseek.py new file mode 100644 index 000000000..75c1ead5f --- /dev/null +++ b/tests/llms/test_deepseek.py @@ -0,0 +1,88 @@ +import unittest + +from types import SimpleNamespace +from unittest.mock import MagicMock + +from memos.configs.llm import DeepSeekLLMConfig +from memos.llms.deepseek import DeepSeekLLM + + +class TestDeepSeekLLM(unittest.TestCase): + def test_deepseek_llm_generate_with_and_without_think_prefix(self): + """Test DeepSeekLLM generate method with and without tag removal.""" + + # Simulated full content including tag + full_content = "Thinking in progress...Hello from DeepSeek!" + + # Mock response object + mock_response = MagicMock() + mock_response.model_dump_json.return_value = '{"mock": "true"}' + mock_response.choices[0].message.content = full_content + + # Config with think prefix preserved + config_with_think = DeepSeekLLMConfig.model_validate( + { + "model_name_or_path": "deepseek-chat", + "temperature": 0.7, + "max_tokens": 512, + "top_p": 0.9, + "api_key": "sk-test", + "api_base": "https://api.deepseek.com/v1", + "remove_think_prefix": False, + } + ) + llm_with_think = DeepSeekLLM(config_with_think) + llm_with_think.client.chat.completions.create = MagicMock(return_value=mock_response) + + output_with_think = llm_with_think.generate([{"role": "user", "content": "Hello"}]) + self.assertEqual(output_with_think, full_content) + + # Config with think tag removed + config_without_think = config_with_think.model_copy(update={"remove_think_prefix": True}) + llm_without_think = DeepSeekLLM(config_without_think) + llm_without_think.client.chat.completions.create = MagicMock(return_value=mock_response) + + output_without_think = llm_without_think.generate([{"role": "user", "content": "Hello"}]) + self.assertEqual(output_without_think, "Hello from DeepSeek!") + + def test_deepseek_llm_generate_stream(self): + """Test DeepSeekLLM generate_stream with reasoning_content and content chunks.""" + + def make_chunk(delta_dict): + # Create a simulated stream chunk with delta fields + delta = SimpleNamespace(**delta_dict) + choice = SimpleNamespace(delta=delta) + return SimpleNamespace(choices=[choice]) + + # Simulate chunks: reasoning + answer + mock_stream_chunks = [ + make_chunk({"reasoning_content": "Analyzing..."}), + make_chunk({"content": "Hello"}), + make_chunk({"content": ", "}), + make_chunk({"content": "DeepSeek!"}), + ] + + mock_chat_completions_create = MagicMock(return_value=iter(mock_stream_chunks)) + + config = DeepSeekLLMConfig.model_validate( + { + "model_name_or_path": "deepseek-chat", + "temperature": 0.7, + "max_tokens": 512, + "top_p": 0.9, + "api_key": "sk-test", + "api_base": "https://api.deepseek.com/v1", + "remove_think_prefix": False, + } + ) + llm = DeepSeekLLM(config) + llm.client.chat.completions.create = mock_chat_completions_create + + messages = [{"role": "user", "content": "Say hello"}] + streamed = list(llm.generate_stream(messages)) + full_output = "".join(streamed) + + self.assertIn("Analyzing...", full_output) + self.assertIn("Hello, DeepSeek!", full_output) + self.assertTrue(full_output.startswith("Analyzing...")) + self.assertTrue(full_output.endswith("DeepSeek!")) diff --git a/tests/llms/test_openai.py b/tests/llms/test_openai.py index 90bc7ab2d..dff57c058 100644 --- a/tests/llms/test_openai.py +++ b/tests/llms/test_openai.py @@ -1,5 +1,6 @@ import unittest +from types import SimpleNamespace from unittest.mock import MagicMock from memos.configs.llm import LLMConfigFactory @@ -40,3 +41,62 @@ def test_llm_factory_with_mocked_openai_backend(self): response, "Hello! I'm an AI language model created by OpenAI. I'm here to help answer questions, provide information, and assist with a wide range of topics. How can I assist you today?", ) + + def test_llm_factory_with_stream_openai_backend(self): + """Test LLMFactory stream generation with mocked OpenAI backend.""" + + def make_chunk(delta_dict): + # Create a mock response chunk with a simulated delta dictionary + delta = SimpleNamespace(**delta_dict) + choice = SimpleNamespace(delta=delta, finish_reason="stop", index=0) + return SimpleNamespace(choices=[choice]) + + # Simulate a stream response with both reasoning_content and content + mock_stream_chunks = [ + make_chunk({"reasoning_content": "I am thinking"}), + make_chunk({"content": "Hello"}), + make_chunk({"content": ", "}), + make_chunk({"content": "world!"}), + ] + + # Mock the streaming chat completion call + mock_chat_completions_create = MagicMock(return_value=iter(mock_stream_chunks)) + + # Create the LLM config with think prefix enabled + config = LLMConfigFactory.model_validate( + { + "backend": "openai", + "config": { + "model_name_or_path": "gpt-4.1-nano", + "temperature": 0.8, + "max_tokens": 1024, + "top_p": 0.9, + "top_k": 50, + "api_key": "sk-xxxx", + "api_base": "https://api.openai.com/v1", + "remove_think_prefix": False, + # Ensure tag is emitted + }, + } + ) + + # Instantiate the LLM and inject the mocked stream method + llm = LLMFactory.from_config(config) + llm.client.chat.completions.create = mock_chat_completions_create + + # Input message to the model + messages = [{"role": "user", "content": "Think and say hello"}] + + # Collect streamed output as a list of chunks + response_parts = list(llm.generate_stream(messages)) + response = "".join(response_parts) + + # Assert the presence of the tag and expected content + self.assertIn("", response) + self.assertIn("I am thinking", response) + self.assertIn("Hello, world!", response) + + # Optional: check structure of stream response + self.assertEqual(response_parts[0], "") + self.assertTrue(response.startswith("I am thinking")) + self.assertTrue(response.endswith("Hello, world!")) diff --git a/tests/llms/test_qwen.py b/tests/llms/test_qwen.py new file mode 100644 index 000000000..90f31e47f --- /dev/null +++ b/tests/llms/test_qwen.py @@ -0,0 +1,101 @@ +import unittest + +from types import SimpleNamespace +from unittest.mock import MagicMock + +from memos.configs.llm import QwenLLMConfig +from memos.llms.qwen import QwenLLM + + +class TestQwenLLM(unittest.TestCase): + def test_qwen_llm_generate_with_and_without_think_prefix(self): + """Test QwenLLM non-streaming response generation with and without prefix removal.""" + + # Simulated full response content with tag + full_content = "Analyzing your request...Hello, world!" + + # Prepare the mock response object with expected structure + mock_response = MagicMock() + mock_response.model_dump_json.return_value = '{"mocked": "true"}' + mock_response.choices[0].message.content = full_content + + # Create config with remove_think_prefix = False + config_with_think = QwenLLMConfig.model_validate( + { + "model_name_or_path": "qwen-test", + "temperature": 0.7, + "max_tokens": 100, + "top_p": 0.9, + "api_key": "sk-test", + "api_base": "https://dashscope.aliyuncs.com/api/v1", + "remove_think_prefix": False, + } + ) + + # Instance with think tag enabled + llm_with_think = QwenLLM(config_with_think) + llm_with_think.client.chat.completions.create = MagicMock(return_value=mock_response) + + response_with_think = llm_with_think.generate([{"role": "user", "content": "Hi"}]) + self.assertEqual(response_with_think, full_content) + + # Create config with remove_think_prefix = True + config_without_think = config_with_think.model_copy(update={"remove_think_prefix": True}) + + # Instance with think tag removed + llm_without_think = QwenLLM(config_without_think) + llm_without_think.client.chat.completions.create = MagicMock(return_value=mock_response) + + response_without_think = llm_without_think.generate([{"role": "user", "content": "Hi"}]) + self.assertEqual(response_without_think, "Hello, world!") + self.assertNotIn("", response_without_think) + + def test_qwen_llm_generate_stream(self): + """Test QwenLLM stream generation with both reasoning_content and content.""" + + def make_chunk(delta_dict): + # Construct a mock chunk with delta fields + delta = SimpleNamespace(**delta_dict) + choice = SimpleNamespace(delta=delta) + return SimpleNamespace(choices=[choice]) + + # Simulate a sequence of streamed chunks + mock_stream_chunks = [ + make_chunk({"reasoning_content": "Analyzing input..."}), + make_chunk({"content": "Hello"}), + make_chunk({"content": ", "}), + make_chunk({"content": "world!"}), + ] + + # Mock the client's streaming response + mock_chat_completions_create = MagicMock(return_value=iter(mock_stream_chunks)) + + # Build QwenLLM config with think prefix enabled + config = QwenLLMConfig.model_validate( + { + "model_name_or_path": "qwen-test", + "temperature": 0.7, + "max_tokens": 100, + "top_p": 0.9, + "api_key": "sk-test", + "api_base": "https://dashscope.aliyuncs.com/api/v1", + "remove_think_prefix": False, + } + ) + + # Create QwenLLM instance and inject mock client + llm = QwenLLM(config) + llm.client.chat.completions.create = mock_chat_completions_create + + messages = [{"role": "user", "content": "Say hello"}] + + # Collect the streamed output + response_parts = list(llm.generate_stream(messages)) + response = "".join(response_parts) + + # Assertions for structure and content + self.assertIn("", response) + self.assertIn("Analyzing input...", response) + self.assertIn("Hello, world!", response) + self.assertTrue(response.startswith("Analyzing input...")) + self.assertTrue(response.endswith("Hello, world!")) diff --git a/tests/mem_os/test_memos.py b/tests/mem_os/test_memos.py index 1e0997fd0..ebd742678 100644 --- a/tests/mem_os/test_memos.py +++ b/tests/mem_os/test_memos.py @@ -100,3 +100,76 @@ def test_mos_has_core_methods(mock_llm, mock_reader, mock_user_manager, simple_c assert callable(mos.chat) assert callable(mos.search) assert callable(mos.add) + + +@patch("memos.mem_os.core.UserManager") +@patch("memos.mem_os.core.MemReaderFactory") +@patch("memos.mem_os.core.LLMFactory") +@patch("memos.mem_os.main.MOSCore.chat") +def test_mos_chat_with_custom_prompt_no_cot( + mock_core_chat, mock_llm, mock_reader, mock_user_manager, simple_config +): + """Test that MOS.chat passes base_prompt to MOSCore.chat when CoT is disabled.""" + # Mock all dependencies + mock_llm.from_config.return_value = MagicMock() + mock_reader.from_config.return_value = MagicMock() + user_manager_instance = MagicMock() + user_manager_instance.validate_user.return_value = True + mock_user_manager.return_value = user_manager_instance + + # Disable CoT + simple_config.PRO_MODE = False + mos = MOS(simple_config) + + # Call chat with a custom prompt + custom_prompt = "You are a helpful bot." + mos.chat("Hello", user_id="test_user", base_prompt=custom_prompt) + + # Assert that the core chat method was called with the custom prompt + mock_core_chat.assert_called_once_with("Hello", "test_user", base_prompt=custom_prompt) + + +@patch("memos.mem_os.core.UserManager") +@patch("memos.mem_os.core.MemReaderFactory") +@patch("memos.mem_os.core.LLMFactory") +@patch("memos.mem_os.main.MOS._generate_enhanced_response_with_context") +@patch("memos.mem_os.main.MOS.cot_decompose") +@patch("memos.mem_os.main.MOS.get_sub_answers") +def test_mos_chat_with_custom_prompt_with_cot( + mock_get_sub_answers, + mock_cot_decompose, + mock_generate_enhanced_response, + mock_llm, + mock_reader, + mock_user_manager, + simple_config, +): + """Test that MOS.chat passes base_prompt correctly when CoT is enabled.""" + # Mock dependencies + mock_llm.from_config.return_value = MagicMock() + mock_reader.from_config.return_value = MagicMock() + user_manager_instance = MagicMock() + user_manager_instance.validate_user.return_value = True + user_manager_instance.get_user_cubes.return_value = [MagicMock(cube_id="test_cube")] + mock_user_manager.return_value = user_manager_instance + + # Mock CoT process + mock_cot_decompose.return_value = {"is_complex": True, "sub_questions": ["Sub-question 1"]} + mock_get_sub_answers.return_value = (["Sub-question 1"], ["Sub-answer 1"]) + + # Enable CoT + simple_config.PRO_MODE = True + mos = MOS(simple_config) + + # Mock the search engine to avoid errors + mos.mem_cubes["test_cube"] = MagicMock() + mos.mem_cubes["test_cube"].text_mem = MagicMock() + + # Call chat with a custom prompt + custom_prompt = "You are a super helpful bot. Context: {memories}" + mos.chat("Complex question", user_id="test_user", base_prompt=custom_prompt) + + # Assert that the enhanced response generator was called with the prompt + mock_generate_enhanced_response.assert_called_once() + call_args = mock_generate_enhanced_response.call_args[1] + assert call_args.get("base_prompt") == custom_prompt diff --git a/tests/mem_os/test_memos_core.py b/tests/mem_os/test_memos_core.py index 3623bd900..88923fc94 100644 --- a/tests/mem_os/test_memos_core.py +++ b/tests/mem_os/test_memos_core.py @@ -6,6 +6,7 @@ import pytest from memos.configs.mem_os import MOSConfig +from memos.mem_cube.general import GeneralMemCube from memos.mem_os.core import MOSCore from memos.mem_user.user_manager import UserRole from memos.memories.textual.item import TextualMemoryItem, TextualMemoryMetadata @@ -284,16 +285,15 @@ def test_register_mem_cube( mock_user_manager_class.return_value = mock_user_manager mock_user_manager.get_cube.return_value = None # Cube doesn't exist - with patch("memos.mem_os.core.GeneralMemCube") as mock_general_cube: - mock_general_cube.init_from_dir.return_value = mock_mem_cube - + # Mock only the static method, not the entire class + with patch.object(GeneralMemCube, "init_from_dir", return_value=mock_mem_cube): mos = MOSCore(MOSConfig(**mock_config)) with patch("os.path.exists", return_value=True): mos.register_mem_cube("test_cube_path", "test_cube_1") assert "test_cube_1" in mos.mem_cubes - mock_general_cube.init_from_dir.assert_called_once_with("test_cube_path") + GeneralMemCube.init_from_dir.assert_called_once_with("test_cube_path") @patch("memos.mem_os.core.UserManager") @patch("memos.mem_os.core.MemReaderFactory") @@ -328,6 +328,124 @@ def test_search_memories( assert result["text_mem"][0]["cube_id"] == "test_cube_1" mock_mem_cube.text_mem.search.assert_called_once_with("football", top_k=5) + @patch("memos.mem_os.core.UserManager") + @patch("memos.mem_os.core.MemReaderFactory") + @patch("memos.mem_os.core.LLMFactory") + @patch("memos.mem_os.core.logger") + def test_register_mem_cube_embedder_consistency_warning( + self, + mock_logger, + mock_llm_factory, + mock_reader_factory, + mock_user_manager_class, + mock_config, + mock_llm, + mock_mem_reader, + mock_user_manager, + mock_mem_cube, + ): + """Test embedder consistency warning when cube embedder differs from MOS config.""" + # Setup mocks + mock_llm_factory.from_config.return_value = mock_llm + mock_reader_factory.from_config.return_value = mock_mem_reader + mock_user_manager_class.return_value = mock_user_manager + mock_user_manager.get_cube.return_value = None # Cube doesn't exist + + # Create different embedder configs for MOS and cube + mos_embedder_config = { + "backend": "ollama", + "config": { + "model_name_or_path": "nomic-embed-text:latest", + }, + } + + cube_embedder_config = { + "backend": "sentence_transformer", + "config": { + "model_name_or_path": "all-MiniLM-L6-v2", + }, + } + + # Mock the cube's text memory embedder config + mock_mem_cube.text_mem.config.embedder = cube_embedder_config + + # Mock only the static method, not the entire class + with patch.object(GeneralMemCube, "init_from_dir", return_value=mock_mem_cube): + mos = MOSCore(MOSConfig(**mock_config)) + + # Ensure MOS config has different embedder + mos.config.mem_reader.config.embedder = mos_embedder_config + + with patch("os.path.exists", return_value=True): + mos.register_mem_cube("test_cube_path", "test_cube_1") + + # Verify warning was logged + mock_logger.warning.assert_called_with( + f"Cube Embedder is not consistent with MOSConfig for cube: test_cube_1, will use Cube Embedder: {cube_embedder_config}" + ) + + # Verify cube was still registered + assert "test_cube_1" in mos.mem_cubes + GeneralMemCube.init_from_dir.assert_called_once_with("test_cube_path") + + @patch("memos.mem_os.core.UserManager") + @patch("memos.mem_os.core.MemReaderFactory") + @patch("memos.mem_os.core.LLMFactory") + @patch("memos.mem_os.core.logger") + def test_register_mem_cube_embedder_consistency_no_warning( + self, + mock_logger, + mock_llm_factory, + mock_reader_factory, + mock_user_manager_class, + mock_config, + mock_llm, + mock_mem_reader, + mock_user_manager, + mock_mem_cube, + ): + """Test no warning when cube embedder is consistent with MOS config.""" + # Setup mocks + mock_llm_factory.from_config.return_value = mock_llm + mock_reader_factory.from_config.return_value = mock_mem_reader + mock_user_manager_class.return_value = mock_user_manager + mock_user_manager.get_cube.return_value = None # Cube doesn't exist + + # Create same embedder config for both MOS and cube + embedder_config = { + "backend": "ollama", + "config": { + "model_name_or_path": "nomic-embed-text:latest", + }, + } + + # Mock the cube's text memory embedder config to be the same + mock_mem_cube.text_mem.config.embedder = embedder_config + + # Mock only the static method, not the entire class + with patch.object(GeneralMemCube, "init_from_dir", return_value=mock_mem_cube): + mos = MOSCore(MOSConfig(**mock_config)) + + # Ensure MOS config has same embedder + mos.config.mem_reader.config.embedder = embedder_config + + with patch("os.path.exists", return_value=True): + mos.register_mem_cube("test_cube_path", "test_cube_1") + + # Verify no embedder consistency warning was logged + warning_calls = [ + call + for call in mock_logger.warning.call_args_list + if "Cube Embedder is not consistent" in str(call) + ] + assert len(warning_calls) == 0, ( + "No embedder consistency warning should be logged when configs match" + ) + + # Verify cube was still registered + assert "test_cube_1" in mos.mem_cubes + GeneralMemCube.init_from_dir.assert_called_once_with("test_cube_path") + @patch("memos.mem_os.core.UserManager") @patch("memos.mem_os.core.MemReaderFactory") @patch("memos.mem_os.core.LLMFactory") @@ -453,11 +571,13 @@ def test_chat_with_memories( mos = MOSCore(MOSConfig(**mock_config)) mos.mem_cubes["test_cube_1"] = mock_mem_cube + mos.mem_cubes["test_cube_2"] = mock_mem_cube # Add the second cube to avoid KeyError response = mos.chat("What do I like?") - # Verify memory search was called - mock_mem_cube.text_mem.search.assert_called_once_with("What do I like?", top_k=5) + # Verify memory search was called (called twice because we have two cubes) + assert mock_mem_cube.text_mem.search.call_count == 2 + mock_mem_cube.text_mem.search.assert_any_call("What do I like?", top_k=5) # Verify LLM was called mock_llm.generate.assert_called_once() @@ -470,6 +590,44 @@ def test_chat_with_memories( assert mos.chat_history_manager["test_user"].chat_history[1]["role"] == "assistant" assert mos.chat_history_manager["test_user"].chat_history[1]["content"] == response + @patch("memos.mem_os.core.UserManager") + @patch("memos.mem_os.core.MemReaderFactory") + @patch("memos.mem_os.core.LLMFactory") + def test_chat_with_custom_base_prompt( + self, + mock_llm_factory, + mock_reader_factory, + mock_user_manager_class, + mock_config, + mock_llm, + mock_mem_reader, + mock_user_manager, + mock_mem_cube, + ): + """Test chat functionality with a custom base prompt.""" + # Setup mocks + mock_llm_factory.from_config.return_value = mock_llm + mock_reader_factory.from_config.return_value = mock_mem_reader + mock_user_manager_class.return_value = mock_user_manager + + mos = MOSCore(MOSConfig(**mock_config)) + mos.mem_cubes["test_cube_1"] = mock_mem_cube + mos.mem_cubes["test_cube_2"] = mock_mem_cube + + custom_prompt = "You are a pirate. Answer as such. User memories: {memories}" + mos.chat("What do I like?", base_prompt=custom_prompt) + + # Verify that the system prompt passed to the LLM is the custom one + mock_llm.generate.assert_called_once() + call_args = mock_llm.generate.call_args[0] + messages = call_args[0] + system_prompt = messages[0]["content"] + + assert "You are a pirate." in system_prompt + assert "You are a knowledgeable and helpful AI assistant." not in system_prompt + assert "User memories:" in system_prompt + assert "I like playing football" in system_prompt # Check if memory is interpolated + @patch("memos.mem_os.core.UserManager") @patch("memos.mem_os.core.MemReaderFactory") @patch("memos.mem_os.core.LLMFactory") @@ -494,6 +652,8 @@ def test_chat_without_memories( config_dict["enable_textual_memory"] = False mos = MOSCore(MOSConfig(**config_dict)) + mos.mem_cubes["test_cube_1"] = MagicMock() # Add the cube to avoid KeyError + mos.mem_cubes["test_cube_2"] = MagicMock() # Add the second cube to avoid KeyError response = mos.chat("Hello") @@ -540,6 +700,60 @@ def test_clear_messages( assert mos.chat_history_manager["test_user"].user_id == "test_user" +class TestMOSSystemPrompt: + """Test the _build_system_prompt method in MOSCore.""" + + @pytest.fixture + def mos_core_instance(self, mock_config, mock_user_manager): + """Fixture to create a MOSCore instance for testing the prompt builder.""" + with patch("memos.mem_os.core.LLMFactory"), patch("memos.mem_os.core.MemReaderFactory"): + return MOSCore(MOSConfig(**mock_config), user_manager=mock_user_manager) + + def test_build_prompt_with_template_and_memories(self, mos_core_instance): + """Test prompt with a template and memories.""" + base_prompt = "You are a sales agent. Here are past interactions: {memories}" + memories = [TextualMemoryItem(memory="User likes blue cars.")] + prompt = mos_core_instance._build_system_prompt(memories, base_prompt) + assert "You are a sales agent." in prompt + assert "1. User likes blue cars." in prompt + assert "{memories}" not in prompt + + def test_build_prompt_with_template_no_memories(self, mos_core_instance): + """Test prompt with a template but no memories.""" + base_prompt = "You are a sales agent. Here are past interactions: {memories}" + prompt = mos_core_instance._build_system_prompt(None, base_prompt) + assert "You are a sales agent." in prompt + assert "Here are past interactions:" in prompt + # The placeholder should be replaced with an empty string + assert "{memories}" not in prompt + # Check that the output is clean + assert prompt.strip() == "You are a sales agent. Here are past interactions:" + assert "## Memories:" not in prompt + + def test_build_prompt_no_template_with_memories(self, mos_core_instance): + """Test prompt without a template but with memories (backward compatibility).""" + base_prompt = "You are a helpful assistant." + memories = [TextualMemoryItem(memory="User is a developer.")] + prompt = mos_core_instance._build_system_prompt(memories, base_prompt) + assert "You are a helpful assistant." in prompt + assert "## Memories:" in prompt + assert "1. User is a developer." in prompt + + def test_build_prompt_default_with_memories(self, mos_core_instance): + """Test default prompt with memories.""" + memories = [TextualMemoryItem(memory="User lives in New York.")] + prompt = mos_core_instance._build_system_prompt(memories) + assert "You are a knowledgeable and helpful AI assistant." in prompt + assert "## Memories:" in prompt + assert "1. User lives in New York." in prompt + + def test_build_prompt_default_no_memories(self, mos_core_instance): + """Test default prompt without any memories.""" + prompt = mos_core_instance._build_system_prompt() + assert "You are a knowledgeable and helpful AI assistant." in prompt + assert "## Memories:" not in prompt + + class TestMOSErrorHandling: """Test MOS error handling.""" diff --git a/tests/mem_reader/test_simple_structure.py b/tests/mem_reader/test_simple_structure.py index 0df521338..91996da91 100644 --- a/tests/mem_reader/test_simple_structure.py +++ b/tests/mem_reader/test_simple_structure.py @@ -75,7 +75,9 @@ def test_process_doc_data(self): info = {"user_id": "user1", "session_id": "session1"} # Mock LLM response - mock_response = '{"summary": "A sample document about testing.", "tags": ["document"]}' + mock_response = ( + '{"value": "A sample document about testing.", "tags": ["document"], "key": "title"}' + ) self.reader.llm.generate.return_value = mock_response self.reader.chunker.chunk.return_value = [ Chunk(text="Parsed document text", token_count=3, sentences=["Parsed document text"]) diff --git a/tests/mem_scheduler/test_retriever.py b/tests/mem_scheduler/test_retriever.py new file mode 100644 index 000000000..6d2e79edd --- /dev/null +++ b/tests/mem_scheduler/test_retriever.py @@ -0,0 +1,138 @@ +import sys +import unittest + +from pathlib import Path +from unittest.mock import MagicMock, patch + +from memos.configs.mem_scheduler import SchedulerConfigFactory +from memos.llms.base import BaseLLM +from memos.mem_cube.general import GeneralMemCube +from memos.mem_scheduler.scheduler_factory import SchedulerFactory +from memos.memories.textual.tree import TreeTextMemory + + +FILE_PATH = Path(__file__).absolute() +BASE_DIR = FILE_PATH.parent.parent.parent +sys.path.insert(0, str(BASE_DIR)) # Enable execution from any working directory + + +class TestSchedulerRetriever(unittest.TestCase): + def setUp(self): + """Initialize test environment with mock objects.""" + example_scheduler_config_path = ( + f"{BASE_DIR}/examples/data/config/mem_scheduler/general_scheduler_config.yaml" + ) + scheduler_config = SchedulerConfigFactory.from_yaml_file( + yaml_path=example_scheduler_config_path + ) + mem_scheduler = SchedulerFactory.from_config(scheduler_config) + self.scheduler = mem_scheduler + self.llm = MagicMock(spec=BaseLLM) + self.mem_cube = MagicMock(spec=GeneralMemCube) + self.tree_text_memory = MagicMock(spec=TreeTextMemory) + self.mem_cube.text_mem = self.tree_text_memory + self.mem_cube.act_mem = MagicMock() + + # Initialize modules with mock LLM + self.scheduler.initialize_modules(chat_llm=self.llm, process_llm=self.llm) + self.scheduler.mem_cube = self.mem_cube + + self.retriever = self.scheduler.retriever + + # Mock logging to verify messages + self.logging_warning_patch = patch("logging.warning") + self.mock_logging_warning = self.logging_warning_patch.start() + + self.logger_info_patch = patch("memos.mem_scheduler.modules.retriever.logger.info") + self.mock_logger_info = self.logger_info_patch.start() + + def tearDown(self): + """Clean up patches.""" + self.logging_warning_patch.stop() + self.logger_info_patch.stop() + + def test_filter_similar_memories_empty_input(self): + """Test filter_similar_memories with empty input list.""" + result = self.retriever.filter_similar_memories([]) + self.assertEqual(result, []) + self.mock_logging_warning.assert_called_with( + "Received empty memories list - nothing to filter" + ) + + def test_filter_similar_memories_no_duplicates(self): + """Test filter_similar_memories with no duplicate memories.""" + memories = [ + "This is a completely unique first memory", + "This second memory is also totally unique", + "And this third one has nothing in common with the others", + ] + + result = self.retriever.filter_similar_memories(memories) + self.assertEqual(len(result), 3) + self.assertEqual(set(result), set(memories)) + + def test_filter_similar_memories_with_duplicates(self): + """Test filter_similar_memories with duplicate memories.""" + memories = [ + "The user is planning to move to Chicago next month, although the exact date of the move is unclear.", + "The user is planning to move to Chicago next month, which reflects a significant change in their living situation.", + "The user is planning to move to Chicago in the upcoming month, indicating a significant change in their living situation.", + ] + result = self.retriever.filter_similar_memories(memories, similarity_threshold=0.75) + self.assertLess(len(result), len(memories)) + + # Verify logging was called for removed items + self.assertGreater(self.mock_logger_info.call_count, 0) + + def test_filter_similar_memories_error_handling(self): + """Test filter_similar_memories error handling.""" + # Test with non-string input (should return original list due to error) + memories = ["valid text", 12345, "another valid text"] + result = self.retriever.filter_similar_memories(memories) + self.assertEqual(result, memories) + + def test_filter_too_short_memories_empty_input(self): + """Test filter_too_short_memories with empty input list.""" + result = self.retriever.filter_too_short_memories([]) + self.assertEqual(result, []) + + def test_filter_too_short_memories_all_valid(self): + """Test filter_too_short_memories with all valid memories.""" + memories = [ + "This memory is definitely long enough to be kept", + "This one is also sufficiently lengthy to pass the filter", + "And this third memory meets the minimum length requirements too", + ] + + result = self.retriever.filter_too_short_memories(memories, min_length_threshold=5) + self.assertEqual(len(result), 3) + self.assertEqual(result, memories) + + def test_filter_too_short_memories_with_short_ones(self): + """Test filter_too_short_memories with some short memories.""" + memories = [ + "This is long enough", # 5 words + "Too short", # 2 words + "This one passes", # 3 words (assuming threshold is 3) + "Nope", # 1 word + "This is also acceptable", # 4 words + ] + + # Test with word count threshold of 3 + result = self.retriever.filter_too_short_memories(memories, min_length_threshold=3) + self.assertEqual(len(result), 3) + self.assertNotIn("Too short", result) + self.assertNotIn("Nope", result) + + def test_filter_too_short_memories_edge_case(self): + """Test filter_too_short_memories with edge case length.""" + memories = ["Exactly three words here", "Two words only", "One", "Four words right here"] + + # Test with threshold exactly matching some memories + # The implementation uses word count, not character count + result = self.retriever.filter_too_short_memories(memories, min_length_threshold=3) + self.assertEqual( + len(result), 3 + ) # "Exactly three words here", "Two words only", "Four words right here" + self.assertIn("Exactly three words here", result) + self.assertIn("Four words right here", result) diff --git a/tests/mem_scheduler/test_scheduler.py b/tests/mem_scheduler/test_scheduler.py index 313561c53..cf750a755 100644 --- a/tests/mem_scheduler/test_scheduler.py +++ b/tests/mem_scheduler/test_scheduler.py @@ -1,9 +1,9 @@ -import json import sys import unittest +from datetime import datetime from pathlib import Path -from unittest.mock import MagicMock, call, patch +from unittest.mock import ANY, MagicMock, call, patch from memos.configs.mem_scheduler import SchedulerConfigFactory from memos.llms.base import BaseLLM @@ -12,11 +12,10 @@ from memos.mem_scheduler.modules.retriever import SchedulerRetriever from memos.mem_scheduler.modules.schemas import ( ANSWER_LABEL, - DEFAULT_ACT_MEM_DUMP_PATH, QUERY_LABEL, ScheduleLogForWebItem, ScheduleMessageItem, - TextMemory_SEARCH_METHOD, + TreeTextMemory_SEARCH_METHOD, ) from memos.mem_scheduler.scheduler_factory import SchedulerFactory from memos.memories.textual.tree import TextualMemoryItem, TreeTextMemory @@ -29,6 +28,7 @@ class TestGeneralScheduler(unittest.TestCase): def setUp(self): + """Initialize test environment with mock objects and test scheduler instance.""" example_scheduler_config_path = ( f"{BASE_DIR}/examples/data/config/mem_scheduler/general_scheduler_config.yaml" ) @@ -43,64 +43,59 @@ def setUp(self): self.mem_cube.text_mem = self.tree_text_memory self.mem_cube.act_mem = MagicMock() - # 初始化模块 - self.scheduler.initialize_modules(self.llm) + # Initialize modules with mock LLM + self.scheduler.initialize_modules(chat_llm=self.llm, process_llm=self.llm) self.scheduler.mem_cube = self.mem_cube - # 设置当前用户和内存立方体ID + # Set current user and memory cube ID for testing self.scheduler._current_user_id = "test_user" self.scheduler._current_mem_cube_id = "test_cube" def test_initialization(self): - # 测试初始化参数 - self.assertEqual(self.scheduler.top_k, 10) - self.assertEqual(self.scheduler.top_n, 5) - self.assertEqual(self.scheduler.act_mem_update_interval, 300) - self.assertEqual(self.scheduler.context_window_size, 5) - self.assertEqual(self.scheduler.activation_mem_size, 5) - self.assertEqual(self.scheduler.act_mem_dump_path, DEFAULT_ACT_MEM_DUMP_PATH) - self.assertEqual(self.scheduler.search_method, TextMemory_SEARCH_METHOD) - self.assertEqual(self.scheduler._last_activation_mem_update_time, 0.0) - self.assertEqual(self.scheduler.query_list, []) - - # 测试处理程序注册 + """Test that scheduler initializes with correct default values and handlers.""" + # Verify handler registration self.assertTrue(QUERY_LABEL in self.scheduler.dispatcher.handlers) self.assertTrue(ANSWER_LABEL in self.scheduler.dispatcher.handlers) def test_initialize_modules(self): - # 测试模块初始化 + """Test module initialization with proper component assignments.""" self.assertEqual(self.scheduler.chat_llm, self.llm) self.assertIsInstance(self.scheduler.monitor, SchedulerMonitor) self.assertIsInstance(self.scheduler.retriever, SchedulerRetriever) - def test_query_message_consume(self): - # Create test message + def test_query_message_consumer(self): + # Create test message with all required fields message = ScheduleMessageItem( user_id="test_user", mem_cube_id="test_cube", - mem_cube=self.mem_cube, + mem_cube=self.mem_cube, # or could be str like "test_cube" label=QUERY_LABEL, content="Test query", ) - # Mock the detect_intent method to return a valid JSON string - mock_intent_result = {"trigger_retrieval": False, "missing_evidence": []} + # Mock the detect_intent method to return a valid result + mock_intent_result = {"trigger_retrieval": False, "missing_evidences": []} # Mock the process_session_turn method with ( patch.object(self.scheduler, "process_session_turn") as mock_process_session_turn, - # Also mock detect_intent to avoid JSON parsing issues patch.object(self.scheduler.monitor, "detect_intent") as mock_detect_intent, ): mock_detect_intent.return_value = mock_intent_result # Test message handling - self.scheduler._query_message_consume([message]) - - # Verify method call - mock_process_session_turn.assert_called_once_with(query="Test query", top_k=10, top_n=5) + self.scheduler._query_message_consumer([message]) + + # Verify method call - updated to match new signature + mock_process_session_turn.assert_called_once_with( + queries=["Test query"], # or ["Test query"] depending on implementation + user_id="test_user", + mem_cube_id="test_cube", + mem_cube=self.mem_cube, + top_k=10, + ) - def test_process_session_turn_with_trigger(self): + def test_process_session_turn(self): """Test session turn processing with retrieval trigger.""" # Setup mock working memory working_memory = [ @@ -109,155 +104,148 @@ def test_process_session_turn_with_trigger(self): ] self.tree_text_memory.get_working_memory.return_value = working_memory - # Setup mock memory manager and memory sizes - memory_manager = MagicMock() - memory_manager.memory_size = { - "LongTermMemory": 1000, - "UserMemory": 500, - "WorkingMemory": 100, - } - self.tree_text_memory.memory_manager = memory_manager + # Setup mock memory cube + mem_cube = MagicMock() + mem_cube.text_mem = self.tree_text_memory # Setup intent detection result intent_result = { "trigger_retrieval": True, - "missing_evidence": ["Evidence 1", "Evidence 2"], + "missing_evidences": ["Evidence 1", "Evidence 2"], } + # Create test results that we'll return and expect + result1 = TextualMemoryItem(memory="Result 1") + result2 = TextualMemoryItem(memory="Result 2") + expected_new_memory = [result1, result2] + # Mock methods with ( - patch.object(self.scheduler, "search") as mock_search, - patch.object(self.scheduler, "replace_working_memory") as mock_replace, patch.object(self.scheduler.monitor, "detect_intent") as mock_detect, + patch.object(self.scheduler.retriever, "search") as mock_search, + patch.object(self.scheduler.retriever, "replace_working_memory") as mock_replace, ): mock_detect.return_value = intent_result mock_search.side_effect = [ - [TextualMemoryItem(memory="Result 1")], - [TextualMemoryItem(memory="Result 2")], + [result1], + [result2], ] + mock_replace.return_value = expected_new_memory # Test session turn processing - self.scheduler.process_session_turn(query="Test query") + self.scheduler.process_session_turn( + queries=["Test query"], + user_id="test_user", + mem_cube_id="test_cube", + mem_cube=mem_cube, + top_k=10, + ) # Verify method calls mock_detect.assert_called_once_with( q_list=["Test query"], text_working_memory=["Memory 1", "Memory 2"] ) + + # Verify search calls - using ANY for the method since we can't predict the exact value mock_search.assert_has_calls( [ - call(query="Evidence 1", top_k=5, method=TextMemory_SEARCH_METHOD), - call(query="Evidence 2", top_k=5, method=TextMemory_SEARCH_METHOD), - ] + call(query="Evidence 1", mem_cube=mem_cube, top_k=5, method=ANY), + call(query="Evidence 2", mem_cube=mem_cube, top_k=5, method=ANY), + ], + any_order=True, ) - mock_replace.assert_called_once() + + # Verify replace call - we'll check the structure but not the exact memory items + self.assertEqual(mock_replace.call_count, 1) def test_submit_web_logs(self): - """Test submission of web logs.""" + """Test submission of web logs with updated data structure.""" # Create log message with all required fields log_message = ScheduleLogForWebItem( user_id="test_user", mem_cube_id="test_cube", label=QUERY_LABEL, - log_title="Test Log", + from_memory_type="WorkingMemory", # 新增字段 + to_memory_type="LongTermMemory", # 新增字段 log_content="Test Content", current_memory_sizes={ "long_term_memory_size": 0, "user_memory_size": 0, "working_memory_size": 0, "transformed_act_memory_size": 0, - "parameter_memory_size": 0, }, memory_capacities={ "long_term_memory_capacity": 1000, "user_memory_capacity": 500, "working_memory_capacity": 100, "transformed_act_memory_capacity": 0, - "parameter_memory_capacity": 0, }, ) # Empty the queue by consuming all elements while not self.scheduler._web_log_message_queue.empty(): - self.scheduler._web_log_message_queue.get_nowait() + self.scheduler._web_log_message_queue.get() # Submit the log message self.scheduler._submit_web_logs(messages=log_message) # Verify the message was added to the queue self.assertEqual(self.scheduler._web_log_message_queue.qsize(), 1) - self.assertEqual(self.scheduler._web_log_message_queue.get(), log_message) - def test_memory_reranking(self): - """Test memory reranking process with LLM interaction.""" - # Setup original and new memory - original_memory = [TextualMemoryItem(memory="Original 1")] - new_memory = [TextualMemoryItem(memory="New 1"), TextualMemoryItem(memory="New 2")] - - # Setup LLM response - llm_response = json.dumps({"new_order": ["New 2", "Original 1", "New 1"]}) - self.llm.generate.return_value = llm_response - - # Test memory reranking - result = self.scheduler.replace_working_memory( - original_memory, new_memory, top_k=2, top_n=1 - ) - - # Verify result - self.assertEqual(len(result), 1) - self.assertEqual(result[0].memory, "New 2") + # Get the actual message from the queue + actual_message = self.scheduler._web_log_message_queue.get() + + # Verify core fields + self.assertEqual(actual_message.user_id, "test_user") + self.assertEqual(actual_message.mem_cube_id, "test_cube") + self.assertEqual(actual_message.label, QUERY_LABEL) + self.assertEqual(actual_message.from_memory_type, "WorkingMemory") + self.assertEqual(actual_message.to_memory_type, "LongTermMemory") + self.assertEqual(actual_message.log_content, "Test Content") + + # Verify memory sizes + self.assertEqual(actual_message.current_memory_sizes["long_term_memory_size"], 0) + self.assertEqual(actual_message.current_memory_sizes["user_memory_size"], 0) + self.assertEqual(actual_message.current_memory_sizes["working_memory_size"], 0) + self.assertEqual(actual_message.current_memory_sizes["transformed_act_memory_size"], 0) + + # Verify memory capacities + self.assertEqual(actual_message.memory_capacities["long_term_memory_capacity"], 1000) + self.assertEqual(actual_message.memory_capacities["user_memory_capacity"], 500) + self.assertEqual(actual_message.memory_capacities["working_memory_capacity"], 100) + self.assertEqual(actual_message.memory_capacities["transformed_act_memory_capacity"], 0) + + # Verify auto-generated fields exist + self.assertTrue(hasattr(actual_message, "item_id")) + self.assertTrue(isinstance(actual_message.item_id, str)) + self.assertTrue(hasattr(actual_message, "timestamp")) + self.assertTrue(isinstance(actual_message.timestamp, datetime)) def test_search_with_empty_results(self): """Test search method with empty results.""" - # Setup mock search results - self.tree_text_memory.search.return_value = [] + # Setup mock memory cube and text memory + mock_mem_cube = MagicMock() + mock_mem_cube.text_mem = self.tree_text_memory + + # Setup mock search results for both memory types + self.tree_text_memory.search.side_effect = [ + [], # results_long_term + [], # results_user + ] # Test search - results = self.scheduler.search( - query="Test query", top_k=5, method=TextMemory_SEARCH_METHOD + results = self.scheduler.retriever.search( + query="Test query", mem_cube=mock_mem_cube, top_k=5, method=TreeTextMemory_SEARCH_METHOD ) # Verify results self.assertEqual(results, []) - def test_multiple_messages_processing(self): - """Test processing of multiple messages in a batch.""" - # Create multiple test messages - query_message = ScheduleMessageItem( - user_id="test_user", - mem_cube_id="test_cube", - mem_cube=self.mem_cube, - label=QUERY_LABEL, - content="Query 1", + # Verify search was called twice (for LongTermMemory and UserMemory) + self.assertEqual(self.tree_text_memory.search.call_count, 2) + self.tree_text_memory.search.assert_any_call( + query="Test query", top_k=5, memory_type="LongTermMemory" ) - answer_message = ScheduleMessageItem( - user_id="test_user", - mem_cube_id="test_cube", - mem_cube=self.mem_cube, - label=ANSWER_LABEL, - content="Answer 1", + self.tree_text_memory.search.assert_any_call( + query="Test query", top_k=5, memory_type="UserMemory" ) - - # Mock message handlers - with ( - patch.object(self.scheduler, "_query_message_consume") as mock_query, - patch.object(self.scheduler, "_answer_message_consume") as mock_answer, - ): - # Ensure message handlers are registered - self.scheduler.dispatcher.register_handlers( - { - QUERY_LABEL: self.scheduler._query_message_consume, - ANSWER_LABEL: self.scheduler._answer_message_consume, - } - ) - - # Process messages - self.scheduler.dispatcher.enable_parallel_dispatch = False - self.scheduler.dispatcher.dispatch([query_message, answer_message]) - - # Verify call arguments manually - mock_query.assert_called_once_with([query_message]) - mock_answer.assert_called_once_with([answer_message]) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/memories/textual/test_general.py b/tests/memories/textual/test_general.py index 7019b1216..b5a6b0126 100644 --- a/tests/memories/textual/test_general.py +++ b/tests/memories/textual/test_general.py @@ -98,7 +98,7 @@ def test_embed_one_sentence(self): embedding = self.memory._embed_one_sentence(sentence) - self.mock_embedder.embed.assert_called_once_with(sentence) + self.mock_embedder.embed.assert_called_once_with([sentence]) self.assertEqual(embedding, expected_embedding) def test_extract(self): @@ -206,7 +206,7 @@ def test_update_memory(self): self.memory.update(memory_id_to_update, new_memory_dict) self.mock_embedder.embed.assert_called_once_with( - "This is the updated memory content via dict." + ["This is the updated memory content via dict."] ) args, _ = self.mock_vector_db.update.call_args @@ -271,7 +271,7 @@ def test_search_memories(self): search_results = self.memory.search(query, top_k) - self.mock_embedder.embed.assert_called_once_with(query) + self.mock_embedder.embed.assert_called_once_with([query]) self.mock_vector_db.search.assert_called_once_with(query_embedding, top_k) self.assertEqual(len(search_results), top_k) diff --git a/tests/memories/textual/test_tree.py b/tests/memories/textual/test_tree.py index 356217945..f3e662992 100644 --- a/tests/memories/textual/test_tree.py +++ b/tests/memories/textual/test_tree.py @@ -138,3 +138,27 @@ def test_drop_creates_backup_and_cleans(mock_tree_text_memory): mock_tree_text_memory.dump.assert_called_once() mock_tree_text_memory._cleanup_old_backups.assert_called_once() mock_tree_text_memory.graph_store.drop_database.assert_called_once() + + +def test_add_returns_ids(mock_tree_text_memory): + # Mock the memory_manager.add to return specific IDs + dummy_ids = ["id1", "id2"] + mock_tree_text_memory.memory_manager.add = MagicMock(return_value=dummy_ids) + + mock_items = [ + TextualMemoryItem( + id=str(uuid.uuid4()), + memory="Memory 1", + metadata=TreeNodeTextualMemoryMetadata(updated_at=None), + ), + TextualMemoryItem( + id=str(uuid.uuid4()), + memory="Memory 2", + metadata=TreeNodeTextualMemoryMetadata(updated_at=None), + ), + ] + + result = mock_tree_text_memory.add(mock_items) + + assert result == dummy_ids + mock_tree_text_memory.memory_manager.add.assert_called_once_with(mock_items) diff --git a/tests/memories/textual/test_tree_manager.py b/tests/memories/textual/test_tree_manager.py index e885d0231..2a97e185c 100644 --- a/tests/memories/textual/test_tree_manager.py +++ b/tests/memories/textual/test_tree_manager.py @@ -145,3 +145,14 @@ def test_ensure_structure_path_reuses_existing(memory_manager, mock_graph_store) meta = TreeNodeTextualMemoryMetadata(key="hobby") node_id = memory_manager._ensure_structure_path("UserMemory", meta) assert node_id == "existing_node_id" + + +def test_add_returns_written_node_ids(memory_manager): + memory = TextualMemoryItem( + memory="test memory", + metadata=TreeNodeTextualMemoryMetadata(embedding=[0.1] * 5, memory_type="UserMemory"), + ) + ids = memory_manager.add([memory]) + assert isinstance(ids, list) + assert all(isinstance(i, str) for i in ids) + assert len(ids) > 0 diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 000000000..9750af121 --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,106 @@ +""" +Tests for the MemOS CLI tool. +""" + +import zipfile + +from io import BytesIO +from unittest.mock import MagicMock, mock_open, patch + +import pytest +import requests + +from memos.cli import download_examples, export_openapi, main + + +class TestExportOpenAPI: + """Test the export_openapi function.""" + + @patch("memos.api.start_api.app") + @patch("builtins.open", new_callable=mock_open) + @patch("os.makedirs") + def test_export_openapi_success(self, mock_makedirs, mock_file, mock_app): + """Test successful OpenAPI export.""" + mock_openapi_data = {"openapi": "3.0.0", "info": {"title": "Test API"}} + mock_app.openapi.return_value = mock_openapi_data + + result = export_openapi("/test/path/openapi.json") + + assert result is True + mock_makedirs.assert_called_once_with("/test/path", exist_ok=True) + mock_file.assert_called_once_with("/test/path/openapi.json", "w") + + @patch("memos.api.start_api.app") + @patch("builtins.open", side_effect=OSError("Permission denied")) + def test_export_openapi_error(self, mock_file, mock_app): + """Test OpenAPI export when file writing fails.""" + mock_app.openapi.return_value = {"test": "data"} + + with pytest.raises(IOError): + export_openapi("/invalid/path/openapi.json") + + +class TestDownloadExamples: + """Test the download_examples function.""" + + def create_mock_zip_content(self): + """Create mock zip file content for testing.""" + zip_buffer = BytesIO() + with zipfile.ZipFile(zip_buffer, "w") as zip_file: + zip_file.writestr("MemOS-main/examples/test_example.py", "# Test example content") + zip_file.writestr( + "MemOS-main/examples/subfolder/another_example.py", "# Another example" + ) + return zip_buffer.getvalue() + + @patch("requests.get") + @patch("os.makedirs") + @patch("builtins.open", new_callable=mock_open) + def test_download_examples_success(self, mock_file, mock_makedirs, mock_requests): + """Test successful examples download.""" + mock_response = MagicMock() + mock_response.content = self.create_mock_zip_content() + mock_requests.return_value = mock_response + + result = download_examples("/test/dest") + + assert result is True + mock_requests.assert_called_once_with( + "https://github.com/MemTensor/MemOS/archive/refs/heads/main.zip" + ) + mock_response.raise_for_status.assert_called_once() + + @patch("requests.get") + def test_download_examples_error(self, mock_requests): + """Test download examples when request fails.""" + mock_requests.side_effect = requests.RequestException("Network error") + + result = download_examples("/test/dest") + + assert result is False + + +class TestMainCLI: + """Test the main CLI function.""" + + @patch("memos.cli.download_examples") + def test_main_download_examples(self, mock_download): + """Test main function with download_examples command.""" + mock_download.return_value = True + + with patch("sys.argv", ["memos", "download_examples", "--dest", "/test/dest"]): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + mock_download.assert_called_once_with("/test/dest") + + @patch("memos.cli.export_openapi") + def test_main_export_openapi(self, mock_export): + """Test main function with export_openapi command.""" + mock_export.return_value = True + + with patch("sys.argv", ["memos", "export_openapi", "--output", "/test/openapi.json"]): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == 0 + mock_export.assert_called_once_with("/test/openapi.json") diff --git a/tests/vec_dbs/test_qdrant.py b/tests/vec_dbs/test_qdrant.py index c2eabd218..828240ae1 100644 --- a/tests/vec_dbs/test_qdrant.py +++ b/tests/vec_dbs/test_qdrant.py @@ -28,7 +28,7 @@ def config(): @pytest.fixture def mock_qdrant_client(): - with patch("memos.vec_dbs.qdrant.QdrantClient") as mockclient: + with patch("qdrant_client.QdrantClient") as mockclient: yield mockclient