feat: add SillyTavern worldbook import#216
Merged
EterUltimate merged 3 commits intoJun 18, 2026
Merged
Conversation
Contributor
Reviewer's GuideAdds a SillyTavern worldbook JSON importer wired into existing Self Learning review/jargon/knowledge-graph tables, with new WebUI integration endpoints, documentation, and tests for parsing, import behavior, and HTTP routes. Sequence diagram for SillyTavern worldbook import flowsequenceDiagram
actor Client
participant IntegrationsBP as integrations_bp
participant WorldBookImporter
participant DB as database_manager
Client->>IntegrationsBP: POST /integrations/worldbook/preview
IntegrationsBP->>WorldBookImporter: preview(payload, json_text, json_path)
WorldBookImporter->>WorldBookImporter: load_package(...)
WorldBookImporter->>WorldBookImporter: package_summary(package)
WorldBookImporter-->>IntegrationsBP: summary
IntegrationsBP-->>Client: 200 { success, data }
Client->>IntegrationsBP: POST /integrations/worldbook/import
IntegrationsBP->>DB: get_session()
IntegrationsBP->>WorldBookImporter: import_from_source(payload, json_text, json_path, ...)
WorldBookImporter->>WorldBookImporter: load_package(...)
WorldBookImporter->>WorldBookImporter: import_package(package, ...)
loop for each entry
WorldBookImporter->>WorldBookImporter: _entry_review_exists(...)
alt import_memories
WorldBookImporter->>DB: _add_memory_review(session, package, entry, ...)
end
alt import_jargons
WorldBookImporter->>DB: _import_jargon_candidates(session, package, entry, ...)
end
alt import_knowledge_graph
WorldBookImporter->>DB: _import_knowledge_graph(session, package, entry, ...)
end
end
WorldBookImporter-->>IntegrationsBP: { success, entries_imported, ... }
IntegrationsBP-->>Client: 200 { success, data }
Client->>IntegrationsBP: GET /integrations/worldbook/imports
IntegrationsBP->>DB: get_session()
IntegrationsBP->>WorldBookImporter: import_history(limit, offset)
WorldBookImporter->>DB: select PersonaLearningReview
WorldBookImporter-->>IntegrationsBP: { total, items, imports }
IntegrationsBP-->>Client: 200 { success, data }
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The
json_path/worldbook_pathhandling in_worldbook_source_argsandWorldBookImporter.load_packagewill read arbitrary server-side paths supplied by the caller; consider restricting this to a known base directory or explicitly disablingjson_pathfor WebUI-driven requests to avoid unintended file disclosure. worldbook_importer.pycurrently mixes parsing/normalization logic and persistence concerns in a single, ~800-line module; consider splitting out the pure parsing layer (e.g.,WorldBookPackageconstruction and helpers) from the DB import layer to keep each unit smaller and easier to reason about and reuse.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The `json_path`/`worldbook_path` handling in `_worldbook_source_args` and `WorldBookImporter.load_package` will read arbitrary server-side paths supplied by the caller; consider restricting this to a known base directory or explicitly disabling `json_path` for WebUI-driven requests to avoid unintended file disclosure.
- `worldbook_importer.py` currently mixes parsing/normalization logic and persistence concerns in a single, ~800-line module; consider splitting out the pure parsing layer (e.g., `WorldBookPackage` construction and helpers) from the DB import layer to keep each unit smaller and easier to reason about and reuse.
## Individual Comments
### Comment 1
<location path="services/integration/worldbook_importer.py" line_range="126" />
<code_context>
+ source_paths["worldbook_json"] = str(path.resolve())
+ if isinstance(payload, str):
+ payload = _json_decode(payload)
+ if not isinstance(payload, Mapping):
+ raise ValueError("请提供 SillyTavern 世界书 JSON 对象或 JSON 字符串")
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Top-level list payloads can never reach the list-handling branch in `_parse_sillytavern_payload`.
Because `load_package` rejects any non-mapping payloads, `_parse_sillytavern_payload` will only ever receive a `Mapping`. The branch that tries to handle a top-level `list` (`if entries_payload is None and isinstance(payload, list):`) is therefore dead code. If SillyTavern is supposed to export a bare `list`, this path will never be taken and a `ValueError` will be raised instead; if that case is impossible, the list-handling logic should be removed. Please either allow `list` payloads in `load_package` or simplify `_parse_sillytavern_payload` to only handle mappings.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Collaborator
Author
|
Addressed review feedback in 13249cc:
Validation:
|
Collaborator
Author
|
Added release bump in 33d0f85:
All PR checks are passing; merge is currently waiting on required maintainer review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WorldBookImporterfor SillyTavern worldbook JSON preview/import/history.entriesas object or array, normalizekey/secondaryKeys, and preserve ST metadata.Issue Evidence
services/integration/worldbook_importer.py, compatibility withentriesobject/list, mappingcontentto review/KG andkey+secondaryKeysto jargon, plus preview/import/imports endpoints._inject_worldbook_contextis left for a follow-up because it needs config, scan-depth, and token-budget behavior wired intoservices/hooks/llm_hook_handler.pysafely.Tests
python -m pytest tests\unit\test_worldbook_importer.py -qpython -m pytest tests\integration\test_worldbook_integration_routes.py -qpython -m ruff check services\integration\worldbook_importer.py webui\blueprints\integrations.py tests\unit\test_worldbook_importer.py tests\integration\test_worldbook_integration_routes.pypython -m py_compile services\integration\worldbook_importer.py webui\blueprints\integrations.pygit diff --checkInstall Smoke Test
Could not complete the requested install smoke test under
C:\Users\zacza\Desktop\x\AstrBot\data\plugins\astrbot_plugin_self_learning: the directory returnsAccess is deniedeven forTest-Path,Get-ChildItem,icacls, andgit -C. I did not change ACLs or force ownership.Summary by Sourcery
Add support for importing SillyTavern worldbook JSON into the Self Learning plugin and expose corresponding WebUI APIs and documentation.
New Features:
Enhancements:
Documentation:
Tests: