refactor(database): Refactor database connection management to avoid sqlite database is locked #1303
Conversation
…t read/write separation This commit introduces a new `Connection` struct to abstract underlying database read/write connections. Multiple files were modified to replace the original `*gorm.DB` instances with the new `Connection` instances, and related function calls were adjusted to accommodate the new connection management approach. Key changes include: - Added `Connection` struct and its methods in `internal/model/connection.go` to implement read/write connection separation - Modified database initialization logic in `internal/bootstrap/db.go`, `internal/db/db.go` etc. to use new `Connection` instances instead of original `*gorm.DB` instances - Updated database operation functions in `internal/db/meta.go`, `internal/db/searchnode.go` and other files to properly use `Connection` instances for read/write operations - Added new files `internal/bootstrap/dbengine/mysql.go`, `internal/bootstrap/dbengine/postgres.go` and `internal/bootstrap/dbengine/sqlite.go` implementing connection creation logic for MySQL, PostgreSQL and SQLite respectively These changes improve code maintainability and extensibility while laying the foundation for future performance optimizations.
PaienNate
changed the title
There was a problem hiding this comment.
Pull Request Overview
This PR refactors database connection management to optimize SQLite performance and resolve "database is locked" errors by implementing separate read and write connections based on SQLite best practices from "SQLite For Server" guidance.
- Introduces a new
Connectionstruct that abstracts read/write database access - Implements optimized SQLite connection management with separate read/write connections and performance pragmas
- Refactors all database operations to use the new connection abstraction pattern
Reviewed Changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/model/connection.go | New Connection struct providing read/write database abstraction |
| internal/bootstrap/dbengine/*.go | Database engine implementations for SQLite, MySQL, and PostgreSQL with optimized connection settings |
| internal/bootstrap/db.go | Updated database initialization to use new connection management |
| internal/db/db.go | Refactored to use Connection struct instead of single gorm.DB instance |
| internal/db/*.go | Updated all database operations to use rwDb.R() for reads and rwDb.W() for writes |
| internal/search/db/init.go | Updated search indexing to use new connection pattern |
| internal/op/storage_test.go | Updated test to use new SQLite connection creation |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
已确认并修复了Copilot查出的所有问题。AI查的很准确。
|
refactor(database): 基于SQLite For Server对数据库代码进行调优
Description / 描述
该PR旨在基于 https://kerkour.com/sqlite-for-servers 的方案，以解决sqlite下 database is locked的问题。
它参考了 https://github.com/bihe/monorepo/blob/477e534bd4c0814cdca73fea774b518148cebd3f/pkg/persistence/sqlite.go#L59 并做出部分修改，以适配该项目。它同时兼容了litestream对其进行备份（来源monorepo库，该库使用apache-2.0开源协议）。
Motivation and Context / 背景
在另外一个开源项目（sealdice-core）中由于使用SQLite时容易发生database is locked的情况，在调研后在对应项目予以修复。观察到OpenList有相同问题，故而移植代码核心方案到OpenList。
解决问题的核心：
How Has This Been Tested? / 测试
该方案在另外一个开源项目中测试过可行。 但不幸的是，我确实没有对我的代码进行令人满意的测试。我仅仅运行了storage_test并通过了测试，但我想这远远不够。
Checklist / 检查清单
我已阅读 CONTRIBUTING 文档。
go fmtor prettier.我已使用
go fmt或 prettier 格式化提交的代码。我已为此 PR 添加了适当的标签（如无权限或需要的标签不存在，请在描述中说明，管理员将后续处理）。
我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
我已相应更新了相关仓库（若适用）。