[Feature] Add BlockNote editor #500
Conversation
|
@AnishSarkar22 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on 70f3381..abbaa84
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (23)
• surfsense_backend/alembic/versions/38_add_blocknote_fields_to_documents.py
• surfsense_backend/app/db.py
• surfsense_backend/app/routes/__init__.py
• surfsense_backend/app/routes/editor_routes.py
• surfsense_backend/app/tasks/document_processors/extension_processor.py
• surfsense_backend/app/tasks/document_processors/file_processors.py
• surfsense_backend/app/tasks/document_processors/markdown_processor.py
• surfsense_backend/app/tasks/document_processors/url_crawler.py
• surfsense_backend/app/tasks/document_processors/youtube_processor.py
• surfsense_backend/app/utils/blocknote_converter.py
• surfsense_web/app/api/convert-to-blocknote/route.ts
• surfsense_web/app/api/convert-to-markdown/route.ts
• surfsense_web/app/dashboard/[search_space_id]/documents/(manage)/components/DocumentsTableShell.tsx
• surfsense_web/app/dashboard/[search_space_id]/documents/(manage)/components/RowActions.tsx
• surfsense_web/app/dashboard/[search_space_id]/editor/[documentId]/page.tsx
• surfsense_web/components/BlockNoteEditor.tsx
• surfsense_web/components/DynamicBlockNoteEditor.tsx
• surfsense_web/components/dashboard-breadcrumb.tsx
• surfsense_web/messages/en.json
• surfsense_web/messages/zh.json
• surfsense_web/next.config.ts
• surfsense_web/package.json
• surfsense_web/pnpm-lock.yaml
|
@MODSetter I am unsure whether we should update documents with empty |
AnishSarkar22
changed the title
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on abbaa84..289b4de
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (17)
• surfsense_backend/alembic/versions/38_add_blocknote_fields_to_documents.py
• surfsense_backend/app/db.py
• surfsense_backend/app/routes/editor_routes.py
• surfsense_backend/app/tasks/document_processors/extension_processor.py
• surfsense_backend/app/tasks/document_processors/file_processors.py
• surfsense_backend/app/tasks/document_processors/markdown_processor.py
• surfsense_backend/app/tasks/document_processors/url_crawler.py
• surfsense_backend/app/tasks/document_processors/youtube_processor.py
• surfsense_backend/app/utils/blocknote_converter.py
• surfsense_web/app/api/convert-to-blocknote/route.ts
• surfsense_web/app/api/convert-to-markdown/route.ts
• surfsense_web/app/dashboard/[search_space_id]/documents/(manage)/components/RowActions.tsx
• surfsense_web/app/dashboard/[search_space_id]/editor/[documentId]/page.tsx
• surfsense_web/components/BlockNoteEditor.tsx
• surfsense_web/components/DynamicBlockNoteEditor.tsx
• surfsense_web/components/dashboard-breadcrumb.tsx
• surfsense_web/next.config.ts
- Updated the document saving endpoint to trigger reindexing after saving. - Introduced a new Celery task for reindexing documents. - Refactored the editor page to reflect the changes in the API endpoint and method.
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on 289b4de..e419702
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (50)
• README.md
• README.zh-CN.md
• surfsense_backend/alembic/versions/36_remove_fk_constraints_for_global_llm_configs.py
• surfsense_backend/alembic/versions/37_add_system_prompts_to_searchspaces.py
• surfsense_backend/alembic/versions/38_add_blocknote_fields_to_documents.py
• surfsense_backend/alembic/versions/38_add_webcrawler_connector_enum.py
• surfsense_backend/alembic/versions/39_add_rbac_tables.py
• surfsense_backend/alembic/versions/40_move_llm_preferences_to_searchspace.py
• surfsense_backend/alembic/versions/41_backfill_rbac_for_existing_searchspaces.py
• surfsense_backend/alembic/versions/42_drop_user_search_space_preferences.py
• surfsense_backend/app/agents/researcher/nodes.py
• surfsense_backend/app/agents/researcher/qna_agent/configuration.py
• surfsense_backend/app/agents/researcher/qna_agent/default_prompts.py
• surfsense_backend/app/agents/researcher/qna_agent/nodes.py
• surfsense_backend/app/agents/researcher/utils.py
• surfsense_backend/app/celery_app.py
• surfsense_backend/app/config/__init__.py
• surfsense_backend/app/connectors/webcrawler_connector.py
• surfsense_backend/app/db.py
• surfsense_backend/app/retriver/chunks_hybrid_search.py
• surfsense_backend/app/retriver/documents_hybrid_search.py
• surfsense_backend/app/routes/__init__.py
• surfsense_backend/app/routes/chats_routes.py
• surfsense_backend/app/routes/documents_routes.py
• surfsense_backend/app/routes/editor_routes.py
• surfsense_backend/app/routes/llm_config_routes.py
• surfsense_backend/app/routes/logs_routes.py
• surfsense_backend/app/routes/podcasts_routes.py
• surfsense_backend/app/routes/rbac_routes.py
• surfsense_backend/app/routes/search_source_connectors_routes.py
• surfsense_backend/app/routes/search_spaces_routes.py
• surfsense_backend/app/schemas/__init__.py
• surfsense_backend/app/schemas/rbac_schemas.py
• surfsense_backend/app/schemas/search_space.py
• surfsense_backend/app/services/connector_service.py
• surfsense_backend/app/services/llm_service.py
• surfsense_backend/app/services/query_service.py
• surfsense_backend/app/tasks/celery_tasks/blocknote_migration_tasks.py
• surfsense_backend/app/tasks/celery_tasks/connector_tasks.py
• surfsense_backend/app/tasks/celery_tasks/document_reindex_tasks.py
• surfsense_backend/app/tasks/celery_tasks/document_tasks.py
• surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py
• surfsense_backend/app/tasks/connector_indexers/__init__.py
• surfsense_backend/app/tasks/connector_indexers/webcrawler_indexer.py
• surfsense_backend/app/tasks/document_processors/__init__.py
• surfsense_backend/app/tasks/document_processors/file_processors.py
• surfsense_backend/app/tasks/document_processors/url_crawler.py
• surfsense_backend/app/utils/check_ownership.py
• surfsense_backend/app/utils/periodic_scheduler.py
• surfsense_backend/app/utils/rbac.py
⏭️ Files skipped (38)
| Locations |
|---|
surfsense_backend/app/utils/validators.py |
surfsense_backend/pyproject.toml |
surfsense_backend/uv.lock |
surfsense_browser_extension/package.json |
surfsense_browser_extension/pnpm-lock.yaml |
surfsense_web/app/dashboard/[search_space_id]/client-layout.tsx |
surfsense_web/app/dashboard/[search_space_id]/connectors/[connector_id]/edit/page.tsx |
surfsense_web/app/dashboard/[search_space_id]/connectors/[connector_id]/page.tsx |
surfsense_web/app/dashboard/[search_space_id]/connectors/add/webcrawler-connector/page.tsx |
surfsense_web/app/dashboard/[search_space_id]/documents/webpage/page.tsx |
surfsense_web/app/dashboard/[search_space_id]/editor/[documentId]/page.tsx |
surfsense_web/app/dashboard/[search_space_id]/layout.tsx |
surfsense_web/app/dashboard/[search_space_id]/logs/(manage)/page.tsx |
surfsense_web/app/dashboard/[search_space_id]/sources/add/page.tsx |
surfsense_web/app/dashboard/[search_space_id]/team/page.tsx |
surfsense_web/app/dashboard/page.tsx |
surfsense_web/app/invite/[invite_code]/page.tsx |
surfsense_web/components/chat/ChatInputGroup.tsx |
surfsense_web/components/dashboard-breadcrumb.tsx |
surfsense_web/components/editConnector/types.ts |
surfsense_web/components/homepage/integrations.tsx |
surfsense_web/components/onboard/completion-step.tsx |
surfsense_web/components/pricing/pricing-section.tsx |
surfsense_web/components/settings/llm-role-manager.tsx |
surfsense_web/components/sidebar/app-sidebar.tsx |
surfsense_web/components/sidebar/nav-main.tsx |
surfsense_web/components/sources/ConnectorsTab.tsx |
surfsense_web/components/sources/connector-data.tsx |
surfsense_web/content/docs/docker-installation.mdx |
surfsense_web/contracts/enums/connector.ts |
surfsense_web/contracts/enums/connectorIcons.tsx |
surfsense_web/hooks/index.ts |
surfsense_web/hooks/use-connector-edit-page.ts |
surfsense_web/hooks/use-rbac.ts |
surfsense_web/hooks/use-search-spaces.ts |
surfsense_web/lib/connectors/utils.ts |
surfsense_web/messages/en.json |
surfsense_web/messages/zh.json |
…s for blocknote in `file_processors.py` file
|
@MODSetter Blocknote editor is complete I think, please merge it if there are no issues. |
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on e419702..f92112a
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (6)
• surfsense_backend/alembic/versions/43_add_blocknote_fields_to_documents.py
• surfsense_backend/app/routes/__init__.py
• surfsense_backend/app/routes/editor_routes.py
• surfsense_backend/app/tasks/celery_tasks/blocknote_migration_tasks.py
• surfsense_backend/app/tasks/celery_tasks/document_reindex_tasks.py
• surfsense_backend/app/tasks/document_processors/file_processors.py
|
@AnishSarkar22 Awesome work man. I love it. Few issues but I can handle them from here. |
Thank you. Let me know if you run into anything, I am always happy to help. |
2 participants
Added BlockNote editor integration with comprehensive document reindexing system. This PR enables rich text editing of documents with automatic background reindexing to keep search results synchronized, plus migration support to convert existing documents to editable format.
Description
This PR adds a complete BlockNote editor integration with intelligent reindexing to SurfSense:
Frontend Changes:
BlockNoteEditorcomponent using@blocknote/reactand@blocknote/mantinewith theme supportDynamicBlockNoteEditorwrapper for proper SSR handling/dashboard/[search_space_id]/editor/[documentId]) with:Backend Changes:
New API Endpoints:
GET /api/v1/documents/{document_id}/editor-content- Fetches document for editing with lazy migration supportPOST /api/v1/documents/{document_id}/save- Saves document and triggers background reindexingDatabase Changes:
38_add_blocknote_fields_to_documents.pywith three new columns:blocknote_document(JSONB) - Stores BlockNote JSON structurecontent_needs_reindexing(Boolean) - Flags documents needing reindexlast_edited_at(Timestamp) - Tracks last edit timeCelery Background Tasks:
reindex_document_task- Handles post-edit reindexing:populate_blocknote_for_documents_task- Migration task:Lazy Migration Support:
blocknote_documentare auto-converted on first accessBlockNote Converter Utilities:
convert_markdown_to_blocknote()- Markdown → BlockNote JSON (via Next.js API)convert_blocknote_to_markdown()- BlockNote JSON → Markdown (via Next.js API)Updated Document Processors:
nullfor pre-migration simulationReindexing Architecture:
Migration Strategy:
Motivation and Context
This change enables users to edit documents directly within SurfSense while maintaining search accuracy. Key improvements:
Technical Decisions:
Screenshots
API Changes
Change Type
Testing Performed
Checklist
High-level PR Summary
This PR introduces a rich text editing capability to SurfSense by integrating the BlockNote editor. Users can now edit documents directly in the browser with a modern block-based interface. The implementation includes database schema changes to store BlockNote JSON documents, new API endpoints for fetching and saving editor content, and automatic conversion of markdown to BlockNote format during document processing. The frontend features a complete editor page with auto-save functionality every 30 seconds, theme support (dark/light), and unsaved changes tracking. All document processors have been updated to generate BlockNote-compatible content on upload, though legacy documents will require re-uploading to enable editing.
⏱️ Estimated Review Time: 3+ hours
💡 Review Order Suggestion
surfsense_backend/alembic/versions/38_add_blocknote_fields_to_documents.pysurfsense_backend/app/db.pysurfsense_backend/app/utils/blocknote_converter.pysurfsense_backend/app/routes/editor_routes.pysurfsense_backend/app/routes/__init__.pysurfsense_backend/app/tasks/document_processors/extension_processor.pysurfsense_backend/app/tasks/document_processors/file_processors.pysurfsense_backend/app/tasks/document_processors/markdown_processor.pysurfsense_backend/app/tasks/document_processors/url_crawler.pysurfsense_backend/app/tasks/document_processors/youtube_processor.pysurfsense_web/app/api/convert-to-blocknote/route.tssurfsense_web/app/api/convert-to-markdown/route.tssurfsense_web/components/BlockNoteEditor.tsxsurfsense_web/components/DynamicBlockNoteEditor.tsxsurfsense_web/app/dashboard/[search_space_id]/editor/[documentId]/page.tsxsurfsense_web/app/dashboard/[search_space_id]/documents/(manage)/components/RowActions.tsxsurfsense_web/app/dashboard/[search_space_id]/documents/(manage)/components/DocumentsTableShell.tsxsurfsense_web/components/dashboard-breadcrumb.tsxsurfsense_web/messages/en.jsonsurfsense_web/messages/zh.jsonsurfsense_web/next.config.tssurfsense_web/package.jsonsurfsense_web/pnpm-lock.yamlsurfsense_web/next.config.tsreactStrictMode: false) is a significant configuration change that affects the entire Next.js application beyond just the BlockNote editor feature. This could hide potential issues in other parts of the codebase and should be carefully considered.