Skip to content

Conversation

@Nov1c444
Copy link
Contributor

@Nov1c444 Nov1c444 commented Aug 29, 2025

Summary

optimize SQL queries that perform partial full table scans
Fix #24784

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Aug 29, 2025
@Nov1c444 Nov1c444 marked this pull request as draft August 29, 2025 09:16
@Nov1c444 Nov1c444 force-pushed the chore/full-table-sql-optimize branch from d21c320 to ffd6fd1 Compare August 29, 2025 09:17
@Nov1c444 Nov1c444 requested a review from QuantumGhost August 29, 2025 09:18
@Nov1c444 Nov1c444 marked this pull request as ready for review August 29, 2025 09:18
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Aug 29, 2025
@crazywoola crazywoola requested a review from Copilot August 29, 2025 10:01
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes SQL queries by addressing partial full table scan issues through database indexing improvements and caching strategies to reduce database load. The changes focus on preventing performance bottlenecks caused by frequent database operations.

  • Added a database index on workflow_run_id column for the workflow_app_logs table
  • Implemented Redis-based caching for provider last used timestamps to reduce database update frequency
  • Optimized database deletion operations by using compound indexes and disabling session synchronization

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
api/models/workflow.py Added database index on workflow_run_id column to optimize query performance
api/migrations/versions/2025_08_29_1534-b95962a3885c_add_workflow_app_log_run_id_index.py Database migration to create the new workflow run ID index
api/events/event_handlers/update_provider_when_message_created.py Implemented Redis caching for provider updates with time-window based updates
api/core/rag/index_processor/processor/parent_child_index_processor.py Optimized deletion queries by using compound indexes and disabling session synchronization
api/core/app/apps/advanced_chat/generate_task_pipeline.py Removed message event emission to reduce database operations

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

crazywoola
crazywoola previously approved these changes Aug 31, 2025
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Aug 31, 2025
Copy link
Collaborator

@QuantumGhost QuantumGhost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@QuantumGhost QuantumGhost merged commit ca96350 into main Sep 2, 2025
14 checks passed
@QuantumGhost QuantumGhost deleted the chore/full-table-sql-optimize branch September 2, 2025 03:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Chore/Refactor] Optimize SQL queries that perform full table scans

4 participants