Skip to content

Performance Improvements: Memory Leak Fixes, Label Indexing, and Optimized Edge & Buffer Handling#317

Closed
ParameswaranSajeenthiran wants to merge 93 commits intomiyurud:masterfrom
ParameswaranSajeenthiran:hdfs-large-file-upload
Closed

Performance Improvements: Memory Leak Fixes, Label Indexing, and Optimized Edge & Buffer Handling#317
ParameswaranSajeenthiran wants to merge 93 commits intomiyurud:masterfrom
ParameswaranSajeenthiran:hdfs-large-file-upload

Conversation

@ParameswaranSajeenthiran
Copy link
Collaborator

Description

This PR introduces several key performance and memory optimizations to improve system efficiency and scalability:

  • Memory Leak Fixes:

    • Identified and removed memory leaks in critical components to ensure stable long-term operation and prevent unnecessary memory consumption.
  • Label Indexing:

    • Added efficient label indexing for faster node/edge lookups and reduced query latency.
  • Optimized Shared Buffer Usage:

    • Removed redundant lock waits in the single-threaded use of the shared buffer by introducing partition-wise producer-consumer handling.
    • This change minimizes thread contention and improves throughput for each partition.
  • Edge Storage Optimization:

    • Removed redundant bidirectional edge cut storage, reducing both memory usage and I/O overhead without affecting query correctness.

Impact

  • Performance: Reduced lock contention and improved buffer throughput.
  • Memory: Lower memory footprint due to elimination of leaks and unnecessary edge storage.
  • Scalability: Faster query execution with label indexing and optimized data structures.

Testing

  • Verified memory stability through long-running stress tests.
  • Benchmarked query performance showing reduced latency after label indexing.
  • Validated correctness of producer-consumer operations and graph queries after removing redundant storage.


Related Issues

@sonarqubecloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
26 Security Hotspots
15.9% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments