fix: Investigate and Resolve Slow Fetching from Backend API

[JustinhSE]

Overview

The versify app is experiencing significant delays or failures when fetching k-means cluster results from our Python backend API. This issue is causing poor user experience and needs to be addressed urgently. We need to investigate the entire pipeline from the frontend request to the backend processing and response to identify and resolve bottlenecks.

Tasks

1. Analyze Frontend API Call Implementation

   • Review the existing code for API calls to the Python endpoint
   • Check for proper error handling and timeout settings
   • Verify if requests are being made efficiently (e.g., not over-fetching)

2. Backend API Performance Analysis

   • Investigate the Python backend to identify slow operations
   • Check if the k-means algorithm implementation is optimized

3. Caching Strategy (up for discussion)

   • Implement caching for frequently requested k-means results

[JustinhSE]

Since we will be moving to a larger dataset, pickle files won’t be good enough for the clusters as pickle files would load the entire file, reducing efficiency. We might want to consider Vector Databases.
Use Case:
• Ideal for applications that require similarity searches, such as those involving natural language processing (NLP), recommendation systems, or any task where semantic similarity is important.
Advantages:
• Efficient retrieval of high-dimensional data.
• Optimized for handling large volumes of vector data with fast query performance.
• Scales horizontally by adding more servers to a cluster, which is beneficial for large datasets.
Examples: Pinecone, Milvus

open to ideas but wanted to drop this here.

[JustinhSE]

Hold on this issue - Just changed from TF-IDF to sentence transformers, unsure if it takes as long. Deployment coming soon