Feature/simple crawler#3

Open

dinethjanitha wants to merge 5 commits intomainfrom

feature/simple-crawler

Commits on Nov 10, 2025

Refactor WebCrawSpider and googlesearch function: streamline URL processing, enhance social media link extraction, and improve logging for crawled data.
dinethjanitha
committed

Commits on Nov 12, 2025

Update WebCrawSpider settings: reduce CLOSESPIDER_PAGECOUNT and max_pages to 3 for stricter crawling limits.
dinethjanitha
committed
Refactor logging in getCrawlContent and related functions: remove emojis and adjust formatting for improved readability.
dinethjanitha
committed

Commits on Nov 26, 2025

Refactor code structure for improved readability and maintainability
dinethjanitha
committed

Commits on Nov 30, 2025

Add vector embedding service for generating text embeddings using VoyageAI
dinethjanitha
committed