Commit 7343c98
Refactor scraper to async PC component scraper with parallel execution
- Convert from keyboard-only scraper to multi-component PC parts scraper
- Implement async/await pattern with AsyncClient from scrapegraph-py
- Add parallel page scraping within each component using asyncio.gather()
- Support 8 PC component categories: CPU, GPU, RAM, Motherboard, SSD, PSU, Case, CPU Cooler
- Scrape 10 pages per component (80 total pages) with parallelization
- Add component-specific URL generation and categorization
- Update Product objects with component_type in specifications
- Enhance statistics tracking per component
- Update queries to reflect PC component dataset
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>1 parent 1dd953a commit 7343c98
1 file changed
+230
-121
lines changed
0 commit comments