Skip to content

Commit 7343c98

Browse files
lurenssclaude
andcommitted
Refactor scraper to async PC component scraper with parallel execution
- Convert from keyboard-only scraper to multi-component PC parts scraper - Implement async/await pattern with AsyncClient from scrapegraph-py - Add parallel page scraping within each component using asyncio.gather() - Support 8 PC component categories: CPU, GPU, RAM, Motherboard, SSD, PSU, Case, CPU Cooler - Scrape 10 pages per component (80 total pages) with parallelization - Add component-specific URL generation and categorization - Update Product objects with component_type in specifications - Enhance statistics tracking per component - Update queries to reflect PC component dataset 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
1 parent 1dd953a commit 7343c98

File tree

1 file changed

+230
-121
lines changed

1 file changed

+230
-121
lines changed

0 commit comments

Comments
 (0)