This is a Python web scraper designed to fetch product data from PriceSmart's API at https://www.pricesmart.com and store them in a SQLite database. It utilizes asyncio and aiohttp for asynchronous API calls and SQLAlchemy for database operations.
PriceSmart is a membership-based warehouse club operator in the Caribbean and Central America. This scraper focuses on extracting product information from their Trinidad and Tobago (TT) store, including groceries, electronics, household items, and more.
MOBILE APP -- > see our Launch App
- API-Based Scraping: Uses PriceSmart's internal API for reliable data extraction
- Asynchronous Processing: Efficient fetching using asyncio and aiohttp
- Rate Limiting: Implements proper rate limiting (1 request per 2 seconds) to be respectful
- JSON Response Parsing: Robust JSON parsing for structured data extraction
- SQLite Database Storage: Structured storage of product data using SQLAlchemy ORM
- Error Handling: Comprehensive error handling with retry logic and connection recovery
- JSON Response Saving: Saves raw JSON responses for debugging and verification
- Data Analysis: Generates comprehensive analysis reports including brand analysis and pricing statistics
- Multi-Category Support: Can scrape multiple product categories
- Pagination Handling: Automatically handles pagination to get all available products
- Python 3.7 or higher
- aiohttp
- pandas
- SQLAlchemy
- beautifulsoup4
- lxml
- html5lib
- Clone the repository or download the files
- Install the required dependencies:
pip install -r pricesmart_requirements.txt
- The scraper is pre-configured to scrape PriceSmart products from the Trinidad and Tobago store
- Run the scraper:
python pricesmart_scraper.py
- The scraper will:
- Fetch product data from PriceSmart's API with proper rate limiting
- Parse the JSON responses and extract product information
- Store results in the SQLite database
- Generate analysis reports
- Save JSON responses for debugging
You can modify the categories list in the main section to scrape different product categories:
categories = ['G10D03', 'G10D04', 'G10D05'] # Add more category codesThe scraper implements a 2-second delay between requests to be respectful to the API:
- Built-in rate limiting ensures 1 request per 2 seconds
- Automatic delays between category processing
- Database Name:
PriceSmart_Products_Database.db - Location:
Database/folder - Customize by modifying
Database_NameandLocationvariables
The scraper extracts the following data for each product:
- pid: Product ID
- title: Product title/name
- price: Base price
- thumb_image: Product thumbnail image URL
- brand: Product brand
- slug: URL slug
- skuid: SKU ID
- currency: Currency code (TTD)
- fractionDigits: Price decimal places
- master_sku: Master SKU reference
- sold_by_weight_TT: Whether sold by weight
- weight_TT: Weight value
- weight_uom_description_TT: Weight unit of measure
- sign_price_TT: Sign price
- price_per_uom_TT: Price per unit of measure
- uom_description_TT: Unit of measure description
- availability_TT: Product availability
- price_TT: Price in Trinidad and Tobago dollars
- inventory_TT: Inventory count
- promoid_TT: Promotion ID
- category: Product category
- Database:
Database/PriceSmart_Products_Database.db- SQLite database with all scraped product data - HTML Reports:
pricesmart_analysis_report.html- Generated analysis report - Debug Files:
response_{category}_{start}_{rows}.json- Raw JSON responses for debugging
The generated HTML report includes:
- Basic Statistics: Total products, total value, average price
- Brand Analysis: Top 5 brands by product count
- Recent Products: Sample of scraped products with details
- Pricing Analysis: Price distribution and trends
- URL:
https://www.pricesmart.com/api/br_discovery/getProductsByKeyword - Method: POST
- Content-Type: application/json
The scraper sends properly formatted JSON payloads including:
- Category codes
- Pagination parameters
- Authentication keys
- Request metadata
- Retry logic for connection failures (3 attempts)
- Graceful handling of JSON parsing errors
- Comprehensive logging for debugging
- Rate limiting to prevent API overload
- Uses SQLAlchemy ORM for data management
- Auto-incrementing primary key
- Automatic timestamp tracking for data freshness
- Unique ID generation for each record
- Duplicate detection and update logic
This scraper is designed to be respectful to PriceSmart's servers:
- Rate Limiting: 2-second delays between requests
- Proper Headers: Uses appropriate User-Agent and headers
- Session Management: Maintains proper session cookies
- Error Recovery: Graceful handling of temporary failures
- Data Usage: Intended for research and analysis purposes only
Contributions are welcome! If you find any bugs or have suggestions for improvement, please open an issue or submit a pull request.
This project is licensed under the MIT License. See the LICENSE file for details.
- This project was inspired by the need to analyze PriceSmart's product offerings and pricing
- Special thanks to the developers of aiohttp, pandas, SQLAlchemy, and other libraries
- Respectful API usage following best practices for data collection
This scraper is for educational and research purposes only. Please respect PriceSmart's terms of service and use the data responsibly. The scraper is designed to be respectful to their servers and should not be used for commercial purposes without proper authorization.
This project has recently gained unexpected attention. It was created for personal, educational purposes ONLY.
- DO NOT ABUSE THIS SCRIPT: Do not run it excessively or use it for commercial purposes.
- RESPECT THE WEBSITE: Scraping places a load on a website's servers. This script includes a 10-second delay between requests to be respectful. Please do not remove it.
- USE AT YOUR OWN RISK: The user is solely responsible for their use of this script. I (the author) am not responsible for any misuse, server overloads, IP bans, or any legal action that may result from its use. This project is provided as-is for educational demonstration.
{{analysis_placeholder}}
- Total products scraped: 1146
- Total value: $129,532.26
- Average price: $113.03
- New products added: 2
- Existing products updated: 1144
- Price changes detected: 26
- Stock/availability changes: 28
- Discontinued products: 4
| Brand | Count |
|---|---|
| Member's Selection | 184 |
| 136 | |
| Badia | 19 |
| Swiss | 14 |
| Brunswick | 11 |
| Title | Brand | Price (TTD) | Availability |
|---|---|---|---|
| Happy Village Organic Fig and Nut Mini Bars 24 Units / 20 g / 0.7 oz | Happy Village | $146.95 | true |
| Maeva Extra Virgin Olive Oil Spray 2 Units / 400 mL / 13.5 oz | Maeva | $94.95 | true |
| President Brie Cheese Spreadable 3 Units / 139 g / 4.9 oz | President | $74.95 | true |
| Activia Assorted Probiotics Low Fat Yogurt Drink 24 Units / 93 mL / 3.1 oz | Activia | $157.95 | true |
| Bussetto Prosciutto Classico 189 g / 7 oz | Bussetto | $164.95 | true |
| Wellsley Farms Crab Cakes 510 g / 1 lb | Wellsley Farms | $219.95 | true |
| Carrington Farms Coconut Oil Spray 2 Units / 141.7 g / 5 oz | Carrington Farms | $69.95 | true |
| Badia Parsley Flakes 85 g / 3 oz | Badia | $22.18 | true |
| Belgioioso Artigiano Vino Rosso Cheese 453 g / 16 oz | Belgioioso | $99.95 | true |
| Kashi Strawberry Banana Multi-Grain Cereal 720 g / 25.4 oz | Kashi | $97.95 | true |
- Total price changes: 1020
- Price increases: 448
- Price decreases: 537
- Average increase: 11.6%
- Average decrease: -5.3%
| Product | Old Price | New Price | Change | % Change | Type |
|---|---|---|---|---|---|
| KIWA Organic Vegetable Chip Mix 453 g / 16 oz | $0.00 | $66.95 | $+66.95 | +100.0% | New |
| Member's Selection Chilled Boneless Beef Eye of Round Roast, Tray | $226.18 | $224.83 | $-1.35 | -0.6% | Decrease |
| Pork Belly with Skin Frozen Vacuum Packaged | $191.72 | $192.37 | $+0.65 | +0.3% | Increase |
| Chilled Chicken Gizzard Tray Pack | $58.38 | $58.20 | $-0.18 | -0.3% | Decrease |
| Ribeye Choice Fillet Fresh Tray | $285.73 | $290.45 | $+4.72 | +1.7% | Increase |
| Member's Selection Frozen Sliced Turkey Wings, Bag | $153.85 | $153.95 | $+0.10 | +0.1% | Increase |
| Frozen Bone In Pork Shoulder Sliced Tray | $79.85 | $79.74 | $-0.11 | -0.1% | Decrease |
| Prairie Farms Half and Half 946 mL / 32 oz | $0.00 | $37.95 | $+37.95 | +100.0% | New |
| Ovaltine Nutritional Beverage 800 mL / 27 oz | $109.95 | $97.95 | $-12.00 | -10.9% | Decrease |
| Fresh Chicken Mixed Parts Tray | $86.05 | $86.16 | $+0.11 | +0.1% | Increase |
| Fresh Ground Chicken Tray | $101.28 | $101.04 | $-0.24 | -0.2% | Decrease |
| Fresh Chicken Breast Bone In Tray | $93.02 | $92.93 | $-0.09 | -0.1% | Decrease |
| Fresh Seasoned BBQ Chicken Quarters Bag | $88.64 | $88.55 | $-0.09 | -0.1% | Decrease |
| Frozen Skinless Boneless Beef Shoulder Clod Steaks Tray | $108.04 | $108.19 | $+0.15 | +0.1% | Increase |
| Frozen Boneless Skinless Chicken Breast Tray | $122.91 | $122.66 | $-0.25 | -0.2% | Decrease |
| Product | Old Price | New Price | % Increase |
|---|---|---|---|
| Hunt's Diced Tomatoes 8 Units / 411 g / 14.25 oz | $104.95 | $1999.00 | +1804.7% |
| Fresh Beef Ribeye Steak Vacuum Packed | $246.08 | $2434.41 | +889.3% |
| Member's Selection Premium Carved Cooked Ham with Natural Juices 2 Units / 340 g / 12 oz | $9.70 | $69.95 | +621.1% |
| Belgioioso Fresh Mozzarella Cheese Pearls 2 Units / 225 g / 8 oz | $9.70 | $57.95 | +497.4% |
| Pillsbury Cookie Dough Mix 1.3 kg / 3 lb | $19.70 | $109.95 | +458.1% |
| Tropical Frying Cheese 907 g / 32 oz | $19.70 | $89.95 | +356.6% |
| Belgioioso Fresh Mozzarella Snack Cheese 18 Units / 28 g / 1 oz | $19.70 | $89.95 | +356.6% |
| Frozen Boneless Pork Loin Vacuum Packaged | $56.90 | $253.53 | +345.6% |
| Carrot 2.27 kg / 5 lb | $9.70 | $37.95 | +291.2% |
| Silk Unsweetened Original 2 Units 1.89 L / 64 oz | $29.70 | $99.95 | +236.5% |
| Product | Old Price | New Price | % Decrease |
|---|---|---|---|
| Hunt's Diced Tomatoes 8 Units / 411 g / 14.25 oz | $1999.00 | $104.95 | -94.7% |
| Member's Selection Premium Carved Cooked Ham with Natural Juices 2 Units / 340 g / 12 oz | $69.95 | $9.70 | -86.1% |
| Belgioioso Fresh Mozzarella Cheese Pearls 2 Units / 225 g / 8 oz | $57.95 | $9.70 | -83.3% |
| Tropical Frying Cheese 907 g / 32 oz | $89.95 | $19.70 | -78.1% |
| Belgioioso Fresh Mozzarella Snack Cheese 18 Units / 28 g / 1 oz | $89.95 | $19.70 | -78.1% |
| Bombolo Biscotti Decorated Cookies with Autumn Patterns 18 Units | $52.70 | $13.77 | -73.9% |
| Sabra Classic Hummus 850 g / 30 oz | $79.95 | $24.70 | -69.1% |
| Avocado 2 Units | $29.95 | $9.70 | -67.6% |
| Kozyshack Cinnamon Rice Pudding 1.36 kg / 3 lb | $87.95 | $29.70 | -66.2% |
| Green pepper | $49.48 | $16.93 | -65.8% |
| Product | Brand | Last Known Price | Discontinued Date |
|---|---|---|---|
| Green Seedless Grape 1.36 kg / 3 lb | $48.95 | 2026-03-16 | |
| Nature Valley Sweet & Salty Nut Bar 24 Units / 1.2 oz / 35 g | Nature Valley | $112.95 | 2026-03-16 |
| Lifeway Strawberry Kefir 2 Units / 946 mL / 32 oz | Lifeway | $89.95 | 2026-03-16 |
| Member's Selection Fresh Baked Vanilla Cake 10 to 15 Slices | Member's Selection | $58.95 | 2026-03-16 |
| Tostitos Santa Elena Chips 430 g / 15.17 oz | Tostitos | $32.95 | 2026-03-15 |
| Anchor Gouda Cheese Block 500 g / 1.1 lb | Anchor | $59.95 | 2026-03-15 |
| Dewlands Natural Lychee and Apple Juice 3 Units / 1 L | Dewlands | $59.95 | 2026-03-15 |
| Naz Incredible Creations Red Onion Artisan Loaf Bread | Naz Incredible Creations | $36.95 | 2026-03-15 |
| Eggo Chocolatey Chip Waffles 2 Pack / 349 g / 12.3 oz | Eggo | $97.95 | 2026-03-13 |
| Member's Selection Frozen Bone-In Goat Stew Bag | Member's Selection | $87.54 | 2026-03-12 |
| Product | Brand | Price | Category |
|---|---|---|---|
| KIWA Organic Vegetable Chip Mix 453 g / 16 oz | Kiwa | $66.95 | G10D03 |
| Prairie Farms Half and Half 946 mL / 32 oz | Prairie Farms | $37.95 | G10D03 |