Skip to content

e-commerce-auth-scrapingcourse

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

e-commerce-auth-scrapingcourse

README.md

E-Commerce Auth Product Scraper (Python)

Authenticated e-commerce scraping automation that extracts product information from a protected dashboard using Auth Sessions.

Run on Intuned

Run on Intuned

APIs

API Description
list List products from the authenticated dashboard
details Get detailed information for a specific product

Getting started

Install dependencies

uv sync

If the intuned CLI is not installed, install it globally:

npm install -g @intuned/cli

After installing dependencies, intuned command should be available in your environment.

Prepare the project

Before running any API, provision and deploy the project first.

intuned dev provision
intuned dev deploy

Run an API

intuned dev run api list .parameters/api/list/default.json --auth-session test-authsession
intuned dev run api details .parameters/api/details/default.json --auth-session test-authsession

Auth Sessions

# Create
intuned dev run authsession create .parameters/auth-sessions/create/default.json

# Validate
intuned dev run authsession validate test-authsession

# Update
intuned dev run authsession update test-authsession

Project structure

/
├── api/
│   ├── list.py                       # List products from the authenticated dashboard
│   └── details.py                    # Get detailed product information
├── auth-sessions/
│   ├── check.py                      # Validates if the auth session is still active
│   └── create.py                     # Creates/recreates the auth session
├── auth-sessions-instances/
│   └── test-authsession/            # Example local auth session
│       ├── auth-session.json
│       └── metadata.json
├── utils/
│   └── types_and_schemas.py          # Python types and Pydantic models
├── intuned-resources/
│   ├── jobs/
│   │   └── list.job.jsonc            # Top-level job example for the list -> details flow
│   └── auth-sessions/
│       └── test-authsession.auth-session.jsonc  # Auth session credentials
├── .parameters/api/                  # Test parameters
├── Intuned.jsonc                      # Project config
├── pyproject.toml                     # Python dependencies
└── README.md

API Reference

list — Product List Scraper

Scrapes products from the authenticated dashboard.

Parameters: None

Returns: List of products with:

  • name: Product name
  • detailsUrl: URL to product details page

Features:

  • Requires authenticated session
  • Automatically navigates to dashboard
  • Triggers details API for each product using extend_payload

details — Product Details Scraper

Scrapes detailed information for a specific product.

Parameters:

  • name: Product name
  • detailsUrl: URL to the product details page

Returns: Product details object with:

  • name: Product name
  • price: Product price
  • sku: Stock Keeping Unit
  • category: Product category
  • shortDescription: Brief product description
  • fullDescription: Complete product description
  • imageAttachments: List of product images (uploaded to S3)
  • availableSizes: List of available sizes
  • availableColors: List of available colors
  • variants: List of product variants with stock information

Using intuned_browser SDK

This project uses the Intuned browser SDK for enhanced reliability:

  • go_to_url: Navigate to URLs with automatic retries and intelligent timeout detection
  • save_file_to_s3: Automatically upload images and files to S3 storage
  • extend_payload: Trigger additional API calls dynamically (used to trigger details API for each product)

For more information, check out the Intuned Browser SDK documentation.

Related