Repositories list

• scrapy-playwright

  Public
  🎭 Playwright integration for Scrapy

  hacktoberfestchrome-headlesspython-asyncio+ 9headless-browserjavascript-rendererfirefox-headlessplaywrightplaywright-pythonwebkit-headlesspython+ 2

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •143•1.3k•18•1•Updated Aug 18, 2025Aug 18, 2025

• scrapy-zyte-api

  Public
  Zyte API integration for Scrapy

  plugincrawlerproxy+ 2scrapingscrapy

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •22•38•17•3•Updated Aug 8, 2025Aug 8, 2025

• scrapy-crawl-maps

  Public
  Scrapy plugin that allows defining the logic of a spider using a directed graph defined in JSON format.

  BSD 3-Clause "New" or "Revised" License

  •0•2•0•1•Updated May 21, 2025May 21, 2025

• scrapy-spider-metadata

  Public
  Python

  •

  MIT License

  •3•5•3•0•Updated Mar 24, 2025Mar 24, 2025

• scrapy-zyte-smartproxy

  Public
  Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

  plugincrawlerproxy+ 3scrapingscrapycrawler-detection

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •90•365•7•1•Updated Mar 24, 2025Mar 24, 2025

• scrapy-deltafetch

  Public
  Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls

  hacktoberfesthacktoberfest2021

  Python

  •48•274•13•4•Updated Feb 26, 2025Feb 26, 2025

• scrapy-feedexporter-sftp

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •4•6•0•0•Updated Feb 12, 2025Feb 12, 2025

• scrapy-splash

  Public
  Scrapy+Splash for JavaScript integration

  scrapyheadless-browsers

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •456•3.2k•62•7•Updated Feb 11, 2025Feb 11, 2025

• zyte-spidermon

  Public
  Python

  •1•0•0•1•Updated Sep 5, 2024Sep 5, 2024

• scrapy-settings-log

  Public
  An extension that allows a user to display all or some of their scrapy spider settings at runtime.

  Python

  •

  GNU General Public License v3.0

  •4•0•0•0•Updated Aug 20, 2024Aug 20, 2024

• scrapy-feedexporter-dropbox

  Public
  Scrapy feed exporter for Dropbox

  Python

  •3•1•0•0•Updated Jul 11, 2024Jul 11, 2024

• scrapy-crawlera-fetch

  Public archive
  Scrapy Downloader Middleware for Crawlera Fetch API

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •10•8•5•5•Updated Jun 5, 2024Jun 5, 2024

• scrapy-feedexporter-azure-storage

  Public
  Python

  •6•3•0•3•Updated May 18, 2024May 18, 2024

• scrapy-feedexporter-onedrive

  Public
  Export to OneDrive

  Python

  •1•1•0•0•Updated Feb 23, 2024Feb 23, 2024

• scrapy-djangoitem

  Public
  Scrapy extension to write scraped items using Django models

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •87•503•10•3•Updated Oct 15, 2023Oct 15, 2023

• scrapy-feedexporter-google-sheets

  Public
  Python

  •1•1•0•0•Updated Aug 15, 2023Aug 15, 2023

• scrapy-incremental

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •1•1•0•0•Updated May 1, 2023May 1, 2023

• scrapy-feedexporter-google-drive

  Public
  Python

  •3•3•1•0•Updated Jan 4, 2023Jan 4, 2023

• scrapy-statsd

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •4•6•1•3•Updated Oct 19, 2022Oct 19, 2022

• scrapy-dotpersistence

  Public
  A scrapy extension to sync `.scrapy` folder to an S3 bucket

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •8•17•2•1•Updated Mar 28, 2022Mar 28, 2022

• scrapy-magicfields

  Public
  Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •7•56•0•0•Updated Mar 16, 2022Mar 16, 2022

• scrapy-pagestorage

  Public
  A scrapy extension to store requests and responses information in storage service

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •6•26•3•0•Updated Mar 11, 2022Mar 11, 2022

• scrapy-snowflake-stage-exporter

  Public
  Snowflake database loading utility with Scrapy integration

  snowflakescrapyetl

  Python

  •2•5•0•0•Updated Dec 3, 2021Dec 3, 2021

• scrapy-headless

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •8•28•2•2•Updated Apr 28, 2021Apr 28, 2021

• scrapy-jsonschema

  Public
  Scrapy schema validation pipeline and Item builder using JSON Schema

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •12•44•1•0•Updated Mar 26, 2021Mar 26, 2021

• scrapy-jsonrpc

  Public
  Scrapy extension to control spiders using JSON-RPC

  Python

  •71•300•7•2•Updated Aug 26, 2019Aug 26, 2019

• scrapy-hcf

  Public
  Scrapy spider middleware to use Scrapinghub's Hub Crawl Frontier as a backend for URLs

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •6•4•1•1•Updated Aug 28, 2018Aug 28, 2018

• scrapy-monkeylearn

  Public
  A Scrapy pipeline to categorize items using MonkeyLearn

  Python

  •13•37•0•0•Updated Apr 28, 2017Apr 28, 2017

• scrapy-splitvariants

  Public
  Scrapy spider middleware to split an item into multiple items using a multi-valued key

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •1•20•0•0•Updated Feb 8, 2017Feb 8, 2017

• scrapy-streaming

  Public
  Python

  •5•18•2•6•Updated Oct 12, 2016Oct 12, 2016