Repositories list

• scrapy-zyte-api

  Public
  Zyte API integration for Scrapy

  plugincrawlerproxy+ 2scrapingscrapy

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •22•40•16•5•Updated Feb 10, 2026Feb 10, 2026

• scrapy-playwright

  Public
  🎭 Playwright integration for Scrapy

  pythonpython3scrapy+ 9hacktoberfestchrome-headlesspython-asyncioheadless-browserjavascript-rendererfirefox-headlessplaywright+ 2

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •154•1.4k•15•1•Updated Jan 21, 2026Jan 21, 2026

• scrapy-crawl-maps

  Public

  scrapy-plugins/scrapy-crawl-maps’s past year of commit activity
  Scrapy plugin that allows defining the logic of a spider using a directed graph defined in JSON format.

  BSD 3-Clause "New" or "Revised" License

  •0•3•0•1•Updated May 21, 2025May 21, 2025

• scrapy-spider-metadata

  Public
  Python

  •

  MIT License

  •3•7•3•1•Updated Mar 24, 2025Mar 24, 2025

• scrapy-zyte-smartproxy

  Public
  Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

  plugincrawlerproxy+ 3scrapingscrapycrawler-detection

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •91•368•7•1•Updated Mar 24, 2025Mar 24, 2025

• scrapy-deltafetch

  Public
  Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls

  hacktoberfesthacktoberfest2021

  Python

  •48•277•13•4•Updated Feb 26, 2025Feb 26, 2025

• scrapy-feedexporter-sftp

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •4•7•0•0•Updated Feb 12, 2025Feb 12, 2025

• scrapy-splash

  Public

  scrapy-plugins/scrapy-splash’s past year of commit activity
  Scrapy+Splash for JavaScript integration

  scrapyheadless-browsers

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •454•3.2k•62•7•Updated Feb 11, 2025Feb 11, 2025

• zyte-spidermon

  Public
  Python

  •1•1•0•1•Updated Sep 5, 2024Sep 5, 2024

• scrapy-settings-log

  Public
  An extension that allows a user to display all or some of their scrapy spider settings at runtime.

  Python

  •

  GNU General Public License v3.0

  •4•1•0•0•Updated Aug 20, 2024Aug 20, 2024

• scrapy-feedexporter-dropbox

  Public
  Scrapy feed exporter for Dropbox

  Python

  •3•2•0•0•Updated Jul 11, 2024Jul 11, 2024

• scrapy-crawlera-fetch

  Public archive
  Scrapy Downloader Middleware for Crawlera Fetch API

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •10•9•5•5•Updated Jun 5, 2024Jun 5, 2024

• scrapy-feedexporter-azure-storage

  Public
  Python

  •6•4•0•3•Updated May 18, 2024May 18, 2024

• scrapy-feedexporter-onedrive

  Public
  Export to OneDrive

  Python

  •1•3•0•0•Updated Feb 23, 2024Feb 23, 2024

• scrapy-djangoitem

  Public
  Scrapy extension to write scraped items using Django models

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •87•505•10•3•Updated Oct 15, 2023Oct 15, 2023

• scrapy-feedexporter-google-sheets

  Public
  Python

  •1•2•0•0•Updated Aug 15, 2023Aug 15, 2023

• scrapy-incremental

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •1•3•0•0•Updated May 1, 2023May 1, 2023

• scrapy-feedexporter-google-drive

  Public
  Python

  •3•4•1•0•Updated Jan 4, 2023Jan 4, 2023

• scrapy-statsd

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •3•7•1•2•Updated Oct 19, 2022Oct 19, 2022

• scrapy-dotpersistence

  Public
  A scrapy extension to sync `.scrapy` folder to an S3 bucket

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •8•18•2•1•Updated Mar 28, 2022Mar 28, 2022

• scrapy-magicfields

  Public
  Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •6•57•0•0•Updated Mar 16, 2022Mar 16, 2022

• scrapy-pagestorage

  Public
  A scrapy extension to store requests and responses information in storage service

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •6•27•3•0•Updated Mar 11, 2022Mar 11, 2022

• scrapy-snowflake-stage-exporter

  Public
  Snowflake database loading utility with Scrapy integration

  etlsnowflakescrapy

  Python

  •2•6•0•0•Updated Dec 3, 2021Dec 3, 2021

• scrapy-headless

  Public
  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •8•29•2•2•Updated Apr 28, 2021Apr 28, 2021

• scrapy-jsonschema

  Public
  Scrapy schema validation pipeline and Item builder using JSON Schema

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •12•45•1•0•Updated Mar 26, 2021Mar 26, 2021

• scrapy-jsonrpc

  Public
  Scrapy extension to control spiders using JSON-RPC

  Python

  •71•300•7•2•Updated Aug 26, 2019Aug 26, 2019

• scrapy-hcf

  Public
  Scrapy spider middleware to use Scrapinghub's Hub Crawl Frontier as a backend for URLs

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •6•5•1•1•Updated Aug 28, 2018Aug 28, 2018

• scrapy-monkeylearn

  Public
  A Scrapy pipeline to categorize items using MonkeyLearn

  Python

  •13•38•1•0•Updated Apr 28, 2017Apr 28, 2017

• scrapy-splitvariants

  Public
  Scrapy spider middleware to split an item into multiple items using a multi-valued key

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •1•21•0•0•Updated Feb 8, 2017Feb 8, 2017

• scrapy-streaming

  Public
  Python

  •5•19•2•0•Updated Oct 12, 2016Oct 12, 2016