Skip to content

@scrapinghub Scrapinghub

Change the repository type filter

All

    Repositories list

    183 repositories

    • shub-workflow

      Public
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •14•15•2•1•Updated Sep 5, 2025Sep 5, 2025

    • python-crfsuite

      Public
      A python binding for crfsuite
      data-sciencecrfcrfsuite
      Python
      •
      MIT License
      •222•772•45•3•Updated Sep 5, 2025Sep 5, 2025

    • Formasaurus

      Public
      Formasaurus tells you the type of an HTML form and its fields using machine learning
      HTML
      •48•7•1•1•Updated Sep 4, 2025Sep 4, 2025

    • sklearn-crfsuite

      Public
      scikit-learn inspired API for CRFsuite
      Python
      •214•2•0•1•Updated Sep 4, 2025Sep 4, 2025

    • extruct

      Public
      Extract embedded metadata from HTML markup
      microformatssemantic-webrdfajson-ldmicrodataopengraphhacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •118•929•39•15•Updated Sep 2, 2025Sep 2, 2025

    • dateparser

      Public
      python parser for human readable dates
      hacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •479•2.7k•296•53•Updated Aug 22, 2025Aug 22, 2025

    • shub

      Public
      Scrapinghub Command Line Client
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •81•132•48•15•Updated Aug 22, 2025Aug 22, 2025

    • web-poet

      Public
      Web scraping Page Objects core library
      pythonweb-scrapinghacktoberfestpage-objects
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •16•101•17•13•Updated Aug 22, 2025Aug 22, 2025

    • andi

      Public
      Library for annotation-based dependency injection
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •6•23•4•1•Updated Aug 22, 2025Aug 22, 2025

    • scrapy-poet

      Public
      Page Object pattern for Scrapy
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •28•121•13•5•Updated Aug 22, 2025Aug 22, 2025

    • scrapinghub-stack-scrapy

      Public
      Software stack with latest Scrapy and updated deps
      Dockerfile
      •
      BSD 3-Clause "New" or "Revised" License
      •20•65•1•1•Updated Aug 2, 2025Aug 2, 2025

    • scrapyrt

      Public
      HTTP API for Scrapy spiders
      pythoncrawlerscrapercrawlingtwistedscrapywebcrawlerhacktoberfestwebcrawlinghacktoberfest2021
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •162•869•24•6•Updated Jul 28, 2025Jul 28, 2025

    • scrapinghub-entrypoint-scrapy

      Public
      Scrapy entrypoint for Scrapinghub job runner
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •16•26•6•0•Updated Jul 21, 2025Jul 21, 2025

    • scrapy-frontera

      Public
      More flexible and featured Frontera scheduler for Scrapy
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •5•35•2•1•Updated Jun 6, 2025Jun 6, 2025

    • hcf-backend

      Public
      Crawl Frontier HCF backend
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •5•8•2•1•Updated Jun 6, 2025Jun 6, 2025

    • frontera

      Public
      A scalable frontier for web crawlers
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •218•1.3k•78•17•Updated Jun 6, 2025Jun 6, 2025

    • spidermon

      Public
      Scrapy Extension for monitoring spiders execution.
      testingmonitoringscrapingcrawlingspidershacktoberfestmonitoring-toolscrapinghub
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •100•545•42•4•Updated Apr 11, 2025Apr 11, 2025

    • python-scrapinghub

      Public
      A client interface for Scrapinghub's API
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •60•204•25•4•Updated Feb 21, 2025Feb 21, 2025

    • price-parser

      Public
      Extract price amount and currency symbol from a raw text string
      hacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •51•337•17•9•Updated Feb 13, 2025Feb 13, 2025

    • social-app-django

      Public
      Python Social Auth - Application - Django
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •389•2•0•1•Updated Nov 18, 2024Nov 18, 2024

    • number-parser

      Public
      Parse numbers written in natural language
      hacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •25•123•13•6•Updated Oct 23, 2024Oct 23, 2024

    • streamparse

      Public
      streamparse lets you run Python code against real-time streams of data. Integrates with Apache Storm.
      Python
      •
      Apache License 2.0
      •219•2•0•1•Updated Sep 20, 2024Sep 20, 2024

    • splash

      Public
      Lightweight, scriptable browser as a service with an HTTP API
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •516•4.2k•374•26•Updated Aug 2, 2024Aug 2, 2024

    • pgcontents

      Public
      A Postgres-backed ContentsManager implementation for IPython
      Python
      •
      Apache License 2.0
      •86•2•0•1•Updated Jul 18, 2024Jul 18, 2024

    • shublang

      Public
      Pluggable DSL that uses pipes to perform a series of linear transformations to extract data
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •8•16•23•6•Updated Jul 9, 2024Jul 9, 2024

    • woodpecker

      Public
      An opinionated fork of the Drone CI system
      Go
      •
      Other
      •454•0•0•5•Updated Jul 7, 2024Jul 7, 2024

    • varanus

      Public
      A command line spider monitoring tool
      spidermonitoringpython36
      Python
      •7•8•2•2•Updated Jul 6, 2024Jul 6, 2024

    • portia

      Public
      Visual scraping for Scrapy
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •1.4k•9.5k•111•19•Updated Jun 26, 2024Jun 26, 2024

    • webstruct-demo

      Public
      HTTP demo for https://github.com/scrapinghub/webstruct
      Python
      •
      MIT License
      •2•4•0•3•Updated Jun 17, 2024Jun 17, 2024

    • autologin

      Public
      A project to attempt to automatically login to a website given a single seed
      Python
      •
      Apache License 2.0
      •43•11•0•2•Updated Jun 17, 2024Jun 17, 2024