Repositories list

• ungoliant

  Public
  :spider: The pipeline for the OSCAR corpus

  nlpcrawlercorpus-linguistics+ 5

  nlpcrawlercorpus-linguisticsfasttextoscarcommoncrawlcommon-crawllanguage-classification

  Rust

  •

  Apache License 2.0

  •17•176•18•14•Updated Nov 9, 2025Nov 9, 2025

• oscar-website

  Public
  The website of the Oscar Project

  nlpwebsitemachine-learning+ 3

  nlpwebsitemachine-learninghugolanguage-modelcommon-crawl

  TeX

  •

  Apache License 2.0

  •13•11•2•1•Updated Mar 27, 2025Mar 27, 2025

• documentation

  Public
  Apache License 2.0

  •4•5•9•0•Updated Nov 13, 2024Nov 13, 2024

• oscar-statistics

  Public
  Compute statistics for OSCAR Monthly releases

  Rust

  •

  Apache License 2.0

  •0•2•0•0•Updated Sep 2, 2024Sep 2, 2024

• oscar-io

  Public
  Readers/Writers for OSCAR Corpus

  Rust

  •

  Apache License 2.0

  •1•0•2•1•Updated Apr 16, 2024Apr 16, 2024

• oscar-tools

  Public
  The original tooling for the OSCAR corpus rewritten in Rust

  Rust

  •

  Apache License 2.0

  •3•5•9•6•Updated Dec 19, 2023Dec 19, 2023

• ut1-rs

  Public
  ut1-blocklist rust library

  Rust

  •

  MIT License

  •0•1•1•0•Updated Nov 9, 2023Nov 9, 2023

• .github

  Public
  1•0•0•1•Updated Aug 8, 2023Aug 8, 2023

• OSCAR-CommonCrawl-Collab

  Public
  Jupyter Notebook

  •0•2•2•0•Updated Mar 12, 2023Mar 12, 2023

• download_oscar

  Public
  Downloading all files of a language from the OSCAR (Open Super-large Crawled Aggregated coRpus)

  Python

  •

  MIT License

  •2•10•0•1•Updated Feb 28, 2023Feb 28, 2023

• data-hub

  Public
  Collab around OSCAR: Data soucing..

  Python

  •0•1•1•0•Updated Feb 9, 2023Feb 9, 2023

• discussions

  Public
  0•0•0•0•Updated Jan 18, 2023Jan 18, 2023

• oscar-blocklists

  Public
  A compilation of multilingual URL blocklist

  Creative Commons Zero v1.0 Universal

  •1•2•0•0•Updated Dec 21, 2022Dec 21, 2022

• corpus

  Public
  corpus issues.

  Apache License 2.0

  •0•5•21•0•Updated Oct 18, 2021Oct 18, 2021

• oscar-tools-go

  Public archive
  A tooling for the OSCAR corpus

  Go

  •

  Apache License 2.0

  •0•1•0•0•Updated Jun 25, 2021Jun 25, 2021

• goclassy

  Public archive
  An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.

  nlpcorpus-linguisticsfasttext+ 2

  nlpcorpus-linguisticsfasttextcommon-crawllanguage-classification

  Go

  •

  Apache License 2.0

  •6•86•0•0•Updated Apr 21, 2021Apr 21, 2021