Discourse#45

Open

jbaicoianu wants to merge 27 commits intoCarperAI:workingfrom

jbaicoianu:discourse

Commits on Oct 4, 2022

Commits on Oct 5, 2022

Commits on Oct 10, 2022

Crawl list cleanup
jbaicoianu
committed
Profiler support, set retry option based on crawl type
jbaicoianu
committed
Improved crawl data structure, initial support for writing direct to tarballs, code parallelism and optimizations
jbaicoianu
committed

Commits on Oct 19, 2022

Improved caching to make resuming a crawl more reliable and faster, improved stats generation
jbaicoianu
committed
Tweaked spider defaults, add quick summarize step for stats
jbaicoianu
committed

Commits on Nov 2, 2022

Fix for problematic URL structures, improved category caching, first-pass attempt at verify function
jbaicoianu
committed

Commits on Nov 4, 2022

Commits on Nov 8, 2022

Commits on Nov 9, 2022

Commits on Nov 11, 2022

Improved topic processor output
jbaicoianu
committed

Commits on Nov 12, 2022

Commits on Nov 18, 2022

Improved metadata, support for paginated topics
jbaicoianu
committed

Commits on Dec 16, 2022

Commits on Feb 17, 2023

Handle licenses, use jsonl files instead of loose files
jbaicoianu
committed