List view

feat: add utility function (and/or data) for URL datasets if necessary

documentationImprovements or additions to documentationImprovements or additions to documentation

enhancementNew feature or requestNew feature or request

wontfixThis will not be worked onThis will not be worked on

Status: Closed (completed).

#20 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Apr 8, 2022

docs: what if we also want to separate metadata from their target (source?) of raw data?

questionFurther information is requestedFurther information is requested

Status: Closed (completed).

#17 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Aug 23, 2021

feat: add special tokens to request (or not) metadata generation

enhancementNew feature or requestNew feature or request

Status: Closed (completed).

#36 In bigscience-workshop/metadata;

· by SaulLu was closed on Oct 14, 2021

feat: support HF dataset loader and streaming mode for bs-modeling-metadata/openwebtext-html-cc

enhancementNew feature or requestNew feature or request

wontfixThis will not be worked onThis will not be worked on

Status: Closed (completed).

#62 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Apr 8, 2022

feat: add URL tokenizer(s) for data source, time stamp, and website description

enhancementNew feature or requestNew feature or request

wontfixThis will not be worked onThis will not be worked on

Status: Closed (completed).

#54 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Apr 8, 2022

perf: filter 404 URLs out by Common Crawl's cluster.idx

enhancementNew feature or requestNew feature or request

wontfixThis will not be worked onThis will not be worked on

Status: Closed (completed).

#58 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Dec 1, 2021

perf: separate C4 or CommonCrawl URLs from OpenWebText URLs

enhancementNew feature or requestNew feature or request

wontfixThis will not be worked onThis will not be worked on

Status: Closed (completed).

#59 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Apr 8, 2022

perf: reduce file sizes of openwebtext-html-cc

enhancementNew feature or requestNew feature or request

Status: Closed (completed).

#60 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Nov 13, 2021

fix: clean up OpenWebText URL fragments, duplicates, variants, and malformed ones

bugSomething isn't workingSomething isn't working

wontfixThis will not be worked onThis will not be worked on

Status: Closed (completed).

#61 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Apr 8, 2022

adding support to resume from a saved model (checkpoint)

enhancementNew feature or requestNew feature or request

Status: Merged (completed).

bigscience-workshop/metadatanumber 42

#42 In bigscience-workshop/metadata;

· by shanyas10 was closed on Jul 2, 2022

feat: get OpenWebText from Common Crawl

enhancementNew feature or requestNew feature or request

Status: Closed (abandoned).

bigscience-workshop/metadatanumber 51

#51 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Jan 21, 2022

feat: add special tokens to request (or not) metadata generation

enhancementNew feature or requestNew feature or request

Status: Merged (completed).

bigscience-workshop/metadatanumber 38

#38 In bigscience-workshop/metadata;

· by SaulLu was closed on Oct 14, 2021

adding jz slurm files

enhancementNew feature or requestNew feature or request

Status: Merged (completed).

bigscience-workshop/metadatanumber 43

#43 In bigscience-workshop/metadata;

· by shanyas10 was closed on Oct 4, 2021

refactor: eval tqdm without newline

enhancementNew feature or requestNew feature or request

Status: Merged (completed).

bigscience-workshop/metadatanumber 28

#28 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Sep 6, 2021

feat: add <code>UrlProcessor</code> for applying <code>unquote_plus()</code> to URLs.

enhancementNew feature or requestNew feature or request

Status: Merged (completed).

bigscience-workshop/metadatanumber 18

#18 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Aug 17, 2021

build: set Python and dependency versions with an optional Poetry env

enhancementNew feature or requestNew feature or request

Status: Merged (completed).

bigscience-workshop/metadatanumber 19

#19 In bigscience-workshop/metadata;

· by tianjianjiang was closed on Sep 6, 2021