Mlm adaptation#287

Merged

thomasw21 merged 180 commits intobigscience-workshop:mainfrom

lintangsutawika:mlm-adaptation

Jun 27, 2022

Commits on May 7, 2022

added train script but with prefix manually declared
Lintang Sutawika
committed

Commits on May 9, 2022

made new dataset
Lintang Sutawika
committed
minor adjustments
Lintang Sutawika
committed
added capabilities for padding and prefix lm index
lintangsutawika
committed
added finetune script
lintangsutawika
committed
removed script
lintangsutawika
committed
added adjustments and new dataset
Lintang Sutawika
committed
try mlm dataset
Lintang Sutawika
committed
minor changes
Lintang Sutawika
committed
minor addition of import packages
Lintang Sutawika
committed
minor error fix
Lintang Sutawika
committed
minor error fix
Lintang Sutawika
committed
samples follow how gpt dataset is loaded
Lintang Sutawika
committed
added masked_lm_prob
Lintang Sutawika
committed
fixed tokenizer abstractions for HF tokenizer
Lintang Sutawika
committed
added mask id
Lintang Sutawika
committed
added mask id
Lintang Sutawika
committed
added mask id
Lintang Sutawika
committed
added mask id
Lintang Sutawika
committed
added fix
Lintang Sutawika
committed
added bos and eos token id
Lintang Sutawika
committed
no need for sentinal token
Lintang Sutawika
committed
add aux functions
Lintang Sutawika
committed
add aux functions
Lintang Sutawika
committed
add aux functions
Lintang Sutawika
committed
add pad_id
Lintang Sutawika
committed

Commits on May 18, 2022

changed lm predictions to t5
Lintang Sutawika
committed
changed lm predictions to t5
Lintang Sutawika
committed
changed lm predictions to t5
Lintang Sutawika
committed
changed lm predictions to t5
Lintang Sutawika
committed
changed lm predictions to t5
Lintang Sutawika
committed
tokenizer add mask, cls, sep tokens
Lintang Sutawika
committed

Commits on May 21, 2022

commit latest changes
Lintang Sutawika
committed
commit latest changes
Lintang Sutawika
committed
added sentinal tokens
Lintang Sutawika
committed
added sentinal tokens
Lintang Sutawika
committed
added sentinal tokens
Lintang Sutawika
committed
added additional_special_tokens
Lintang Sutawika
committed
added additional_special_tokens
Lintang Sutawika
committed
check t5_input and output
Lintang Sutawika
committed
check decoder in and decoder out
Lintang Sutawika
committed

Commits on May 22, 2022

made into input and output tokens
Lintang Sutawika
committed
made into input and output tokens
Lintang Sutawika
committed
made into input and output tokens
Lintang Sutawika
committed
made into input and output tokens
Lintang Sutawika
committed
made into input and output tokens
Lintang Sutawika
committed
made into input and output tokens
Lintang Sutawika
committed
made into input and output tokens
Lintang Sutawika
committed

Commits on May 23, 2022

made into input and output tokens
Lintang Sutawika
committed
made into input and output tokens
Lintang Sutawika
committed
added eos
Lintang Sutawika
committed
added eos
Lintang Sutawika
committed

Commits on May 24, 2022

test text_token
Lintang Sutawika
committed
test text_token
Lintang Sutawika
committed
test text_token
Lintang Sutawika
committed
test text_token
Lintang Sutawika
committed
test text_token
Lintang Sutawika
committed
assigned array
Lintang Sutawika
committed
assigned array
Lintang Sutawika
committed
assigned array
Lintang Sutawika
committed
hardcoded sequence length
Lintang Sutawika
committed

Commits on May 28, 2022

Commits on May 29, 2022

test to see output of get_ltor_masks_and_position_ids
lintangsutawika
committed
test to see output of get_ltor_masks_and_position_ids
lintangsutawika
committed
add new script
Lintang Sutawika
committed
add new script
Lintang Sutawika
committed
add new script
Lintang Sutawika
committed

Commits on May 30, 2022

changed settings
Lintang Sutawika
committed
changed settings
Lintang Sutawika
committed

Commits on May 31, 2022

tidy up
Lintang Sutawika
committed
changed tokenizer and position embedding
Lintang Sutawika
committed

Commits on Jun 2, 2022

modifying mlm to reflect original implementation
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
minor fix
Lintang Sutawika
committed
set correct seq len
Lintang Sutawika
committed

Commits on Jun 8, 2022

refined sampling method
Lintang Sutawika
committed
refined sampling method
Lintang Sutawika
committed
refined sampling method
Lintang Sutawika
committed
refined sampling method
Lintang Sutawika
committed
refined sampling method
Lintang Sutawika
committed
refined sampling method
Lintang Sutawika
committed
first commit, adding non causal mlm dataset
Lintang Sutawika
committed
fixed mlm dataset
Lintang Sutawika
committed
fixed mlm dataset
Lintang Sutawika
committed
fixed mlm dataset
Lintang Sutawika
committed
fixed mlm dataset
Lintang Sutawika
committed
fixed mlm dataset
Lintang Sutawika
committed

Commits on Jun 12, 2022

Merge branch 'bigscience-workshop:main' into mt0
Lintang Sutawika
authored

Commits on Jun 14, 2022

minor changes
Lintang Sutawika
committed
Merge branch 'mt0' of https://github.com/lintangsutawika/Megatron-DeepSpeed into mt0
Lintang Sutawika
committed

Commits on Jun 22, 2022

removed multitask finetuning related scripts
Lintang Sutawika
committed
Merge branch 'bigscience-workshop:main' into mlm-adaptation
Lintang Sutawika
authored
remove any unrelated to dataset, revert arguments.py
Lintang Sutawika
committed
revert tokenizer
Lintang Sutawika
committed

Commits on Jun 23, 2022

Improve MLM
thomasw21
committed
Woops
thomasw21
committed
Remove a bunch of attributes
thomasw21
committed
Fix naming
thomasw21
committed
Woops
thomasw21
committed
Use GPTDataset as underlying implementation
thomasw21
committed
Fix sep tokens
thomasw21
committed
Change attribute naming
thomasw21
committed
GPT Dataset doesn't handle slicing
thomasw21
committed
Remove tokenizer
thomasw21
committed
WIP
thomasw21
committed
WIP
thomasw21
committed
WIP
thomasw21
committed
WIP
thomasw21
committed
WIP
thomasw21
committed
WIP
thomasw21
committed
WIP
thomasw21
committed
MLM
thomasw21
committed
Cleanup
thomasw21
committed

Commits on Jun 26, 2022

Update megatron/data/mlm_dataset.py

Lintang Sutawika
and
Muennighoff
authored

Commits on Jun 27, 2022