[MLM] Train script for non causal decoder#300

Draft

thomasw21 wants to merge 298 commits intomainfrom

thomas/mlm_train_script

This pull request is big! We're only showing the most recent 250 commits

Commits on Jun 29, 2022

made into input and output tokens

Lintang Sutawika
authored and
thomasw21
committed
added eos

Lintang Sutawika
authored and
thomasw21
committed
added eos

Lintang Sutawika
authored and
thomasw21
committed
test text_token

Lintang Sutawika
authored and
thomasw21
committed
test text_token

Lintang Sutawika
authored and
thomasw21
committed
test text_token

Lintang Sutawika
authored and
thomasw21
committed
test text_token

Lintang Sutawika
authored and
thomasw21
committed
test text_token

Lintang Sutawika
authored and
thomasw21
committed
assigned array

Lintang Sutawika
authored and
thomasw21
committed
assigned array

Lintang Sutawika
authored and
thomasw21
committed
assigned array

Lintang Sutawika
authored and
thomasw21
committed
hardcoded sequence length

Lintang Sutawika
authored and
thomasw21
committed
check again

Lintang Sutawika
authored and
thomasw21
committed
show sentinal tokens

lintangsutawika
authored and
thomasw21
committed
show sentinal tokens

lintangsutawika
authored and
thomasw21
committed
show sentinal tokens

lintangsutawika
authored and
thomasw21
committed
show sentinal tokens

lintangsutawika
authored and
thomasw21
committed
add more special tokens

lintangsutawika
authored and
thomasw21
committed
changed how mlm data is loaded

lintangsutawika
authored and
thomasw21
committed
changed how mlm data is loaded

lintangsutawika
authored and
thomasw21
committed
changed how mlm data is loaded

lintangsutawika
authored and
thomasw21
committed
changed how mlm data is loaded

lintangsutawika
authored and
thomasw21
committed
changed how mlm data is loaded

lintangsutawika
authored and
thomasw21
committed
added new script

lintangsutawika
authored and
thomasw21
committed
added new script

lintangsutawika
authored and
thomasw21
committed
added new script

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
try t5 dataset

lintangsutawika
authored and
thomasw21
committed
developing

lintangsutawika
authored and
thomasw21
committed
developing

lintangsutawika
authored and
thomasw21
committed
developing

lintangsutawika
authored and
thomasw21
committed
developing

lintangsutawika
authored and
thomasw21
committed
developing

lintangsutawika
authored and
thomasw21
committed
developing

lintangsutawika
authored and
thomasw21
committed
developing

lintangsutawika
authored and
thomasw21
committed
test to see output of get_ltor_masks_and_position_ids

lintangsutawika
authored and
thomasw21
committed
test to see output of get_ltor_masks_and_position_ids

lintangsutawika
authored and
thomasw21
committed
add new script

Lintang Sutawika
authored and
thomasw21
committed
add new script

Lintang Sutawika
authored and
thomasw21
committed
add new script

Lintang Sutawika
authored and
thomasw21
committed
changed settings

Lintang Sutawika
authored and
thomasw21
committed
changed settings

Lintang Sutawika
authored and
thomasw21
committed
tidy up

Lintang Sutawika
authored and
thomasw21
committed
changed tokenizer and position embedding

Lintang Sutawika
authored and
thomasw21
committed
modifying mlm to reflect original implementation

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
set correct seq len

Lintang Sutawika
authored and
thomasw21
committed
refined sampling method

Lintang Sutawika
authored and
thomasw21
committed
refined sampling method

Lintang Sutawika
authored and
thomasw21
committed
refined sampling method

Lintang Sutawika
authored and
thomasw21
committed
refined sampling method

Lintang Sutawika
authored and
thomasw21
committed
refined sampling method

Lintang Sutawika
authored and
thomasw21
committed
refined sampling method

Lintang Sutawika
authored and
thomasw21
committed
first commit, adding non causal mlm dataset

Lintang Sutawika
authored and
thomasw21
committed
fixed mlm dataset

Lintang Sutawika
authored and
thomasw21
committed
fixed mlm dataset

Lintang Sutawika
authored and
thomasw21
committed
fixed mlm dataset

Lintang Sutawika
authored and
thomasw21
committed
fixed mlm dataset

Lintang Sutawika
authored and
thomasw21
committed
fixed mlm dataset

Lintang Sutawika
authored and
thomasw21
committed
minor changes

Lintang Sutawika
authored and
thomasw21
committed
removed mlm related scripts

Lintang Sutawika
authored and
thomasw21
committed
removed any scipts not related to dataset, revert arguments

Lintang Sutawika
authored and
thomasw21
committed
added sampler and test

Lintang Sutawika
authored and
thomasw21
committed
added testing data

Lintang Sutawika
authored and
thomasw21
committed
adapted test loader

Lintang Sutawika
authored and
thomasw21
committed
Update megatron/data/non_causal_mtf_dataset.py

Lintang Sutawika
and
thomasw21
committed
removed unused files

Lintang Sutawika
authored and
thomasw21
committed
changed with impossible token

Lintang Sutawika
authored and
thomasw21
committed
enable loading multiple indexed_dataset for each field

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
data_prefix is set as dict

Lintang Sutawika
authored and
thomasw21
committed
removed sample_idx lines

Lintang Sutawika
authored and
thomasw21
committed
change line from sample_idx to doc_idx

Lintang Sutawika
authored and
thomasw21
committed
replace shuffling _build_index_mappings with random.sample of the doc_idx

Lintang Sutawika
authored and
thomasw21
committed
minor changes

Lintang Sutawika
authored and
thomasw21
committed
Cleanup artefacts

Muennighoff
authored and
thomasw21
committed
Add packed preprocessing

Muennighoff
authored and
thomasw21
committed
Use seq_length arg

Muennighoff
authored and
thomasw21
committed
Add sources & docstrings

Muennighoff
authored and
thomasw21
committed
added training process for t0

Lintang Sutawika
authored and
thomasw21
committed
Update pretrain_t0.py

Lintang Sutawika
authored and
thomasw21
committed
Remove a bunch of code that's not needed
thomasw21
committed

Commits on Jun 30, 2022

WIP
thomasw21
committed
Cleanup
thomasw21
committed
Add back all configs
thomasw21
committed
Woops
thomasw21
committed
Fix tests
thomasw21
committed
Rename testing files
thomasw21
committed
Do in-place operations
thomasw21
committed
Do in-place operations
thomasw21
committed
Woops
thomasw21
committed
Fix typo
thomasw21
committed
Add test that packing is done optimially via greedy algorithm
thomasw21
committed
Woops
thomasw21
committed
added capabilities for padding and prefix lm index

lintangsutawika
authored and
thomasw21
committed
added adjustments and new dataset

Lintang Sutawika
authored and
thomasw21
committed
added sentinal tokens

Lintang Sutawika
authored and
thomasw21
committed
made into input and output tokens

Lintang Sutawika
authored and
thomasw21
committed
modifying mlm to reflect original implementation

Lintang Sutawika
authored and
thomasw21
committed
minor fix

Lintang Sutawika
authored and
thomasw21
committed
added sampler and test

Lintang Sutawika
authored and
thomasw21
committed
Enable training

Muennighoff
authored and
thomasw21
committed
Add T0 training test

Muennighoff
authored and
thomasw21
committed
Remove artefacts

Muennighoff
authored and
thomasw21
committed
Remove artefacts

Muennighoff
authored and
thomasw21
committed
WIP
thomasw21
committed

Commits on Jul 1, 2022

Commits on Jul 2, 2022

Commits on Jul 4, 2022