Continue the pre-training step

[miguelwon]

Hi,
I don't want to train instructions but just continue the pre-training step. Yes there any example I can quickly use?
Also, I want to use my local dataset. Any info about how should I write the json file?
Thanks

[ZQ-Dev8]

In your config.yml, what you'll do is put "completion" as the type: in the datasets: section, meaning you're intending to train on a dataset containing a column called "text" where the raw text completion data is. If your column is named something else, you can use the field: parameter in the dataset: to override the default.

For the local dataset, just ensure it's a .jsonl file with a column entitled "text", unless you override it as described above. All other columns will be ignored.

[miguelwon]

Thanks!

[mtisz]

Do I only need to have one field in the jsonl file? Only the "text" field is required? And this "text" field should be text with sequence length, say 8K?

[NanoCode012]

Sorry for reviving old post. Just wanted to clarify this part.

Do I only need to have one field in the jsonl file? Only the "text" field is required? And this "text" field should be text with sequence length, say 8K?

It can be variable length. If it's longer than sequence_len, it will be split into smaller sequences. If it's smaller, it'll be concat with other sequences or padded depending on your config.