Finetuning without pre-tokenizing

[hanoonaR]

Hi @winglian,

I have a question regarding fine-tuning a model without pre-tokenizing. I am unclear about the correct configuration settings in this context.

If the original fine-tuning with pre-tokenize for the dataset is as follows:

datasets:
    path: mhenrichsen/alpaca_2k_test
    type: alpaca

Would replacing it with the following be appropriate for this need:

pretraining_dataset:
  - path: mhenrichsen/alpaca_2k_test
    type: alpaca

Would replacing the initial configuration with the pretraining_dataset configuration be suitable for my purpose of fine-tuning without pre-tokenizing? Are there any specific implications or differences I should be aware of when opting for the pretraining_dataset configuration over the datasets configuration in this context?

I look forward to your guidance, Thank you in advance.

[NanoCode012]

Could you clarify without pre-tokenizing? The dataset in your first code block is not tokenized. Axolotl tokenizes it.