How to get a percentage of the training data or a certain count of samples?

[medemi68]

I plan on using axolotl for finetuning a model, but I don't want to train on the entire dataset (from HuggingFace)'s split. I have two datasets that I am using, and I want to use 80% of one, and 20% of the other, or better yet, be able to specify the number of rows. Is there a way to precisely control the amount of training data I am using and the mixing?

[NanoCode012]

Hey, for splits, HF supports split: train[:80%] natively or split: train[:3000] etc. Does this satisfy what you need? https://huggingface.co/docs/datasets/loading#slice-splits

[medemi68]

I think that might work. If possible it would be great if this could be reflected somewhere in the documentation, unless it already is and I missed it.

[NanoCode012]

It is but very nested I'd say. The link above is pointed here https://docs.axolotl.ai/docs/dataset_loading.html#loading-datasets

If you're interested, you can make a small PR to that section to add an explanation on how to do split :)