Does axolotl automatically interleave data from the datasets?

[medemi68]

Does axolotl automatically interleave data from the datasets? I can't seem to find a definitive answer for this.

[NanoCode012]

@medemi68 , could you clarify what you mean? What do you mean?

Do we mix multiple datasets together for training? Yes.

[medemi68]

I mean, does it train one dataset after the other, or does it mix the datasets.

[NanoCode012]

It mixes

axolotl/src/axolotl/utils/data/shared.py

Lines 525 to 559 in 31a15a4

	def merge_datasets(datasets: list[Dataset], cfg: DictDefault) -> Dataset:
	"""Merge multiple datasets into one with optional shuffling.
	Args:
	datasets: List of datasets to merge.
	cfg: Configuration object containing shuffle settings.
	Returns:
	Merged dataset.
	"""
	if len(datasets) == 1:
	ds = datasets[0]
	# Do not shuffle if curriculum sampling is enabled or
	# shuffle_merged_datasets is disabled
	if cfg.curriculum_sampling or not cfg.shuffle_merged_datasets:
	return ds
	return ds.shuffle(seed=cfg.seed)
	LOG.info("Merging datasets...")
	merged_dataset = concatenate_datasets(datasets)
	if cfg.shuffle_merged_datasets:
	LOG.debug("Shuffling merged datasets...")
	if cfg.curriculum_sampling:
	LOG.warning(
	"Shuffling merged datasets with curriculum sampling is not recommended. "
	"This will randomize the order of samples."
	)
	merged_dataset = merged_dataset.shuffle(seed=cfg.seed)
	else:
	LOG.debug("Not shuffling merged datasets.")
	return merged_dataset