how to split corpus into train/dev ?

[d5555]

Is it possible to add a feature to Corpus to get iterable according persentage from the start and from the end so I can devide my dataset into train and dev ?
train=Corpus (DATASET_Path , 0.8 ) # 80% from the start
dev=Corpus (DATASET_Path , -0.2 ) # 20% from the end

There is an option limit but it won't work if we need to set a limit from the end (to get dev Corpus)

Your Environment

• Operating System:
• spaCy Version Used: 3.0.5

[adrianeboyd]

We can consider whether more built-in options make sense here.

Currently your best option is write your own corpus reader. Here's what the provided one looks like:

spaCy/spacy/training/corpus.py

Lines 22 to 39 in ed561cf

	@util.registry.readers("spacy.Corpus.v1")
	def create_docbin_reader(
	path: Optional[Path],
	gold_preproc: bool,
	max_length: int = 0,
	limit: int = 0,
	augmenter: Optional[Callable] = None,
	) -> Callable[["Language"], Iterable[Example]]:
	if path is None:
	raise ValueError(Errors.E913)
	util.logger.debug(f"Loading corpus from path: {path}")
	return Corpus(
	path,
	gold_preproc=gold_preproc,
	max_length=max_length,
	limit=limit,
	augmenter=augmenter,
	)

You'd want to modify the options and how the corpus is returned to split a single file into partitions.

[mbrunecky]

Doing the first 80% last 20% split assumes the corpus is already thoroughly randomized. But than it should be easy enough to just split the DATASET_path into two directories by hand (assuming there are multiple smaller docBins).
Another approach to splitting the dataset into 'train' and 'dev' is getting a random value for each doc and if the random value is >= 0.8 the doc goes into 'train' else into 'dev'.
I am not sure this enhancements is needed, because (at least in my experience) preparing the dataset takes a lot of effort (I am creating docBins with my code), and splitting it into train/dev is just a small piece of that.

But for those using the convert utility perhaps this could be a convert utility feature...