Add `n_token_limit` to dataset#79

Merged

sfluegel05 merged 2 commits intodevfrom

feature/maxlen-dataset

Apr 7, 2025

Commits on Apr 3, 2025

add a parameter to the dataset that (if set), throws out all instances that have more than x tokens
sfluegel05
committed

Commits on Apr 7, 2025

reformat using black
sfluegel05
committed