Skip to content

PostprocessingDataset: add support for multiprocessing #1765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 61 commits into
base: master
Choose a base branch
from
Open

PostprocessingDataset: add support for multiprocessing #1765

Show file tree
Hide file tree
Changes from 52 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
87864e2
Pull apart map_seq application and dataset iteration
NeoLegends Sep 19, 2025
7e6f78f
make mapping iterator builder static
NeoLegends Sep 19, 2025
823434e
Implement multi-process postprocessing
NeoLegends Sep 19, 2025
7d35731
add test (currently fails due to pickle issues)
NeoLegends Sep 19, 2025
e70135b
add TODO on dataset lock
NeoLegends Sep 19, 2025
bf878f2
ignore broad exception lint
NeoLegends Sep 19, 2025
bc1a7c5
extend docs
NeoLegends Sep 19, 2025
6c463e9
test if tensor pickling makes it work
NeoLegends Sep 19, 2025
4ecfdfd
late night todo to self: fix draining workers
NeoLegends Sep 19, 2025
55abd83
add docs
NeoLegends Sep 19, 2025
b700514
CI: ensure numpy remains at v1 even in espnet installation
NeoLegends Sep 19, 2025
31a5413
fix lints
NeoLegends Sep 22, 2025
4aa1349
optimize
NeoLegends Sep 22, 2025
69a88f3
clean implementation
NeoLegends Sep 22, 2025
72152bf
init wrapped dataset in feeder thread
NeoLegends Sep 22, 2025
8987608
cleaner code
NeoLegends Sep 22, 2025
f38178e
Merge remote-tracking branch 'origin/master' into moritz-pp-multiproc
NeoLegends Sep 22, 2025
22455dc
pass correct epoch, but multiply worker count and worker index in via…
NeoLegends Sep 22, 2025
fb8f317
limit RNG seed range
NeoLegends Sep 22, 2025
715efa2
Merge branch 'master' into moritz-pp-multiproc
NeoLegends Sep 22, 2025
27d76dd
move multi proc implementation into subclass
NeoLegends Sep 22, 2025
55bf701
test dataset across multiple epochs
NeoLegends Sep 23, 2025
0d82829
switch to pipe
NeoLegends Sep 24, 2025
3908b7b
test across multiple epochs
NeoLegends Sep 25, 2025
39047f1
re-add cache to feeder thread
NeoLegends Sep 25, 2025
7584a08
also catch broken pipe error
NeoLegends Sep 25, 2025
e0e17c9
we can actually signal exit outside of the dataset lock
NeoLegends Sep 25, 2025
bc6b025
fix while loop nesting
NeoLegends Sep 25, 2025
576c116
re-merge into single class
NeoLegends Sep 25, 2025
5c0e14a
default num_workers=0
NeoLegends Sep 25, 2025
a0bc8a1
ensure seq order and distributor function is only called in multi-wor…
NeoLegends Sep 25, 2025
5309127
fix blocking bug
NeoLegends Sep 25, 2025
9420fbf
avoid variable shadowing
NeoLegends Sep 25, 2025
e4568bd
keep worker procs alive across subepochs
NeoLegends Sep 26, 2025
a81940e
extend tests
NeoLegends Sep 26, 2025
0dcc6ee
also clean up background thread in __del__
NeoLegends Sep 26, 2025
19689ab
fix lint errors
NeoLegends Sep 26, 2025
f335fb8
fix deadlock
NeoLegends Sep 26, 2025
3e5cff1
enforce that complete_frac is monotonic
NeoLegends Sep 26, 2025
34f829a
fix wrong typing
NeoLegends Sep 26, 2025
0f8bbbf
fix variable shadowing lint
NeoLegends Sep 26, 2025
f6a44ac
periodically check quit_event
NeoLegends Sep 26, 2025
05929ef
assertion msg does not have anything to do with what it does
NeoLegends Sep 26, 2025
f2c8282
naming
NeoLegends Sep 26, 2025
8e6ccdc
simplify
NeoLegends Sep 26, 2025
093464a
comment
NeoLegends Sep 26, 2025
a26d7e8
naming
NeoLegends Sep 26, 2025
6cd0e65
docs
NeoLegends Sep 26, 2025
64a41a5
more docs
NeoLegends Sep 26, 2025
a302c37
close feeder conn after switching epoch
NeoLegends Sep 26, 2025
20f4496
better name for feeder thread
NeoLegends Sep 26, 2025
2a40c0c
do not pass postprocessors on every epoch
NeoLegends Sep 26, 2025
d2a9101
add frequent GC collection
NeoLegends Sep 29, 2025
c2bc4fd
improve test
NeoLegends Sep 29, 2025
1fa9f50
add 100 seqs lower bound if gc_interval is not configured explicitly
NeoLegends Sep 29, 2025
a4c509c
fix naming issue
NeoLegends Sep 29, 2025
ad1e878
pass 0 interval correctly
NeoLegends Sep 29, 2025
c08733f
docs
NeoLegends Sep 29, 2025
fb2d6e1
fix long line
NeoLegends Sep 29, 2025
ba99180
remove unused import
NeoLegends Sep 30, 2025
d182725
Merge branch 'master' into moritz-pp-multiproc
NeoLegends Oct 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading