Skip to content

replace multiprocessing.Pool #12

@andreasvc

Description

@andreasvc

multiprocessing pools work fine unless any kind of error condition arises...

  • properly detect segmentation faults, out of memory, &c.
    concurrent.futures does this, but doesn't take an initializer ... (works on both python 3.3 and backported python 2 version)
  • handle ctrl-C (resolved in python 3.3+ for both multiprocessing and concurrent.futures; cf. http://bugs.python.org/issue9205 )
  • robustness: retry failed jobs (but this might make things worse in certain cases, so better to design jobs so that half-finished jobs can be resumed later)
  • do not rely on copy-on-write memory inherited from parent process; load data in child processes.
  • Full Python traceback from children. parser.workerfunc() decorator does this.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions