Skip to content

Parallel is greedy #592

@iguinn

Description

@iguinn

The parallelism in math enabled by PYGAMA_PARALLEL=True by default creates as many threads as possible. This has a couple of problems. First, it can slow things down in some situations due to overhead associated with moving things to threads. Second, NERSC doesn't want us doing this on login/jupyter nodes.

For now the solution will be to default it to False. However, in the future, we should add a way to control the number of threads used (perhaps replace PYGAMA_PARALLEL with PYGAMA_NTRHEADS). This can be controlled in numba using numba.set_num_threads() as discussed here: https://numba.readthedocs.io/en/stable/user/threading-layer.html. For default behavior we have a couple of options:

  1. Default to threading off. The disadvantage of this approach is that it means people will be testing their code single-threaded and may miss bugs. For this reason, we would want to make sure tests still ran multi-threaded in some instances
  2. Choose a smaller number of threads as the default.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions