Skip to content

Hard examples mining with multi GPU #3350

@ckobus

Description

@ckobus

Hi,

First, thanks for all the recent developments on the library!

I try to mine hard examples using sentence_transformers.util.mine_hard_negatives. I am using a reranker which slows down the process.
As I have a 4 GPUs machine, I would like to use them all to speed up. But even by putting the argument use_multi_process to True, during the rescoring with the cross-encoder, only 1 GPU is used.

Has anyone else experienced the same issue?

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions