[BUG] Training of the Retrieval API V2 is slower than the V1

[sararb]

Bug description

Running the integration tests defined in #790 using retrieval API v1 and v2, we observed that v2 is slower than v1 (see figure below):

[sararb]

Debugging results

• The reported training times shown in this bug ticket are generated using VS code debugging interface.

• These values cannot be trusted because the VS code is introducing a random overhead. In the following figure, you can observe that the same model configuration (TT-V2-Train-and-Eval or TT-V1-Train-and-Eval) has different execution times and throughputs:

• I ran the same integration tests using the command line instead and the results are more consistent. We can also observe that API V2 (TT-V2) is faster than API V1 thanks to a faster top-k evaluation.

• I also ran the research script to train (and evaluate) a two-tower model using the Tenrec dataset. Here again the retrieval API V2 is faster than the V1 API (faster evaluation while comparable training time).

[sararb]

Closing this ticket as the results of debugging experiments show that the retrieval API V2 is faster than the V1.