[RMP] Improve the speed of training retrieval models with Merlin Models

[karlhigley]

Problem:

Merlin models needs to differentiate itself relative to other RecSys library solutions. One of those areas of differentiation needs to be performance on the GPU. If our libraries don't follow best practices and achieve fast performance that we can measure on GPU then our potential customers have no reason to use the library.

Goal:

• Provide performant retrieval models in production
• Follow best practices by our colleagues for GPU optimization

Constraints

• Merlin models is built on top of Tensorflow

Possible Optimizations

Retrieval models

• [BUG] Building/updating top-k item index for retrieval models evaluations is quite slow models#339
• [BUG] First training batch for retrieval models is slow models#338
• [BUG] Error on retrieval model evaluation when using mixed_precision models#336

[EvenOldridge]

@gabrielspmoreira @marcromeyn Where are we at with the perf regressions we were seeing? Can we close this?

[viswa-nvidia]

Check this bug is done NVIDIA-Merlin/models#339 and this issue should be closed. This is an ongoing effort. The profiling portion will be spun off as a separate RMP ticket

[gabrielspmoreira]

@EvenOldridge @viswa-nvidia As we discussed in the Grooming meeting today, I have tested the pending runtime issue of retrieval models training (NVIDIA-Merlin/models#339) with the current implementation (for both V1 and V2) and it doesn't occur anymore. So that bug was closed.
I also extracted the profiling task of retrieval model pipelines from this RMP to a new RMP #709 , which also addresses ranking model pipelines, as profiling tasks will require an external support (Valerie).
So I am closing this RMP, as it already delivers value with the finished perf improvements on retrieval models.