[RMP] Multi-GPU Data Parallel training for Tensorflow in Merlin Models

[viswa-nvidia]

Problem:

Single GPU training takes significantly longer than multi-gpu. Customers would like to be able to accelerate their training workflows by distributing training across multiple GPUs on a single node.

Goal:

Enable customers to do data parallel training within Merlin Models training pipeline.

Constraints:

• Single node
• Embedding tables fit within the memory of a single gpu
• Use NVIDIA best practices; aka Horovod

Starting Point:

• [BUG] Getting error from XGB model when loading the model back and passing the booster arg to the constructor  models#651

Example

• update logo usecase models#693

[EvenOldridge]

@marcromeyn can you flesh this out a little further.

[gabrielspmoreira]

This is a related ticket #764 - [BUG] Models does not support to tf.distribute.MirroredStrategy() for data parallel training.
It is multi-GPU but not Horovod,

[viswa-nvidia]

@viswa-nvidia to follow up with DLFW team regarding native support

[bschifferer]

@viswa-nvidia @EvenOldridge

We need to add following success criteria :

• Analyze scaling factor by using multiple GPUs: If we go from 1x GPU -> 2x GPUs -> 4x GPUs -> 8 GPUx - how much higher is the throughput?
• Provide performance metrics (accuracy / AUC / etc) to show that there is no negative effect in the model performance
• Provide guidance how to set global batch size, batch size per GPU and learning rate when scaling

If we provide only the technical functionality without testing the points above, we cannot guarantee it is working. I get these questions often from customers.

[bschifferer]

I provided an example to show that Merlin Models work with horovods: NVIDIA-Merlin/models#778

However, we need to address the points above + the bug ( NVIDIA-Merlin/dataloader#75 ).

In addition, we should make it more user-friendly.

[viswa-nvidia]

Noted. @EvenOldridge , please review and add to the goals. I am not sure if this ticket is fully defined.

[edknv]

I've been doing some experimentation with horovod integration with Models API, based on @bschifferer's example NVIDIA-Merlin/models#778, and I fully agree with his point on all the success criteria he listed above, and also the need for dataloader to produce equal number of batches across partitions, as mentioned in NVIDIA-Merlin/dataloader#75.

Some additional notes on NVIDIA-Merlin/dataloader#75, it seems that the dataloader produces an unequal number of batches when the dataset is partitioned, which is problematic for horovod because one worker might finish processing all the batches and wait idle and/or hang and/or time out while the other worker(s) are still processing their batches. There might be some workaround like seeding from the dataloader side as mentioned in the issue, or from horovod side using hvd.join(), but the best solution is to have dataloader produce equal number of batches when partitioned.

[bschifferer]

I provided following example based on the current code: NVIDIA-Merlin/models#855 - as I am OOO next week.

@edknv did a great job to provide the horovod functionality in Merlin Models.

I think we need to review the current flow of multi-GPU. I think it is not fully user-friendly / end-to-end integration:

• I am not sure, if the issues with unequal batch size for the data loader is solved: [BUG] Data parallel training freezes due to different number of batches dataloader#75 - if the solution is moved to the data is generated, correctly - how das that work? How are we ensure it with NVTabular? How about users who do NOT use NVTabular?
• The unittest is written that each worker runs through the FULL dataset per epoch. That is incorrect. If we have 1M data points and 2 GPUs, each GPU should run only through 500k data points. I wrote the example that NVTabular produces distinct files per worker - however, that is not a solution which guarantees the point above.

We havent look on the points here:

• Analyze scaling factor by using multiple GPUs: If we go from 1x GPU -> 2x GPUs -> 4x GPUs -> 8 GPUx - how much higher is the throughput?
• Provide performance metrics (accuracy / AUC / etc) to show that there is no negative effect in the model performance
• Provide guidance how to set global batch size, batch size per GPU and learning rate when scaling