[RMP] T4R fixes: MultiGPU data parallel training for next-item prediction and fixed serving

[karlhigley]

Problem:

We have customers who would like to use multi-GPU Transformers4Rec but are blocked by issues with our existing support for session-based models.

Goal:

• Unblock customer use cases so they can try out T4R to give us feedback

Constraints:

• We don't yet have Torchscript support (which is out of scope this issue)

Starting Point:

• Enable DataParallel / DistributedDataParallel training using HF Trainer for next-item prediction

  • Next item prediction - [BUG] DataParallel training with Trainer is not using multiple GPUs Transformers4Rec#473 - DataParallel works if the model is wrapped manually by the user (i.e. model = torch.nn.DataParallel(model) for training, but that wrapping should happen automatically by the HF Trainer here
  • [BUG] trainer.model.module renamed and DataParallel mode fixed Transformers4Rec#483
  • [feature]Multi-GPU DistributedDataParallel Fixed Transformers4Rec#496 (review)
  • Documentation on multi-GPU training with DataParallel and DistributedDataParallel Transformers4Rec#492

• Fix the serving sections of the existing T4R notebooks

  • Fix PyTorch export/serving with Python back-end for T4R notebooks NVTabular#1628
  • Fix sparse_max dict in the export_pytorch_ensemble() func  Transformers4Rec#468

• [Task] Add multi-GPU example for Transformer4Rec PyTorch ([Task] Add multi-GPU example for Transformer4Rec PyTorch Transformers4Rec#508)

• [BUG] mutli-gpu example is broken when we pull the release-22.11 branch of NVTabular Transformers4Rec#526

Note: The multi-GPU training of the specific use cases of session binary classification / regression is addressed by RMP #708

[karlhigley]

Distributed Data Parallel training is something we want to do, but I don't think it's part of this effort to fix the immediate blockers. Does that match what y'all understand, @EvenOldridge @viswa-nvidia?

[sohn21c]

Binary Classification DP training would unblock the issue although DDP is preferred if more performant. More details are captured here

[karlhigley]

Either DistributedDataParallel training is part of the scope of the quick fixes or it isn't, and it sounds like it isn't so we should track that work somewhere (but not here.)

[viswa-nvidia]

@gabrielspmoreira , please create a ticket for DP training binary classification and link it here.

[gabrielspmoreira]

@viswa-nvidia @EvenOldridge
Me, @rnyak, @sararb and @nzarif met today about the issues related to DataParallel.
We tested DataParallel for Next Item Prediction for one of the examples and it is not working, differently from what Sara found some weeks ago in another example.
So we have both Next Item Prediction and Binary Classification not working with DataParallel currently.
We have associated the issues for both in this RMP ticket description.
Should we remove the scope of DataParallel from this RMP and create another RMP ticket focused in DataParallel support (targeted for release 22.09)?

[karlhigley]

I don't think we should split the issue, let's just target this for 22.09