Further plans on integrations with HuggingFace transformers?

[Yorko]

How I see a possible integration with HuggingFace.

We reuse best practices for training NNs in general, which are also implemented in Catalyst:

• batch accumulation
• warmup
• nvidia apex support
• Cycling Learning Rate
• shedulers

We also built upon Catalyst training environment:

• configs
• logging
• monitoring training (w&b support)
• reproducibility

We can extend it with NLP-specific stuff:

• custom loaders for different tasks
• sequence bucketing
• tensor trimming
• ...

Let's extend and elaborate.

[xelibrion]

I, for one, would also like to see easy inference/debugging built in, especially for seq2seq.

I know Catalyst already has some of that functionality, I was thinking of extending it and potentially having an interactive notebook that would allow for super easy visualization of predictions.
On the other side, this might also be a wasted effort and something like https://prodi.gy/ might be better suited for the task.

Another piece of work I had in mind was adding more meaningful metrics to the seq2seq pipeline, to move away from minimizing NLL.

[lightforever]

There is an idea that we should integrate this repository to Catalyst. @xelibrion , what do you think about it? Just to make it easier to start a new project and create examples.

For an example, Catalyst has already examples for image classification/segmentation. There is a request from Hacktoberfest to have an example for text classification: catalyst-team/catalyst#426

To manage it, I see the following tasks:

• flake8 style support (the rules are the same as for Catalyst), In progress...
• generalize BertCrossEntropyLoss, BertCriterionCallback. We can add mask functionality to standard CriterionCallback/CrossEntropyLoss, In progress...
• union model/modelwrapper. And generalize a unique model, In progress...
• integrate the code into Catalyst framework, In progress...
• TextClassificationDataset, In progress...
• create a text classification example
• text classification tutorial notebook
• text classification tutorial in Colab like classification tutorial