Self-Supervised Pre-Training for Tabular models

There are a number of changes and new features in this release, here is a summary:

1. Refactored the code related to the 3 forms of training in the library:

   • Supervised Training (via the Trainer class)
   • Self-Supervised pre-training: we have implemented two methods or routines for self-supervised pre-training. These are:
     • Encoder-Decoder Pre-Training (via the EncoderDecoderTrainer class): this is inspired by the TabNet paper
     • Constrastive-Denoising Pre-Training (via de ConstrastiveDenoising class): this is inspired by the SAINT paper
   • Bayesian or Probabilistic Training (via the BayesianTrainer: this is inspired by the paper Weight Uncertainty in Neural Networks
     

   Just as a reminder, the current deep learning models for tabular data available in the library are:

   • Wide
   • TabMlp
   • TabResNet
   • TabNet
   • TabTransformer
   • FTTransformer
   • SAINT
   • TabFastformer
   • TabPerceiver
   • BayesianWide
   • BayesianTabMlp

2. The text related component has now 3 available models, all based on RNNs. There are reasons for that although the integration with the Hugginface Transformer library is the next step in the development of the library. The 3 models available are:

   • BasicRNN
   • AttentiveRNN
   • StackedAttentiveRNN

   The last two are based on Hierarchical Attention Networks for Document Classification. See the docs for details

3. The image related component is now fully integrated with the latest torchvision release, with a new Multi-Weight Support API. Currently, the model variants supported by our library are:

   • resnet
   • shufflenet
   • resnext
   • wide_resnet
   • regnet
   • densenet
   • mobilenet
   • mnasnet
   • efficientnet
   • squeezenet