Skip to content

Implement New State of the Art Self-Supervised Architectures #109

@xanderdunn

Description

@xanderdunn

tsai primarily implements BERT-like and InceptionTime architectures for self-supervised learning. BERT was designed for NLP tasks and InceptionTime is an ensemble of CNNs, which were designed for image tasks.

There are new state of the art models for self-supervised learning on images: SwAV and ReLIC. I believe SwAV is currently considered state of the art. ReLIC with stronger augmentations may be superior to SwAV. These architectures were trained and tested on image datasets, but, similar to the generalization of BERT and CNNs, they may generalize well to time series data.

This was intended more as an informational message to @oguiza than an issue to keep open until complete. Even if this is interesting to implement, it's a lot of work. I'll have to stick to using MVP and InceptionTimePlus for the time being, but I might get around to implementing one of these newer architectures after I've completed work using the existing architectures.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions