Removing dependency on `torch` and `torchvision`

[paganpasta]

Currently these are only used to load weights. Is it possible to move the model loading bit into the package so that we dont have to install torch and torchvision. ?