Skip to content

future.md

Latest commit

 

History

History
78 lines (61 loc) · 3.03 KB

future.md

File metadata and controls

78 lines (61 loc) · 3.03 KB

Datasets and evaluation

ImageNet validation split

To be able to use the ImageNet validation split, we need to manually download and copy ILSVRC2012_img_val.tar to ~/tensorflow_datasets/downloads/manual/ and then run

import tensorflow_datasets as tfds

tfds.builder("imagenet2012_real").download_and_prepare()

Then we have access to both original and reassessed (ReaL) labels under the keys original_label and real_label. We use imagenet2012_real, because to process the imagenet2012 dataset via tensorflow-datasets requires us to download validation and train splits, even though we don't intend to use train (for now).

Having downloaded ILSVRC2012_img_val.tar, we can also access imagenet2012_corrupted via tensorflow-datasets.

Other ImageNet variants, such as

  • ImageNet-A (imagenet_a)
  • ImageNet-R (imagenet_r) and
  • ImageNet-v2 (imagenet_v2, defaults to matched-frequency) can be downloaded automatically via tensorflow-datasets.

To do before 0.1.0

Evaluation

  • (must) Evaluate model on ImageNet
  • (must) Run evaluation on GPU, e.g., Google Colab
  • (optional) Evaluate model on ImageNet-ReaL
  • (optional) Evaluate model on ImageNet-v2
  • (optional) Evaluate model on ImageNet-A
  • (optional) Evaluate model on ImageNet-R
  • (optional) Evaluate model on ImageNet-C
  • (must) Profiling single image inference time on CPU
  • (optional) Profiling #params, FLOPS
  • (optional) Profiling memory consumption on CPU

Finetuning

  • (must) Load weights from file to model with nb_classes
  • (must) Set weight decay in loaded model
  • (must) Fine-tune pretrained model on CIFAR-100
  • (must) Evaluate model on CIFAR-100
  • (optional) Load weights from file to model with in_channels != 3
  • (optional) Fine-tune model on MNIST (and variants)
  • (optional) Evaluate model on MNIST (and variants)

To Do

  • Add interpolate_input and fixed_input_size to all transformer models
  • Add no weight_decay rule for certain layers
  • Fix weight initialisation
  • Test that nb_features is set correctly for all models
  • Sort out ViT, DeiT, etc. weight naming + prefixes Add unit test checking that all weights have correct prefix
  • Speed up unit tests by introducing (small) test architectures
  • Convert feature dicts to ordered dicts.

Future