rework CIFAR-100 for DataDeps

Author: Evizero

README.md

@@ -38,10 +38,13 @@ information
 #### CIFAR-100
 
 The [CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html)
-dataset consists of 600 32x32 color images in 100 classes. The
+dataset consists of 60000 32x32 color images in 100 classes. The
 100 classes are grouped into 20 superclasses (fine and coarse
 labels).
 
+Take a look at the [sub-module](src/CIFAR100/README.md) for more
+information
+
 #### MNIST
 
 The [MNIST](http://yann.lecun.com/exdb/mnist/) dataset consists
@@ -101,7 +104,7 @@ testdata = UD_English.devdata()
 | | Type | Train x | Train y | Test x | Test y |
 |:---:|:---:|:---:|:---:|:---:|:---:|
 | **CIFAR-10** | image | 32x32x3x50000 | 50000 | 32x32x3x10000 | 10000 |
-| **CIFAR-100** | image | 32x32x3x500 | 2x500 | 32x32x3x100 | 2x100 |
+| **CIFAR-100** | image | 32x32x3x5000 | 50000 (x2) | 32x32x3x10000 | 10000 (x2) |
 | **MNIST** | image | 28x28x60000 | 60000 | 28x28x10000 | 10000 |
 | **FashionMNIST** | image | 28x28x60000 | 60000 | 28x28x10000 | 10000 |
 | **PTBLM** | text | 42068 | 42068 | 3761 | 3761 |

src/CIFAR100.jl

@@ -0,0 +1,87 @@
+export CIFAR100
+module CIFAR100
+    using DataDeps
+    using BinDeps
+    using FixedPointNumbers
+    using ..bytes_to_type
+    using ..datafile
+    using ..download_dep
+    using ..download_docstring
+    import ..CIFAR10.convert2image
+    import ..CIFAR10.convert2features
+
+    export
+
+        classnames_coarse,
+        classnames_fine,
+
+        traintensor,
+        testtensor,
+
+        trainlabels,
+        testlabels,
+
+        traindata,
+        testdata,
+
+        convert2image,
+        convert2features,
+
+        download
+
+    const DEPNAME = "CIFAR100"
+    const TRAINSET_FILENAME = joinpath("cifar-100-binary", "train.bin")
+    const TESTSET_FILENAME  = joinpath("cifar-100-binary", "test.bin")
+    const COARSE_FILENAME = joinpath("cifar-100-binary", "coarse_label_names.txt")
+    const FINE_FILENAME = joinpath("cifar-100-binary", "fine_label_names.txt")
+
+    const TRAINSET_SIZE = 50_000
+    const TESTSET_SIZE  = 10_000
+
+    download(args...; kw...) = download_dep(DEPNAME, args...; kw...)
+
+    include(joinpath("Reader","Reader.jl"))
+    include("interface.jl")
+
+    function __init__()
+        RegisterDataDep(
+            DEPNAME,
+            """
+            Dataset: The CIFAR-100 dataset
+            Authors: Alex Krizhevsky, Vinod Nair, Geoffrey Hinton
+            Website: https://www.cs.toronto.edu/~kriz/cifar.html
+            Reference: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf
+
+            [Krizhevsky, 2009]
+                Alex Krizhevsky.
+                "Learning Multiple Layers of Features from Tiny Images",
+                Tech Report, 2009.
+
+            The CIFAR-100 dataset is a labeled subsets of the 80
+            million tiny images dataset. It consists of 60000
+            32x32 colour images in 100 classes. Specifically, it
+            has 100 classes containing 600 images each. There are
+            500 training images and 100 testing images per class.
+            The 100 classes in the CIFAR-100 are grouped into 20
+            superclasses. Each image comes with a "fine" label
+            (the class to which it belongs) and a "coarse" label
+            (the superclass to which it belongs).
+
+            The compressed archive file that contains the
+            complete dataset is available for download at the
+            offical website linked above; specifically the binary
+            version for C programs. We can download and unpack
+            this archive for you if you wish, but that doesn't
+            free you from the burden of using the data
+            responsibly and respect copyright. The authors of
+            CIFAR-100 aren't really explicit about any terms of
+            use, so please read the website to make sure you want
+            to download the dataset.
+            """,
+            "https://www.cs.toronto.edu/~kriz/cifar-100-binary.tar.gz",
+            "58a81ae192c23a4be8b1804d68e518ed807d710a4eb253b1f2a199162a40d8ec",
+            fetch_method = (src, dst) -> run(BinDeps.download_cmd(src, dst)),
+            post_fetch_method = file -> (run(BinDeps.unpack_cmd(file,dirname(file), ".gz", ".tar")); rm(file))
+        )
+    end
+end

src/CIFAR100/CIFAR100.jl

@@ -0,0 +1,85 @@
+# CIFAR-100
+
+Description from the [original
+website](https://www.cs.toronto.edu/~kriz/cifar.html)
+
+> The CIFAR-10 and CIFAR-100 are labeled subsets of the
+> [80 million tiny images](http://people.csail.mit.edu/torralba/tinyimages/)
+> dataset. They were collected by Alex Krizhevsky, Vinod Nair,
+> and Geoffrey Hinton.
+>
+> This dataset is just like the CIFAR-10, except it has 100
+> classes containing 600 images each. There are 500 training
+> images and 100 testing images per class. The 100 classes in the
+> CIFAR-100 are grouped into 20 superclasses. Each image comes
+> with a "fine" label (the class to which it belongs) and a
+> "coarse" label (the superclass to which it belongs).
+
+## Usage
+
+This sub-module provides a programmatic interface to download,
+load, and work with the CIFAR-100 dataset.
+
+```julia
+using MLDatasets
+
+# download dataset
+CIFAR100.download()
+
+# load full training set
+train_x, train_y_coarse, train_y_fine = CIFAR100.traindata()
+
+# load full test set
+test_x, test_y_coarse, test_y_fine  = CIFAR100.testdata()
+```
+
+The provided functions also allow for optional arguments, such as
+the directory `dir` where the dataset is located, or the specific
+observation `indices` that one wants to work with. For more
+information on the interface take a look at the documentation
+(e.g. `?CIFAR100.traindata`).
+
+Function | Description
+---------|-------------
+`download([dir])` | Trigger interactive download of the dataset
+`classnames_coarse()` | Return the 20 super-class names as a vector of strings
+`classnames_fine()` | Return the 100 class names as a vector of strings
+`traintensor([T], [indices]; [dir])` | Load the training images as an array of eltype `T`
+`trainlabels([indices]; [dir])` | Load the labels for the training images
+`testtensor([T], [indices]; [dir])` | Load the test images as an array of eltype `T`
+`testlabels([indices]; [dir])` | Load the labels for the test images
+`traindata([T], [indices]; [dir])` | Load images and labels of the training data
+`testdata([T], [indices]; [dir])` | Load images and labels of the test data
+
+This module also provides utility functions to make working with
+the CIFAR100 dataset in Julia more convenient.
+
+You can use the function `convert2features` to convert the given
+CIFAR100 tensor to a feature matrix (or feature vector in the case
+of a single image). The purpose of this function is to drop the
+spatial dimensions such that traditional ML algorithms can
+process the dataset.
+
+```julia
+julia> CIFAR100.convert2features(CIFAR100.traintensor()) # full training data
+3072×50000 Array{N0f8,2}:
+[...]
+```
+
+To visualize an image or a prediction we provide the function
+`convert2image` to convert the given CIFAR100 horizontal-major
+tensor (or feature matrix) to a vertical-major `Colorant` array.
+
+```julia
+julia> CIFAR100.convert2image(CIFAR100.traintensor(1)) # first training image
+32×32 Array{RGB{N0f8},2}:
+[...]
+```
+
+## References
+
+- **Authors**: Alex Krizhevsky, Vinod Nair, Geoffrey Hinton
+
+- **Website**: https://www.cs.toronto.edu/~kriz/cifar.html
+
+- **[Krizhevsky, 2009]** Alex Krizhevsky. ["Learning Multiple Layers of Features from Tiny Images"](https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf), Tech Report, 2009.

src/CIFAR100/README.md

@@ -0,0 +1,56 @@
+module Reader
+
+export
+
+    readdata!,
+    readdata
+
+const NROW = 32
+const NCOL = 32
+const NCHAN = 3
+const NBYTE = NROW * NCOL * NCHAN + 2 # "+ 2" for label
+
+function readnext!(buffer::Array{UInt8}, io::IO)
+    c = Int(read(io, UInt8))
+    f = Int(read(io, UInt8))
+    read!(io, buffer)
+    buffer, c, f
+end
+
+function readdata!(buffer::Array{UInt8}, io::IO, index::Integer)
+    seek(io, (index - 1) * NBYTE)
+    readnext!(buffer, io)
+end
+
+function readdata(io::IO, nobs::Int, index::Integer)
+    buffer = Array{UInt8}(NROW, NCOL, NCHAN)
+    readdata!(buffer, io, index)
+end
+
+function readdata(io::IO, nobs::Int)
+    X = Array{UInt8}(NROW, NCOL, NCHAN, nobs)
+    C = Array{Int}(nobs)
+    F = Array{Int}(nobs)
+    buffer = Array{UInt8}(NROW, NCOL, NCHAN)
+    @inbounds for index in 1:nobs
+        _, tc, tf = readnext!(buffer, io)
+        copy!(view(X,:,:,:,index), buffer)
+        C[index] = tc
+        F[index] = tf
+    end
+    X, C, F
+end
+
+function readdata(file::AbstractString, nobs::Int, index::Integer)
+    open(file, "r") do io
+        readdata(io, nobs, index)
+    end::Tuple{Array{UInt8,3},Int,Int}
+end
+
+function readdata(file::AbstractString, nobs::Int)
+    open(file, "r") do io
+        readdata(io, nobs)
+    end::Tuple{Array{UInt8,4},Vector{Int},Vector{Int}}
+end
+
+end