Skip to content

Commit d98ed58

Browse files
EvizeroEvizero
authored
Merge pull request #15 from JuliaML/svhn
add SVHN dataset (Format 2)
2 parents 2437380 + 3ac7785 commit d98ed58

File tree

12 files changed

+827
-18
lines changed

12 files changed

+827
-18
lines changed

‎.travis.yml

Lines changed: 15 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,28 @@
11
language: julia
2-
32
os:
4-
- linux
5-
- osx
6-
3+
- linux
4+
- osx
75
julia:
8-
- 0.6
9-
- nightly
10-
matrix:
11-
allow_failures:
12-
- julia: nightly
6+
- 0.6
7+
- nightly
8+
notifications:
9+
email: false
1310
git:
14-
depth: 5000
11+
depth: 99999999
1512

16-
notifications:
17-
email: false
13+
matrix:
14+
allow_failures:
15+
- julia: nightly
1816

19-
before_script:
20-
- export PATH=$HOME/.local/bin:$PATH
17+
addons:
18+
apt: # apt-get for linux
19+
packages:
20+
- hdf5-tools
2121

2222
install:
2323
#- sudo pip install pymdown-extensions
2424

2525
after_success:
26+
- julia -e 'cd(Pkg.dir("MLDatasets")); Pkg.add("Coverage"); using Coverage; Coveralls.submit(Coveralls.process_folder())'
2627
- julia -e 'Pkg.add("Documenter")'
2728
- julia -e 'cd(Pkg.dir("MLDatasets")); include(joinpath("docs", "make.jl"))'
28-
29-
script:
30-
- if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
31-
- julia -e 'Pkg.clone(pwd()); Pkg.build("MLDatasets"); Pkg.test("MLDatasets"; coverage=true)'

‎README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,9 @@ Dataset | Classes | `traintensor` | `trainlabels` | `testtensor` | `testlabels`
7070
[**FashionMNIST**](https://juliaml.github.io/MLDatasets.jl/latest/datasets/FashionMNIST/) | 10 | 28x28x60000 | 60000 | 28x28x10000 | 10000
7171
[**CIFAR-10**](https://juliaml.github.io/MLDatasets.jl/latest/datasets/CIFAR10/) | 10 | 32x32x3x50000 | 50000 | 32x32x3x10000 | 10000
7272
[**CIFAR-100**](https://juliaml.github.io/MLDatasets.jl/latest/datasets/CIFAR100/) | 100 (20) | 32x32x3x50000 | 50000 (x2) | 32x32x3x10000 | 10000 (x2)
73+
[**SVHN-2**](https://juliaml.github.io/MLDatasets.jl/latest/datasets/SVHN2/) (*) | 10 | 32x32x3x73257 | 73257 | 32x32x3x26032 | 26032
74+
75+
(*) Note that the SVHN-2 dataset provides an additional 531131 observations aside from the training- and testset
7376

7477
### Language Modeling
7578

‎REQUIRE

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@ ColorTypes 0.4
55
DataDeps
66
GZip
77
BinDeps
8+
MAT

‎docs/make.jl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ makedocs(
1818
"Fashion MNIST" => "datasets/FashionMNIST.md",
1919
"CIFAR-10" => "datasets/CIFAR10.md",
2020
"CIFAR-100" => "datasets/CIFAR100.md",
21+
"SVHN format 2" => "datasets/SVHN2.md",
2122
],
2223
],
2324
hide("Indices" => "indices.md"),

‎docs/src/datasets/SVHN2.md

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# [The Street View House Numbers (SVHN) Dataset](@id SVHN2)
2+
3+
Description from the [official
4+
website](http://ufldl.stanford.edu/housenumbers/):
5+
6+
> SVHN is a real-world image dataset for developing machine
7+
> learning and object recognition algorithms with minimal
8+
> requirement on data preprocessing and formatting. It can be
9+
> seen as similar in flavor to MNIST (e.g., the images are of
10+
> small cropped digits), but incorporates an order of magnitude
11+
> more labeled data (over 600,000 digit images) and comes from a
12+
> significantly harder, unsolved, real world problem (recognizing
13+
> digits and numbers in natural scene images). SVHN is obtained
14+
> from house numbers in Google Street View images.
15+
16+
About Format 2 (Cropped Digits):
17+
18+
> All digits have been resized to a fixed resolution of 32-by-32
19+
> pixels. The original character bounding boxes are extended in
20+
> the appropriate dimension to become square windows, so that
21+
> resizing them to 32-by-32 pixels does not introduce aspect
22+
> ratio distortions. Nevertheless this preprocessing introduces
23+
> some distracting digits to the sides of the digit of interest.
24+
25+
!!! note
26+
27+
For non-commercial use only
28+
29+
## Contents
30+
31+
```@contents
32+
Pages = ["SVHN2.md"]
33+
Depth = 3
34+
```
35+
36+
## Overview
37+
38+
The `MLDatasets.SVHN2` sub-module provides a programmatic
39+
interface to download, load, and work with the SVHN2 dataset of
40+
handwritten digits.
41+
42+
```julia
43+
using MLDatasets
44+
45+
# load full training set
46+
train_x, train_y = SVHN2.traindata()
47+
48+
# load full test set
49+
test_x, test_y = SVHN2.testdata()
50+
51+
# load additional train set
52+
extra_x, extra_y = SVHN2.extradata()
53+
```
54+
55+
The provided functions also allow for optional arguments, such as
56+
the directory `dir` where the dataset is located, or the specific
57+
observation `indices` that one wants to work with. For more
58+
information on the interface take a look at the documentation
59+
(e.g. `?SVHN2.traindata`).
60+
61+
Function | Description
62+
---------|-------------
63+
[`download([dir])`](@ref SVHN2.download) | Trigger interactive download of the dataset
64+
[`classnames()`](@ref SVHN2.classnames) | Return the class names as a vector of strings
65+
[`traintensor([T], [indices]; [dir])`](@ref SVHN2.traintensor) | Load the training images as an array of eltype `T`
66+
[`trainlabels([indices]; [dir])`](@ref SVHN2.trainlabels) | Load the labels for the training images
67+
[`traindata([T], [indices]; [dir])`](@ref SVHN2.traindata) | Load images and labels of the training data
68+
[`testtensor([T], [indices]; [dir])`](@ref SVHN2.testtensor) | Load the test images as an array of eltype `T`
69+
[`testlabels([indices]; [dir])`](@ref SVHN2.testlabels) | Load the labels for the test images
70+
[`testdata([T], [indices]; [dir])`](@ref SVHN2.testdata) | Load images and labels of the test data
71+
[`extratensor([T], [indices]; [dir])`](@ref SVHN2.extratensor) | Load the extra images as an array of eltype `T`
72+
[`extralabels([indices]; [dir])`](@ref SVHN2.extralabels) | Load the labels for the extra training images
73+
[`extradata([T], [indices]; [dir])`](@ref SVHN2.extradata) | Load images and labels of the extra training data
74+
75+
This module also provides utility functions to make working with
76+
the SVHN (format 2) dataset in Julia more convenient.
77+
78+
Function | Description
79+
---------|-------------
80+
[`convert2features(array)`](@ref SVHN2.convert2features) | Convert the SVHN tensor to a flat feature matrix
81+
[`convert2image(array)`](@ref SVHN2.convert2image) | Convert the SVHN tensor/matrix to a colorant array
82+
83+
You can use the function
84+
[`convert2features`](@ref SVHN2.convert2features) to convert
85+
the given SVHN tensor to a feature matrix (or feature vector
86+
in the case of a single image). The purpose of this function is
87+
to drop the spatial dimensions such that traditional ML
88+
algorithms can process the dataset.
89+
90+
```julia
91+
julia> SVHN2.convert2features(SVHN2.traindata()[1]) # full training data
92+
3072×73257 Array{N0f8,2}:
93+
[...]
94+
```
95+
96+
To visualize an image or a prediction we provide the function
97+
[`convert2image`](@ref SVHN2.convert2image) to convert the
98+
given SVHN2 horizontal-major tensor (or feature matrix) to a
99+
vertical-major `Colorant` array.
100+
101+
```julia
102+
julia> SVHN2.convert2image(SVHN2.traindata(1)[1]) # first training image
103+
32×32 Array{RGB{N0f8},2}:
104+
[...]
105+
```
106+
107+
## API Documentation
108+
109+
```@docs
110+
SVHN2
111+
```
112+
113+
### Trainingset
114+
115+
```@docs
116+
SVHN2.traintensor
117+
SVHN2.trainlabels
118+
SVHN2.traindata
119+
```
120+
121+
### Testset
122+
123+
```@docs
124+
SVHN2.testtensor
125+
SVHN2.testlabels
126+
SVHN2.testdata
127+
```
128+
129+
### Extraset
130+
131+
```@docs
132+
SVHN2.extratensor
133+
SVHN2.extralabels
134+
SVHN2.extradata
135+
```
136+
137+
### Utilities
138+
139+
```@docs
140+
SVHN2.download
141+
SVHN2.classnames
142+
SVHN2.convert2features
143+
SVHN2.convert2image
144+
```
145+
146+
## References
147+
148+
- **Authors**: Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng
149+
150+
- **Website**: http://ufldl.stanford.edu/housenumbers
151+
152+
- **[Netzer et al., 2011]** Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng. "Reading Digits in Natural Images with Unsupervised Feature Learning" NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011

‎docs/src/index.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,9 @@ Dataset | Classes | `traintensor` | `trainlabels` | `testtensor` | `testlabels`
7373
[**FashionMNIST**](@ref FashionMNIST) | 10 | 28x28x60000 | 60000 | 28x28x10000 | 10000
7474
[**CIFAR-10**](@ref CIFAR10) | 10 | 32x32x3x50000 | 50000 | 32x32x3x10000 | 10000
7575
[**CIFAR-100**](@ref CIFAR100) | 100 (20) | 32x32x3x50000 | 50000 (x2) | 32x32x3x10000 | 10000 (x2)
76+
[**SVHN-2**](@ref SVHN2) (*) | 10 | 32x32x3x73257 | 73257 | 32x32x3x26032 | 26032
77+
78+
(*) Note that the SVHN-2 dataset provides an additional 531131 observations aside from the training- and testset
7679

7780
### Language Modeling
7881

‎src/MLDatasets.jl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ include("CIFAR10/CIFAR10.jl")
1616
include("CIFAR100/CIFAR100.jl")
1717
include("MNIST/MNIST.jl")
1818
include("FashionMNIST/FashionMNIST.jl")
19+
include("SVHN2/SVHN2.jl")
1920
include("PTBLM/PTBLM.jl")
2021
include("UD_English/UD_English.jl")
2122

‎src/SVHN2/SVHN2.jl

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
export SVHN2
2+
3+
"""
4+
The Street View House Numbers (SVHN) Dataset
5+
6+
- Authors: Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng
7+
- Website: http://ufldl.stanford.edu/housenumbers
8+
9+
SVHN was obtained from house numbers in Google Street View
10+
images. As such they are quite diverse in terms of orientation
11+
and image background. Similar to MNIST, SVHN has 10 classes (the
12+
digits 0-9), but unlike MNIST there is more data and the images
13+
are a little bigger (32x32 instead of 28x28) with an additional
14+
RGB color channel. The dataset is split up into three subsets:
15+
73257 digits for training, 26032 digits for testing, and 531131
16+
additional to use as extra training data.
17+
18+
## Interface
19+
20+
- [`SVHN2.traintensor`](@ref), [`SVHN2.trainlabels`](@ref), [`SVHN2.traindata`](@ref)
21+
- [`SVHN2.testtensor`](@ref), [`SVHN2.testlabels`](@ref), [`SVHN2.testdata`](@ref)
22+
- [`SVHN2.extratensor`](@ref), [`SVHN2.extralabels`](@ref), [`SVHN2.extradata`](@ref)
23+
24+
## Utilities
25+
26+
- [`SVHN2.download`](@ref)
27+
- [`SVHN2.classnames`](@ref)
28+
- [`SVHN2.convert2features`](@ref)
29+
- [`SVHN2.convert2image`](@ref)
30+
"""
31+
module SVHN2
32+
using DataDeps
33+
using MAT
34+
using ImageCore
35+
using ColorTypes
36+
using FixedPointNumbers
37+
using ..bytes_to_type
38+
using ..datafile
39+
using ..download_dep
40+
using ..download_docstring
41+
42+
export
43+
44+
traintensor,
45+
testtensor,
46+
extratensor,
47+
48+
trainlabels,
49+
testlabels,
50+
extralabels,
51+
52+
traindata,
53+
testdata,
54+
extradata,
55+
56+
convert2image,
57+
convert2features,
58+
59+
download
60+
61+
const DEPNAME = "SVHN2"
62+
const TRAINDATA = "train_32x32.mat"
63+
const TESTDATA = "test_32x32.mat"
64+
const EXTRADATA = "extra_32x32.mat"
65+
const CLASSES = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0]
66+
67+
"""
68+
download([dir]; [i_accept_the_terms_of_use])
69+
70+
Trigger the (interactive) download of the full dataset into
71+
"`dir`". If no `dir` is provided the dataset will be
72+
downloaded into "~/.julia/datadeps/$DEPNAME".
73+
74+
This function will display an interactive dialog unless
75+
either the keyword parameter `i_accept_the_terms_of_use` or
76+
the environment variable `DATADEPS_ALWAY_ACCEPT` is set to
77+
`true`. Note that using the data responsibly and respecting
78+
copyright/terms-of-use remains your responsibility.
79+
"""
80+
download(args...; kw...) = download_dep(DEPNAME, args...; kw...)
81+
82+
include("interface.jl")
83+
include("utils.jl")
84+
85+
function __init__()
86+
RegisterDataDep(
87+
DEPNAME,
88+
"""
89+
Dataset: The Street View House Numbers (SVHN) Dataset
90+
Authors: Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng
91+
Website: http://ufldl.stanford.edu/housenumbers
92+
Format: Cropped Digits (Format 2 on the website)
93+
Note: for non-commercial use only
94+
95+
[Netzer et al., 2011]
96+
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng
97+
"Reading Digits in Natural Images with Unsupervised Feature Learning"
98+
NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011
99+
100+
The dataset is split up into three subsets: 73257
101+
digits for training, 26032 digits for testing, and
102+
531131 additional to use as extra training data.
103+
104+
The files are available for download at the official
105+
website linked above. Note that using the data
106+
responsibly and respecting copyright remains your
107+
responsibility. For example the website mentions that
108+
the data is for non-commercial use only. Please read
109+
the website to make sure you want to download the
110+
dataset.
111+
""",
112+
"http://ufldl.stanford.edu/housenumbers/" .* [TRAINDATA, TESTDATA, EXTRADATA],
113+
"2fa3b0b79baf39de36ed7579e6947760e6241f4c52b6b406cabc44d654c13a50"
114+
)
115+
end
116+
end

0 commit comments

Comments
 (0)