Skip to content

Commit 7761bbf

Browse files
committed
some tweaks to clean up the diff
1 parent c2ca8db commit 7761bbf

File tree

2 files changed

+131
-84
lines changed

2 files changed

+131
-84
lines changed

README.md

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,10 @@ file format.
4040
Sometimes you want to read or write files that are larger than your available
4141
memory, or might be an unknown or infinite length (e.g. reading an audio or
4242
video stream from a socket). In these cases it might not make sense to process
43-
the whole file at once, but instead process it a chunk at a time. For these situations FileIO provides the `loadstreaming` and `savestreaming` functions, which return an object that you can `read` or `write`, rather than the file data itself.
43+
the whole file at once, but instead process it a chunk at a time. For these
44+
situations FileIO provides the `loadstreaming` and `savestreaming` functions,
45+
which return an object that you can `read` or `write`, rather than the file data
46+
itself.
4447

4548
This would look something like:
4649

@@ -69,7 +72,8 @@ do loadstreaming("bigfile.wav") audio
6972
end
7073
```
7174

72-
Note that in these cases you may want to use `read!` with a pre-allocated buffer for maximum efficiency.
75+
Note that in these cases you may want to use `read!` with a pre-allocated buffer
76+
for maximum efficiency.
7377

7478
## Adding new formats
7579

@@ -173,7 +177,13 @@ automatically even if the code inside the `do` scope throws an error.)
173177
Conversely, `load(::Stream)` and `save(::Stream)` should not close the
174178
input stream.
175179

176-
`loadstreaming` and `savestreaming` use the same query mechanism, but return a decoded stream that users can `read` or `write`. You should also implement a `close` method on your reader or writer type. Just like with `load` and `save`, if the user provided a filename, your `close` method should be responsible for closing any streams you opened in order to read or write the file. If you are given a `Stream`, your `close` method should only do the clean up for your reader or writer type, not close the stream.
180+
`loadstreaming` and `savestreaming` use the same query mechanism, but return a
181+
decoded stream that users can `read` or `write`. You should also implement a
182+
`close` method on your reader or writer type. Just like with `load` and `save`,
183+
if the user provided a filename, your `close` method should be responsible for
184+
closing any streams you opened in order to read or write the file. If you are
185+
given a `Stream`, your `close` method should only do the clean up for your
186+
reader or writer type, not close the stream.
177187

178188
```julia
179189
struct WAVReader

src/loadsave.jl

Lines changed: 118 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,52 @@ add_loader
4040
"`add_saver(fmt, :Package)` triggers `using Package` before saving format `fmt`"
4141
add_saver
4242

43+
"""
44+
- `load(filename)` loads the contents of a formatted file, trying to infer
45+
the format from `filename` and/or magic bytes in the file.
46+
- `load(strm)` loads from an `IOStream` or similar object. In this case,
47+
the magic bytes are essential.
48+
- `load(File(format"PNG",filename))` specifies the format directly, and bypasses inference.
49+
- `load(f; options...)` passes keyword arguments on to the loader.
50+
"""
51+
load
52+
53+
"""
54+
Some packages may implement a streaming API, where the contents of the file can
55+
be read in chunks and processed, rather than all at once. Reading from these
56+
higher-level streams should return a formatted object, like an image or chunk of
57+
video or audio.
58+
59+
- `loadstreaming(filename)` loads the contents of a formatted file, trying to infer
60+
the format from `filename` and/or magic bytes in the file. It returns a streaming
61+
type that can be read from in chunks, rather than loading the whole contents all
62+
at once
63+
- `loadstreaming(strm)` loads the stream from an `IOStream` or similar object. In this case,
64+
the magic bytes are essential.
65+
- `load(File(format"PNG",filename))` specifies the format directly, and bypasses inference.
66+
- `load(f; options...)` passes keyword arguments on to the loader.
67+
"""
68+
loadstreaming
69+
70+
"""
71+
- `save(filename, data...)` saves the contents of a formatted file,
72+
trying to infer the format from `filename`.
73+
- `save(Stream(format"PNG",io), data...)` specifies the format directly, and bypasses inference.
74+
- `save(f, data...; options...)` passes keyword arguments on to the saver.
75+
"""
76+
save
77+
78+
"""
79+
Some packages may implement a streaming API, where the contents of the file can
80+
be written in chunks, rather than all at once. These higher-level streams should
81+
accept formatted objects, like an image or chunk of video or audio.
82+
83+
- `savestreaming(filename, data...)` saves the contents of a formatted file,
84+
trying to infer the format from `filename`.
85+
- `savestreaming(Stream(format"PNG",io), data...)` specifies the format directly, and bypasses inference.
86+
- `savestreaming(f, data...; options...)` passes keyword arguments on to the saver.
87+
"""
88+
savestreaming
4389

4490
for fn in (:load, :loadstreaming, :save, :savestreaming)
4591
@eval $fn(s::@compat(Union{AbstractString,IO}), args...; options...) =
@@ -63,12 +109,14 @@ function savestreaming{sym}(df::Type{DataFormat{sym}}, s::IO, data...; options..
63109
checked_import(libraries[1])
64110
eval(Main, :($savestreaming($Stream($(DataFormat{sym}), $s),
65111
$data...; $options...)))
112+
end
66113

67114
function save{sym}(df::Type{DataFormat{sym}}, s::IO, data...; options...)
68115
libraries = applicable_savers(df)
69116
checked_import(libraries[1])
70117
eval(Main, :($save($Stream($(DataFormat{sym}), $s),
71118
$data...; $options...)))
119+
end
72120

73121
function savestreaming{sym}(df::Type{DataFormat{sym}}, f::AbstractString, data...; options...)
74122
libraries = applicable_savers(df)
@@ -82,104 +130,93 @@ for fn in (:loadstreaming, :savestreaming)
82130
@eval function $fn(f::Function, args...; kwargs...)
83131
str = $fn(args...; kwargs...)
84132
try
85-
ret = f(str)
133+
f(str)
86134
finally
87135
close(str)
88136
end
89-
90-
ret
91137
end
92138
end
93139

94140
# Fallbacks
95-
for fn in (:load, :loadstreaming)
96-
@eval function $fn{F}(q::Formatted{F}, args...; options...)
97-
if unknown(q)
98-
isfile(filename(q)) || open(filename(q)) # force systemerror
99-
throw(UnknownFormat(q))
100-
end
101-
libraries = applicable_loaders(q)
102-
failures = Any[]
103-
for library in libraries
104-
try
105-
Library = checked_import(library)
106-
if !has_method_from(methods(Library.$fn), Library)
107-
throw(LoaderError(string(library), "$fn not defined"))
108-
end
109-
return eval(Main, :($(Library.$fn)($q, $args...; $options...)))
110-
catch e
111-
push!(failures, (e, q))
141+
142+
# TODO: this definitely should be refactored to reduce duplication
143+
function load{F}(q::Formatted{F}, args...; options...)
144+
if unknown(q)
145+
isfile(filename(q)) || open(filename(q)) # force systemerror
146+
throw(UnknownFormat(q))
147+
end
148+
libraries = applicable_loaders(q)
149+
failures = Any[]
150+
for library in libraries
151+
try
152+
Library = checked_import(library)
153+
if !has_method_from(methods(Library.load), Library)
154+
throw(LoaderError(string(library), "load not defined"))
112155
end
156+
return eval(Main, :($(Library.load)($q, $args...; $options...)))
157+
catch e
158+
push!(failures, (e, q))
113159
end
114-
handle_exceptions(failures, "loading \"$(filename(q))\"")
115160
end
161+
handle_exceptions(failures, "loading \"$(filename(q))\"")
116162
end
117-
for fn in (:save, :savestreaming)
118-
@eval function $fn{F}(q::Formatted{F}, data...; options...)
119-
unknown(q) && throw(UnknownFormat(q))
120-
libraries = applicable_savers(q)
121-
failures = Any[]
122-
for library in libraries
123-
try
124-
Library = checked_import(library)
125-
if !has_method_from(methods(Library.$fn), Library)
126-
throw(WriterError(string(library), "$fn not defined"))
127-
end
128-
return eval(Main, :($(Library.$fn)($q, $data...; $options...)))
129-
catch e
130-
push!(failures, (e, q))
163+
164+
function loadstreaming{F}(q::Formatted{F}, args...; options...)
165+
if unknown(q)
166+
isfile(filename(q)) || open(filename(q)) # force systemerror
167+
throw(UnknownFormat(q))
168+
end
169+
libraries = applicable_loaders(q)
170+
failures = Any[]
171+
for library in libraries
172+
try
173+
Library = checked_import(library)
174+
if !has_method_from(methods(Library.loadstreaming), Library)
175+
throw(LoaderError(string(library), "loadstreaming not defined"))
131176
end
177+
return eval(Main, :($(Library.loadstreaming)($q, $args...; $options...)))
178+
catch e
179+
push!(failures, (e, q))
132180
end
133-
handle_exceptions(failures, "saving \"$(filename(q))\"")
134181
end
182+
handle_exceptions(failures, "opening \"$(filename(q))\" for streamed loading")
135183
end
136184

137-
"""
138-
- `load(filename)` loads the contents of a formatted file, trying to infer
139-
the format from `filename` and/or magic bytes in the file.
140-
- `load(strm)` loads from an `IOStream` or similar object. In this case,
141-
the magic bytes are essential.
142-
- `load(File(format"PNG",filename))` specifies the format directly, and bypasses inference.
143-
- `load(f; options...)` passes keyword arguments on to the loader.
144-
"""
145-
load
146-
147-
"""
148-
Some packages may implement a streaming API, where the contents of the file can
149-
be read in chunks and processed, rather than all at once. Reading from these
150-
higher-level streams should return a formatted object, like an image or chunk of
151-
video or audio.
152-
153-
- `loadstreaming(filename)` loads the contents of a formatted file, trying to infer
154-
the format from `filename` and/or magic bytes in the file. It returns a streaming
155-
type that can be read from in chunks, rather than loading the whole contents all
156-
at once
157-
- `loadstreaming(strm)` loads the stream from an `IOStream` or similar object. In this case,
158-
the magic bytes are essential.
159-
- `load(File(format"PNG",filename))` specifies the format directly, and bypasses inference.
160-
- `load(f; options...)` passes keyword arguments on to the loader.
161-
"""
162-
loadstreaming
163-
164-
"""
165-
- `save(filename, data...)` saves the contents of a formatted file,
166-
trying to infer the format from `filename`.
167-
- `save(Stream(format"PNG",io), data...)` specifies the format directly, and bypasses inference.
168-
- `save(f, data...; options...)` passes keyword arguments on to the saver.
169-
"""
170-
save
171-
172-
"""
173-
Some packages may implement a streaming API, where the contents of the file can
174-
be written in chunks, rather than all at once. These higher-level streams should
175-
accept formatted objects, like an image or chunk of video or audio.
185+
function save{F}(q::Formatted{F}, data...; options...)
186+
unknown(q) && throw(UnknownFormat(q))
187+
libraries = applicable_savers(q)
188+
failures = Any[]
189+
for library in libraries
190+
try
191+
Library = checked_import(library)
192+
if !has_method_from(methods(Library.save), Library)
193+
throw(WriterError(string(library), "save not defined"))
194+
end
195+
return eval(Main, :($(Library.save)($q, $data...; $options...)))
196+
catch e
197+
push!(failures, (e, q))
198+
end
199+
end
200+
handle_exceptions(failures, "saving \"$(filename(q))\"")
201+
end
176202

177-
- `savestreaming(filename, data...)` saves the contents of a formatted file,
178-
trying to infer the format from `filename`.
179-
- `savestreaming(Stream(format"PNG",io), data...)` specifies the format directly, and bypasses inference.
180-
- `savestreaming(f, data...; options...)` passes keyword arguments on to the saver.
181-
"""
182-
savestreaming
203+
function savestreaming{F}(q::Formatted{F}, data...; options...)
204+
unknown(q) && throw(UnknownFormat(q))
205+
libraries = applicable_savers(q)
206+
failures = Any[]
207+
for library in libraries
208+
try
209+
Library = checked_import(library)
210+
if !has_method_from(methods(Library.savestreaming), Library)
211+
throw(WriterError(string(library), "savestreaming not defined"))
212+
end
213+
return eval(Main, :($(Library.savestreaming)($q, $data...; $options...)))
214+
catch e
215+
push!(failures, (e, q))
216+
end
217+
end
218+
handle_exceptions(failures, "opening \"$(filename(q))\" for streamed saving")
219+
end
183220

184221
function has_method_from(mt, Library)
185222
for m in mt

0 commit comments

Comments
 (0)