Skip to content

Commit ebfdf62

Browse files
bicycle1885bicycle1885
committed
add more docs
1 parent b031bc2 commit ebfdf62

File tree

5 files changed

+162
-61
lines changed

5 files changed

+162
-61
lines changed

‎README.md

Lines changed: 3 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -1,69 +1,11 @@
1-
TranscodingStreams.jl - IO streams with transcoding
2-
===================================================
1+
TranscodingStreams.jl
2+
=====================
33

44
<!--[![Docs Latest][docs-latest-img]][docs-latest-url]-->
5-
[![TravisCI Status][travisci-img]][travisci-url]
65
<!--[![Appveyor Status][appveyor-img]][appveyor-url]-->
6+
[![TravisCI Status][travisci-img]][travisci-url]
77
[![codecov.io][codecov-img]][codecov-url]
88

9-
TranscodingStreams.jl exports a type `TranscodingStream{C<:Codec,S<:IO}<:IO`
10-
which wraps `S<:IO` using `C<:Codec`. Codecs are defined in other packages
11-
listed below:
12-
13-
- [CodecZlib.jl](https://github.com/bicycle1885/CodecZlib.jl)
14-
- `GzipCompression` (`GzipCompressionStream`)
15-
- `GzipDecompression` (`GzipDecompressionStream`)
16-
- `ZlibCompression` (`ZlibCompressionStream`)
17-
- `ZlibDecompression` (`ZlibDecompressionStream`)
18-
- `DeflateCompression` (`DeflateCompressionStream`)
19-
- `DeflateDecompression` (`DeflateDecompressionStream`)
20-
- [CodecZstd.jl](https://github.com/bicycle1885/CodecZstd.jl)
21-
- `ZstdCompression` (`ZstdCompressionStream`)
22-
- `ZstdDecompression` (`ZstdDecompressionStream`)
23-
- [CodecBzip2.jl](https://github.com/bicycle1885/CodecBzip2.jl)
24-
- `Bzip2Compression` (`Bzip2CompressionStream`)
25-
- `Bzip2Decompression` (`Bzip2DecompressionStream`)
26-
27-
By convention, codec types have a name that matches `.*(Co|Deco)mpression` and
28-
I/O types have a codec name with `Stream` suffix. The following snippet is an
29-
example of using CodecZlib.jl, which exports `GzipDecompressionStream` as an
30-
alias of `TranscodingStream{GzipDecompression,S}` where `S<:IO`:
31-
```julia
32-
# Read lines from a gzip-compressed file.
33-
using CodecZlib
34-
stream = GzipDecompressionStream(open("data.gzip"))
35-
for line in eachline(stream)
36-
# do something...
37-
end
38-
close(stream)
39-
```
40-
41-
When writing data, the end of a data stream is indicated by calling `close`,
42-
which may write an epilogue if necessary and flush all buffered data to the
43-
underlying I/O stream. If you want to explicitly specify the end position of a
44-
stream for some reason, you can write `TranscodingStreams.TOKEN_END` to the
45-
transcoding stream as follows:
46-
```julia
47-
# Compress a string using zstd.
48-
using CodecZstd
49-
using TranscodingStreams
50-
buf = IOBuffer()
51-
stream = ZstdCompressionStream(buf)
52-
write(stream, "foobarbaz"^100, TranscodingStreams.TOKEN_END)
53-
flush(stream)
54-
compressed = take!(buf)
55-
close(stream)
56-
```
57-
58-
TranscodingStreams.jl extends the `transcode` function to transcode a data
59-
in one shot. `transcode` takes a codec object as its first argument and a data
60-
vector as its second argument:
61-
```julia
62-
using CodecZlib
63-
decompressed = transcode(ZlibDecompression(), b"x\x9cKL*JLNLI\x04R\x00\x19\xf2\x04U")
64-
String(decompressed)
65-
```
66-
679
[travisci-img]: https://travis-ci.org/bicycle1885/TranscodingStreams.jl.svg?branch=master
6810
[travisci-url]: https://travis-ci.org/bicycle1885/TranscodingStreams.jl
6911
[codecov-img]: http://codecov.io/github/bicycle1885/TranscodingStreams.jl/coverage.svg?branch=master

‎docs/make.jl

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
using Documenter
2+
using TranscodingStreams
3+
4+
makedocs(
5+
format=:html,
6+
sitename="TranscodingStreams.jl",
7+
modules=[TranscodingStreams])
8+
9+
deploydocs(
10+
repo="github.com/bicycle1885/TranscodingStreams.jl.git",
11+
julia="0.6",
12+
target="build",
13+
deps=nothing,
14+
make=nothing)

‎docs/src/index.md

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
TranscodingStreams.jl
2+
=====================
3+
4+
TranscodingStreams.jl is a package for transcoding (e.g. compression) data
5+
streams. This package exports a type `TranscodingStream`, which
6+
is a subtype of `IO` and supports various I/O operations like other usual I/O
7+
streams in the standard library.
8+
9+
10+
Introduction
11+
------------
12+
13+
`TranscodingStream` has two type parameters, `C<:Codec` and `S<:IO`, and hence
14+
the actual type should be written as `TranscodingStream{C<:Codec,S<:IO}`. This
15+
type wraps an underlying I/O stream `S` by a codec `C`. The codec defines
16+
transformation (or transcoding) of the stream. For example, when `C` is a
17+
lossless decompression type and `S` is a file, `TranscodingStream{C,S}` behaves
18+
like a data stream that incrementally decompresses data from the file.
19+
20+
Codecs are defined in other packages listed below:
21+
- [CodecZlib.jl](https://github.com/bicycle1885/CodecZlib.jl)
22+
- `GizipCompression` (`GzipCompressionStream`)
23+
- `GzipDecompression` (`GzipDecompressionStream`)
24+
- `ZlibCompression` (`ZlibCompressionStream`)
25+
- `ZlibDecompression` (`ZlibDecompressionStream`)
26+
- `DeflateCompression` (`DeflateCompressionStream`)
27+
- `DeflateDecompression` (`DeflateDecompressionStream`)
28+
- [CodecZngstd.jl](https://github.com/bicycle1885/CodecZstd.jl)
29+
- `ZstdCompression` (`ZstdCompressionStream`)
30+
- `ZstdDecompression` (`ZstdDecompressionStream`)
31+
- [CodecBzip2.jl](https://github.com/bicycle1885/CodecBzip2.jl)
32+
- `Bzip2Compression` (`Bzip2CompressionStream`)
33+
- `Bzip2Decompression` (`Bzip2DecompressionStream`)
34+
35+
By convention, codec types have a name that matches `.*(Co|Deco)mpression` and
36+
I/O types have a codec name with `Stream` suffix. An important thing is these
37+
packages depend on TranscodingStreams.jl and not *vice versa*. This means you
38+
can install any codec package you need without installing all codec packages.
39+
Also, if you want to define your own codec, it is totally feasible like these
40+
packages. TranscodingStreams.jl requests a codec to implement some interface
41+
functions which will be described later.
42+
43+
44+
Examples
45+
--------
46+
47+
### Read lines from a gzip-compressed file
48+
49+
The following snippet is an example of using CodecZlib.jl, which exports
50+
`GzipDecompressionStream{S}` as an alias of
51+
`TranscodingStream{GzipDecompression,S} where S<:IO`:
52+
```julia
53+
using CodecZlib
54+
stream = GzipDecompressionStream(open("data.gzip"))
55+
for line in eachline(stream)
56+
# do something...
57+
end
58+
close(stream)
59+
```
60+
61+
### Save a data matrix with Zstd compression
62+
63+
Writing compressed data is easy. One thing you need to keep in mind is to call
64+
`close` after writing data; otherwise, the output file will be incomplete:
65+
```julia
66+
using CodecZstd
67+
mat = randn(100, 100).^2
68+
file = open("data.mat.zst", "w")
69+
stream = ZstdCompressionStream(file)
70+
writedlm(stream, mat)
71+
close(stream)
72+
```
73+
74+
### Explicitly finish transcoding by writing `TOKEN_END`
75+
76+
When writing data, the end of a data stream is indicated by calling `close`,
77+
which may write an epilogue if necessary and flush all buffered data to the
78+
underlying I/O stream. If you want to explicitly specify the end position of a
79+
stream for some reason, you can write `TranscodingStreams.TOKEN_END` to the
80+
transcoding stream as follows:
81+
```julia
82+
using CodecZstd
83+
using TranscodingStreams
84+
buf = IOBuffer()
85+
stream = ZstdCompressionStream(buf)
86+
write(stream, "foobarbaz"^100, TranscodingStreams.TOKEN_END)
87+
flush(stream)
88+
compressed = take!(buf)
89+
close(stream)
90+
```
91+
92+
### Transcode data in one shot
93+
94+
TranscodingStreams.jl extends the `transcode` function to transcode a data
95+
in one shot. `transcode` takes a codec object as its first argument and a data
96+
vector as its second argument:
97+
```julia
98+
using CodecZlib
99+
decompressed = transcode(ZlibDecompression(), b"x\x9cKL*JLNLI\x04R\x00\x19\xf2\x04U")
100+
String(decompressed)
101+
```
102+
103+
104+
API
105+
---
106+
107+
```@meta
108+
CurrentModule = TranscodingStreams
109+
```
110+
111+
```@docs
112+
TranscodingStream(codec::Codec, stream::IO)
113+
transcode(codec::Codec, data::Vector{UInt8})
114+
TranscodingStreams.TOKEN_END
115+
```
116+
117+
118+
Defining a new codec
119+
--------------------
120+
121+
```@docs
122+
TranscodingStreams.Codec
123+
TranscodingStreams.initialize
124+
TranscodingStreams.finalize
125+
TranscodingStreams.startproc
126+
TranscodingStreams.process
127+
```

‎src/stream.jl

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,24 @@ const DEFAULT_BUFFER_SIZE = 16 * 2^10 # 16KiB
4141
TranscodingStream(codec::Codec, stream::IO; bufsize::Integer=$(DEFAULT_BUFFER_SIZE))
4242
4343
Create a transcoding stream with `codec` and `stream`.
44+
45+
Examples
46+
--------
47+
48+
```jldoctest
49+
julia> using TranscodingStreams
50+
51+
julia> using CodecZlib
52+
53+
julia> file = open(Pkg.dir("TranscodingStreams", "test", "abra.gzip"));
54+
55+
julia> stream = TranscodingStream(GzipDecompression(), file)
56+
TranscodingStreams.TranscodingStream{CodecZlib.GzipDecompression,IOStream}(<state=idle>)
57+
58+
julia> readstring(stream)
59+
"abracadabra"
60+
61+
```
4462
"""
4563
function TranscodingStream(codec::Codec, stream::IO; bufsize::Integer=DEFAULT_BUFFER_SIZE)
4664
if bufsize ≤ 0

‎test/abra.gzip

29 Bytes
Binary file not shown.

0 commit comments

Comments
 (0)