You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: doc/API.md
+14-5Lines changed: 14 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,12 +4,19 @@ This page describes the interface of extended LZ4 implementation for the Linux K
4
4
5
5
## Standard LZ4
6
6
7
-
The standard implementation of LZ4 in the Linux Kernel only handles contiguous buffers, which has several downsides:
8
-
- if data is fragmented, it has to be copied into an allocated buffer;
9
-
- if data is big enough, allocating a contiguous buffer for it can be difficult.
7
+
The standard implementation of LZ4 in the Linux Kernel only handles contiguous buffers. For handling fragmented data there are several workarounds:
8
+
1) Copying the data into a preallocated contiguous buffer.
10
9
11
-
It becomes significant for I/O operations in particular, in which LZ4 is often used for its speed.
12
-
Also, failing to allocate a big enough chunk of memory would require running compression or decompression by parts, which brings unnecessary complexity.
10
+
This would require allocating a big enough chunk of memory, which could cause issues for large data.
11
+
Although this method is easy to implement, allocation followed by copying would take an effect on performance and memory consumption.
12
+
13
+
2) Running LZ4 for each of contiguous blocks.
14
+
15
+
When compressing a single block, we usually want to access as much of preceding data as possible to find matches.
16
+
In case of fragmented data we would use [LZ4 Streaming API](https://github.com/lz4/lz4/blob/dev/examples/streaming_api_basics.md).
17
+
The limitation here is that we are only able to use the previous block as a dictionary, and that can decrease the compression ratio.
18
+
19
+
Described downsides become significant for I/O operations in particular, in which LZ4 is often used for its speed.
13
20
14
21
Signature of the most commonly used [LZ4 function for compression](https://elixir.bootlin.com/linux/v6.16.9/source/include/linux/lz4.h#L175) is as follows:
15
22
```c
@@ -92,3 +99,5 @@ Input and maximum output sizes here are discovered through `srcIter` and `dstIte
92
99
For I/O, iterators can be created and modified before calling the function and are changed at
93
100
runtime to contain current position after the function is called.
94
101
Compression requires additional 1KB of working memory on top of existing 16KB, the exact size is in macro `LZ4E_MEM_COMPRESS`.
102
+
103
+
Described signatures and macros can be found at [lz4e.h](../lz4e/include/lz4e.h)
0 commit comments