You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: vignettes/datatable-fread-and-fwrite.Rmd
+32-21Lines changed: 32 additions & 21 deletions
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ date: "`r Sys.Date()`"
4
4
output:
5
5
markdown::html_format
6
6
vignette: >
7
-
%\VignetteIndexEntry{Importing data.table}
7
+
%\VignetteIndexEntry{Fast Read and Fast Write}
8
8
%\VignetteEngine{knitr::knitr}
9
9
\usepackage[utf8]{inputenc}
10
10
---
@@ -61,7 +61,7 @@ On Windows, command line tools like `grep` are available through various environ
61
61
62
62
#### 1.1.1 Reading directly from a text string
63
63
64
-
`fread()` can read data directly from a character string in R using the text argument. This is particularly handy for creating reproducible examples, testing code snippets, or working with data generated programmatically within your R session. Each line in the string should be separated by a newline character `\n`.
64
+
`fread()` can read data directly from a character string in R using the `text` argument. This is particularly handy for creating reproducible examples, testing code snippets, or working with data generated programmatically within your R session. Each line in the string should be separated by a newline character `\n`.
`fread()` can read data directly from web URLs by passing the URL as a character string to its `file` argument. This allows you to download and read data from the internet in one step.
74
+
`fread()` can read data directly from web URLs by passing the URL as a character string to its `file` argument.
75
+
This allows you to download and read data from the internet in one step.
@@ -86,7 +87,7 @@ In many cases, `fread()` can automatically detect and decompress files with comm
86
87
-`.gz` / `.bz2` (gzip / bzip2): Supported and works out of the box.
87
88
-`.zip` / `.tar` (ZIP / tar archives, single file): Supported—`fread()` will read the first file in the archive if only one file is present.
88
89
89
-
> Note: If there are multiple files in the archive, `fread()` will fail with an error.
90
+
**Note**: If there are multiple files in the archive, `fread()` will fail with an error.
90
91
91
92
### 1.2 Automatic separator and skip detection
92
93
@@ -108,11 +109,15 @@ You can explicitly tell fread whether a header exists using `header = TRUE` or `
108
109
109
110
**Skip Detection**
110
111
111
-
By default (`skip="auto"`), `fread` will automatically skip blank lines and comment lines (e.g., starting with `#`) before the data header. To manually skip a specific number of lines, use `skip=n`.
112
+
By default (`skip="auto"`), `fread` will automatically skip blank lines and comment lines (e.g., starting with `#`) before the data header.
113
+
To manually specify a different number of lines to skip, use
114
+
115
+
*`skip=n` to skip the first `n` lines.
116
+
*`skip="string"` to search for a line containing a substring (typically from the column names, like `skip="Date"`). Reading begins at the first matching line. This is useful for skipping metadata, or selecting sub-tables in multi-table files. This feature is inspired by the `read.xls` function in the `gdata` package.
112
117
113
118
### 1.3 High-Quality Automatic Column Type Detection
114
119
115
-
Many real-world datasets contain columns that are initially blank, zero-filled, or appear numeric but later contain characters. To handle such inconsistencies, `fread()`in `data.table`employs a robust column type detection strategy.
120
+
Many real-world datasets contain columns that are initially blank, zero-filled, or appear numeric but later contain characters. To handle such inconsistencies, `fread()` employs a robust column type detection strategy.
116
121
117
122
Since v1.10.5, `fread()` samples rows by reading blocks of contiguous rows from multiple equally spaced points across the file, including the start, middle, and end. The total number of rows sampled is chosen dynamically based on the file size and structure, and is typically around 10,000, but can be smaller or slightly larger. This wide sampling helps detect type changes that occur later in the data (e.g., `001` to `0A0` or blanks becoming populated).
118
123
@@ -131,7 +136,7 @@ The type for each column is inferred based on the lowest required type from the
131
136
This ensures:
132
137
133
138
- Single up-front allocation of memory using the correct type
134
-
- Avoidance of rereading the file or manually setting colClasses
139
+
- Avoidance of rereading the file or manually setting `colClasses`
135
140
- Improved speed and memory efficiency
136
141
137
142
**Out-of-Sample Type Exceptions**
@@ -142,7 +147,9 @@ All detection logic and any rereads are detailed when `verbose=TRUE` is enabled.
142
147
143
148
### 1.4 Early Error Detection at End-of-File
144
149
145
-
Because the large sample explicitly includes the very end of the file, critical issues—such as an inconsistent number of columns, a malformed footer, or an opening quote without a matching closing quote—can be detected and reported almost instantly. This early error detection avoids the unnecessary overhead of processing the entire file or allocating excessive memory, only to encounter a failure at the final step. It ensures faster feedback and more efficient resource usage, especially when working with large datasets.
150
+
Because the large sample explicitly includes the very end of the file, critical issues—such as an inconsistent number of columns, a malformed footer, or an opening quote without a matching closing quote—can be detected and reported almost instantly.
151
+
This early error detection avoids the unnecessary overhead of processing the entire file or allocating excessive memory, only to encounter a failure at the final step.
152
+
It ensures faster feedback and more efficient resource usage, especially when working with large datasets.
146
153
147
154
### 1.5 `integer64` Support
148
155
@@ -185,21 +192,25 @@ Key points:
185
192
186
193
For details, see the manual page by running `?fread` in R.
187
194
188
-
### 1.7 Skip to a Sub-Table’s Header Row Using a Column Name Substring
189
-
190
-
Use `skip="string"` in `fread` to search for a line containing a substring (typically from the column names, e.g., `skip="Date"`). Reading begins at the first matching line. This is useful for skipping metadata or selecting sub-tables in multi-table files. This feature is inspired by the `read.xls` function in the gdata package.
`fread` automatically detects how quotes are escaped—including doubled ("") or backslash-escaped (\") quotes—without requiring user input. This is determined using a large sample of the data (see point 3), and validated against the entire file.
195
198
196
199
Supported Scenarios:
197
200
- Unescaped quotes inside quoted fields
198
-
e.g., `"This "quote" is invalid"` — supported as long as column count remains consistent.
201
+
e.g., `"This "quote" is invalid, but fread works anyway"` — supported as long as column count remains consistent :
202
+
203
+
```{r}
204
+
data.table::fread(text='x,y\n"This "quote" is invalid, but fread works anyway",1')
205
+
```
199
206
200
207
- Unquoted fields that begin with quotes
201
208
e.g., `Invalid"Field,10,20` — recognized correctly as not a quoted field.
202
209
210
+
```{r}
211
+
data.table::fread(text='x,y\nNot"Valid,1')
212
+
```
213
+
203
214
Requirements & Limitations:
204
215
- Escaping rules and column counts must be consistent throughout the file.
205
216
@@ -210,7 +221,8 @@ From v1.10.6, `fread` resolves ambiguities more reliably across the entire file
210
221
211
222
## 2. fwrite()
212
223
213
-
`fwrite()` is the fast file writer companion to `fread()`. It’s designed for speed, sensible defaults, and ease of use, mirroring many of the conveniences found in fread`.
224
+
`fwrite()` is the fast file writer companion to `fread()`.
225
+
It’s designed for speed, sensible defaults, and ease of use, mirroring many of the conveniences found in `fread`.
214
226
215
227
### 2.1 Intelligent and Minimalist Quoting (quote="auto")
**Full Precision for Large Integers**: `fwrite` writes `bit64::integer64` columns by converting them to strings with full precision. This prevents data loss or silent conversion to double that might occur with less specialized writers. This is crucial for IDs or measurements requiring more than R's standard `32-bit` integer range or `53-bit` double precision.
257
269
@@ -263,14 +275,13 @@ if (requireNamespace("bit64", quietly = TRUE)) {
263
275
temp_i64_out = tempfile(fileext = ".csv")
264
276
fwrite(dt_i64, temp_i64_out)
265
277
cat(readLines(temp_i64_out), sep = "\n")
266
-
267
278
unlink(temp_i64_out)
268
279
}
269
280
```
270
281
271
282
### 2.4 Column Order and Subset Control
272
283
273
-
To control the order and subset of columns written to file, subset the data.table before calling `fwrite()`. The `col.names` argument in `fwrite()` is a logical (TRUE/FALSE) that controls whether the header row is written, not which columns are written.
284
+
To control the order and subset of columns written to file, subset the `data.table` before calling `fwrite()`. The `col.names` argument in `fwrite()` is a logical (TRUE/FALSE) that controls whether the header row is written, not which columns are written.
274
285
275
286
```{r}
276
287
dt = data.table(A = 1:3, B = 4:6, C = 7:9)
@@ -283,7 +294,7 @@ file.remove("out.csv")
283
294
284
295
## 3. A Note on Performance
285
296
286
-
While this vignette focuses on features and usability, the primary motivation for `fread` and `fwrite` is speed. The performance of `data.table`'s I/O is a topic of continuous benchmarking.
297
+
While this vignette focuses on features and usability, the primary motivation for `fread` and `fwrite` is speed.
287
298
288
299
For users interested in detailed, up-to-date performance comparisons, we recommend these external blog posts which use the `atime` package for rigorous analysis:
289
300
@@ -292,4 +303,4 @@ For users interested in detailed, up-to-date performance comparisons, we recomme
292
303
293
304
These benchmarks consistently show that `fread` and `fwrite` are highly competitive and often state-of-the-art for performance in the R ecosystem.
0 commit comments