added fwrite and fread vignette (#7216)

Author: venom1204

man/fread.Rd

@@ -118,6 +118,15 @@ Currently, the \code{yaml} setting is somewhat inflexible with respect to incorp
 
 When \code{input} begins with http://, https://, ftp://, ftps://, or file://, \code{fread} detects this and \emph{downloads} the target to a temporary file (at \code{tempfile()}) before proceeding to read the file as usual. URLS (ftps:// and https:// as well as ftp:// and http://) paths are downloaded with \code{download.file} and \code{method} set to \code{getOption("download.file.method")}, defaulting to \code{"auto"}; and file:// is downloaded with \code{download.file} with \code{method="internal"}. NB: this implies that for file://, even files found on the current machine will be "downloaded" (i.e., hard-copied) to a temporary file. See \code{\link{download.file}} for more details.
 
+\bold{Automatic Decompression:}
+
+In many cases, \code{fread} can automatically detect and decompress files with common compression extensions directly, without needing an explicit connection object or shell commands. This works by checking the file extension.
+
+  \itemize{
+    \item \code{.gz} and \code{.bz2} are supported out of the box.
+    \item \code{.zip} is also supported. If the archive contains a single data file, \code{fread} will read it. If the archive contains multiple files, \code{fread} will produce an error.
+  }
+
 \bold{Shell commands:}
 
 \code{fread} accepts shell commands for convenience. The input command is run and its output written to a file in \code{tmpdir} (\code{\link{tempdir}()} by default) to which \code{fread} is applied "as normal". The details are platform dependent -- \code{system} is used on UNIX environments, \code{shell} otherwise; see \code{\link[base]{system}}.

vignettes/datatable-fread-and-fwrite.Rmd

@@ -0,0 +1,295 @@
+---
+title: "Fast Read and Fast Write"
+date: "`r Sys.Date()`"
+output: 
+  markdown::html_format
+vignette: >
+  %\VignetteIndexEntry{Importing data.table}
+  %\VignetteEngine{knitr::knitr}
+  \usepackage[utf8]{inputenc}
+---
+
+```{r echo=FALSE, file='_translation_links.R'}
+```
+`r .write.translation.links("Translations of this document are available in: %s")`
+
+```{r, echo = FALSE, message = FALSE}
+require(data.table)
+knitr::opts_chunk$set(
+  comment = "#",
+    error = FALSE,
+     tidy = FALSE,
+    cache = FALSE,
+ collapse = TRUE)
+.old.th = setDTthreads(1)
+```
+
+The `fread()` and `fwrite()` functions in the `data.table` R package are not only optimized for speed on large files, but also offer powerful and convenient features for working with small datasets. This vignette highlights their usability, flexibility, and performance for efficient data import and export.
+
+***
+
+## 1. fread()
+
+### **1.1 Using command line tools directly**
+The `fread()` function from `data.table` can read data piped from shell commands, letting you filter or preprocess data before it even enters R.
+
+```{r}
+# Create a sample file with some unwanted lines
+writeLines(
+'HEADER: Some metadata
+HEADER: More metadata
+1 2.0 3.0
+2 4.5 6.7
+HEADER: Yet more
+3 8.9 0.1
+4 1.2 3.4',
+"example_data.txt")
+
+library(data.table)
+fread("grep -v HEADER example_data.txt")
+```
+
+The `-v` option makes `grep` return all lines except those containing the string 'HEADER'.
+
+> "Given the number of high quality engineers that have looked at the command tool grep over the years, it is most likely that it is as fast as you can get, as well as being correct, convenient, well documented online, easy to learn and search for solutions for specific tasks. If you need to perform more complex string filtering (e.g., matching strings at the beginning or end of lines), the grep syntax is very powerful. Learning its syntax is a transferable skill for other languages and environments."
+>
+> — Matt Dowle  
+
+Look at this [example](https://stackoverflow.com/questions/36256706/fread-together-with-grepl/36270543#36270543) for more detail.
+
+On Windows, command line tools like `grep` are available through various environments, such as Rtools, Cygwin, or the Windows Subsystem for Linux (WSL). On Linux and macOS, these tools are typically included with the operating system.
+
+#### 1.1.1  Reading directly from a text string
+
+`fread()` can read data directly from a character string in R using the text argument. This is particularly handy for creating reproducible examples, testing code snippets, or working with data generated programmatically within your R session. Each line in the string should be separated by a newline character `\n`.
+
+```{r}
+my_data_string = "colA,colB,colC\n1,apple,TRUE\n2,banana,FALSE\n3,orange,TRUE"
+dt_from_text = fread(text = my_data_string)
+print(dt_from_text)
+```
+
+#### 1.1.2 Reading from URLs
+
+`fread()` can read data directly from web URLs by passing the URL as a character string to its `file` argument. This allows you to download and read data from the internet in one step.
+
+  ```{r}
+  # dt = fread("https://people.sc.fsu.edu/~jburkardt/data/csv/airtravel.csv")
+  # print(dt)
+  ```
+
+#### 1.1.3 Automatic decompression of compressed files
+
+In many cases, `fread()` can automatically detect and decompress files with common compression extensions directly, without needing an explicit connection object or shell commands. This works by checking the file extension.
+
+**Supported extensions typically include:**
+- `.gz` / `.bz2` (gzip / bzip2): Supported and works out of the box.
+- `.zip` / `.tar`  (ZIP / tar archives, single file): Supported—`fread()` will read the first file in the archive if only one file is present.
+
+> Note: If there are multiple files in the archive, `fread()` will fail with an error.
+
+### 1.2 Automatic separator and skip detection
+
+`fread` automates delimiter and header detection, eliminating the need for manual specification in most cases. You simply provide the filename—`fread` intelligently detects the structure:
+
+**Separator Detection**
+
+`fread` tests common separators (`,`,`\t`, `|`, space, `:`, `;`) and selects the one that results in the most consistent number of fields across sampled rows. For non-standard delimiters, you can override this using the `sep=` parameter.
+
+**Header Detection**
+
+After applying any `skip` or `nrows` settings (if specified), the first row with a consistent number of fields is examined:
+
+If all fields in this line are interpretable as character and the values do not strongly resemble a data row (e.g., a row of purely numeric-looking strings might still be considered data), it is typically used as the header (column names).
+
+Otherwise (e.g., if the line contains detected numeric types, or character strings that strongly resemble numbers and could be data), it is treated as a data row, and default column names (`V1`, `V2`, …) are assigned.
+
+You can explicitly tell fread whether a header exists using `header = TRUE` or `header = FALSE`.
+
+**Skip Detection**
+
+By default (`skip="auto"`), `fread` will automatically skip blank lines and comment lines (e.g., starting with `#`) before the data header. To manually skip a specific number of lines, use `skip=n`.
+
+### 1.3 High-Quality Automatic Column Type Detection
+
+Many real-world datasets contain columns that are initially blank, zero-filled, or appear numeric but later contain characters. To handle such inconsistencies, `fread()` in `data.table` employs a robust column type detection strategy.
+
+Since v1.10.5, `fread()` samples rows by reading blocks of contiguous rows from multiple equally spaced points across the file, including the start, middle, and end. The total number of rows sampled is chosen dynamically based on the file size and structure, and is typically around 10,000, but can be smaller or slightly larger. This wide sampling helps detect type changes that occur later in the data (e.g., `001` to `0A0` or blanks becoming populated).
+
+**Efficient File Access with mmap**
+
+To implement this sampling efficiently, `fread()` uses the operating system's memory-mapped file access (`mmap`), allowing it to jump to arbitrary positions in the file without sequential scanning. This lazy, on-demand strategy makes sampling nearly instantaneous, even for very large files.
+
+If a jump lands within a quoted field that includes newlines, `fread()` tests subsequent lines until it finds 5 consecutive rows with the expected number of fields, ensuring correct parsing even in complex files.
+
+**Accurate and Optimized Type Detection**
+
+The type for each column is inferred based on the lowest required type from the following ordered list:
+
+`logical` < `integer` < `integer64` < `double` < `character`
+
+This ensures:
+
+- Single up-front allocation of memory using the correct type
+- Avoidance of rereading the file or manually setting colClasses
+- Improved speed and memory efficiency
+
+**Out-of-Sample Type Exceptions**
+
+If a type change occurs outside the sampled rows, `fread()` automatically detects it and rereads the file to ensure correct type assignment, without requiring user intervention. For example, a column sampled as integer might later contain `00A` — triggering an automatic reread as character.
+
+All detection logic and any rereads are detailed when `verbose=TRUE` is enabled.
+
+### 1.4 Early Error Detection at End-of-File
+
+Because the large sample explicitly includes the very end of the file, critical issues—such as an inconsistent number of columns, a malformed footer, or an opening quote without a matching closing quote—can be detected and reported almost instantly. This early error detection avoids the unnecessary overhead of processing the entire file or allocating excessive memory, only to encounter a failure at the final step. It ensures faster feedback and more efficient resource usage, especially when working with large datasets.
+
+### 1.5 `integer64` Support
+
+By default, `fread` detects integers larger than 2<sup>31</sup> and reads them as `bit64::integer64` to preserve full precision. This behavior can be overridden in three ways:
+
+- Per-column: Use the `colClasses` argument to specify the type for individual columns.
+
+- Per-call: Use the `integer64` argument in `fread()` to set how all detected `integer64` columns are read.
+
+- Globally: Set the option `datatable.integer64` in your R session or `.Rprofile` file to change the default behavior for all fread calls.
+
+The integer64 argument (and corresponding option) accepts the following values:
+
+- `"integer64"` (default): Reads large integers as `bit64::integer64` with full precision.
+
+- `"double"` or `"numeric"`: Reads large integers as double-precision numbers, potentially losing precision silently (similar to `utils::read.csv` in base R).
+
+- `"character"`: Reads large integers as character strings.
+
+To check or set the global default, use:
+
+```{r}
+# fread's default behavior is to treat large integers as "integer64"; however, this global setting can be changed:
+options(datatable.integer64 = "double")   # Example: set globally to "double"
+getOption("datatable.integer64") 
+```
+
+### 1.6 Drop or Select Columns by Name or Position
+
+To save memory and improve performance, use `fread()`'s `select` or `drop` arguments to read only the columns you need.
+
+- If you need only a few columns, use `select`.
+- If you want to exclude just a few, use `drop`—this avoids listing everything you want to keep.
+
+Key points:
+- `select`: Vector of column names/positions to keep (discards others).
+- `drop`: Vector of column names/positions to discard (keeps others).
+- Do not use `select` and `drop` together—they are mutually exclusive.
+- `fread()` will warn you if any specified column is missing in the file.
+
+For details, see the manual page by running `?fread` in R.
+
+### 1.7 Skip to a Sub-Table’s Header Row Using a Column Name Substring
+
+Use `skip="string"` in `fread` to search for a line containing a substring (typically from the column names, e.g., `skip="Date"`). Reading begins at the first matching line. This is useful for skipping metadata or selecting sub-tables in multi-table files. This feature is inspired by the `read.xls` function in the gdata package.
+
+### 1.8 Automatic Quote Escape Detection (Including No-Escape)
+
+`fread` automatically detects how quotes are escaped—including doubled ("") or backslash-escaped (\") quotes—without requiring user input. This is determined using a large sample of the data (see point 3), and validated against the entire file.
+
+Supported Scenarios:
+- Unescaped quotes inside quoted fields
+e.g., `"This "quote" is invalid"` — supported as long as column count remains consistent.
+
+- Unquoted fields that begin with quotes
+e.g., `Invalid"Field,10,20` — recognized correctly as not a quoted field.
+
+Requirements & Limitations:
+- Escaping rules and column counts must be consistent throughout the file.
+
+- Not supported when `fill=TRUE` — in that case, the file must follow RFC4180-compliant quoting/escaping.
+
+Version-Specific Robustness:
+From v1.10.6, `fread` resolves ambiguities more reliably across the entire file using full-column-count consistency (default is `fill=FALSE`). Warnings are issued if parsing fails due to ambiguity.
+
+## 2. fwrite()
+
+`fwrite()` is the fast file writer companion to `fread()`. It’s designed for speed, sensible defaults, and ease of use, mirroring many of the conveniences found in fread`.
+
+### 2.1 Intelligent and Minimalist Quoting (quote="auto")
+
+When data is written as strings (either inherently, like character columns, or by choice, like `dateTimeAs="ISO"`), `quote="auto"` (default) intelligently quotes fields:
+
+**Contextual Quoting**:Fields are quoted only when necessary. This happens if they contain the delimiter `(sep)`, a double quote `(")`, a newline `(\n)`, a carriage return `(\r)`, or if the field is an empty string `("")`. Quoting the empty string is done to distinguish it from an NA value when the file is read.
+
+**Bypassed for Direct Numeric Output**: If specific columns are written as their underlying numeric types (e.g., via `dateTimeAs="epoch"` for `POSIXct`, or if a user pre-converts Date to integer), then quoting logic is naturally bypassed for those numeric fields, contributing to efficiency.
+
+```{r}
+dt_quoting_scenario = data.table(
+  text_field = c("Contains,a,comma", "Contains \"a quote\"", "Clean_text", "", NA),
+  numeric_field = 1:5
+)
+temp_quote_adv = tempfile(fileext = ".csv")
+
+fwrite(dt_quoting_scenario, temp_quote_adv)
+# Note the output: the empty string is quoted (""), but the NA is not.
+cat(readLines(temp_quote_adv), sep = "\n")
+```
+
+### 2.2 Fine-Grained Date/Time Serialization (dateTimeAs)
+
+Offers precise control for POSIXct/Date types:
+
+- `dateTimeAs="ISO"` (Default for POSIXct): ISO 8601 format (e.g., YYYY-MM-DDTHH:MM:SS.ffffffZ), preserving sub-second precision for unambiguous interchange.
+
+- `dateTimeAs="epoch"`: POSIXct as seconds since epoch (numeric).
+
+```{r}
+dt_timestamps = data.table(
+  ts = as.POSIXct("2023-10-26 14:35:45.123456", tz = "GMT"),
+  dt = as.Date("2023-11-15")
+)
+temp_dt_iso = tempfile(fileext = ".csv")
+fwrite(dt_timestamps, temp_dt_iso, dateTimeAs = "ISO")
+cat(readLines(temp_dt_iso), sep = "\n")
+unlink(temp_dt_iso)
+```
+
+### 2.3 Handling of bit64::integer64
+
+**Full Precision for Large Integers**: `fwrite` writes `bit64::integer64` columns by converting them to strings with full precision. This prevents data loss or silent conversion to double that might occur with less specialized writers. This is crucial for IDs or measurements requiring more than R's standard `32-bit` integer range or `53-bit` double precision.
+
+**Direct Handling**: This direct and careful handling of specialized numerics ensures data integrity and efficient I/O, without unnecessary intermediate conversions to less precise types.
+
+```{r}
+if (requireNamespace("bit64", quietly = TRUE)) {
+  dt_i64 = data.table(uid = bit64::as.integer64("1234567890123456789"), val = 100)
+  temp_i64_out = tempfile(fileext = ".csv")
+  fwrite(dt_i64, temp_i64_out)
+  cat(readLines(temp_i64_out), sep = "\n")
+
+  unlink(temp_i64_out)
+}
+```
+
+### 2.4 Column Order and Subset Control
+
+To control the order and subset of columns written to file, subset the data.table before calling `fwrite()`. The `col.names` argument in `fwrite()` is a logical (TRUE/FALSE) that controls whether the header row is written, not which columns are written.
+
+```{r}
+dt = data.table(A = 1:3, B = 4:6, C = 7:9)
+
+# Write only columns C and A, in that order
+fwrite(dt[, .(C, A)], "out.csv")
+cat(readLines("out.csv"), sep = "\n")
+file.remove("out.csv")
+```
+
+## 3. A Note on Performance
+
+While this vignette focuses on features and usability, the primary motivation for `fread` and `fwrite` is speed. The performance of `data.table`'s I/O is a topic of continuous benchmarking.
+
+For users interested in detailed, up-to-date performance comparisons, we recommend these external blog posts which use the `atime` package for rigorous analysis:
+
+- **[data.table asymptotic timings](https://tdhock.github.io/blog/2023/dt-atime-figures/)**: Compares `fread` and `fwrite` performance against other popular R packages like `readr` and `arrow`.
+- **[Benchmarking data.table with polars, duckdb, and pandas](https://tdhock.github.io/blog/2024/pandas-dt/)**: Compares `data.table` I/O and grouping performance against leading Python libraries.
+
+These benchmarks consistently show that `fread` and `fwrite` are highly competitive and often state-of-the-art for performance in the R ecosystem.
+
+***