Skip to content

Commit 8957458

Browse files
Martin Hsuehmeta-codesync[bot]
authored andcommitted
refactor verifiable matcher btd to explicitly use file reader for reading file
Summary: previously.. json::read_file_lines_parallel implictely reads and decompress the file. This wasn't very obvious (2 of us missed this) We just want to refactor so it's clear that BTD already takes care of decompressing file before calling json decoder. Differential Revision: D90789486 fbshipit-source-id: 33cbb06b5edaff0a11bc7b9d62de23688f27205b
1 parent ddb518e commit 8957458

File tree

1 file changed

+12
-0
lines changed

1 file changed

+12
-0
lines changed

td_util/src/json.rs

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,18 @@ pub fn read_file_lines_par_iter<T: for<'a> Deserialize<'a> + Send>(
8989
Ok(rdr.lines().par_bridge().map(parse_line::<T>))
9090
}
9191

92+
/// Read JSON lines from a reader. The order of entries is not guaranteed.
93+
/// Use this when you want explicit control over file opening (e.g., with file_io::file_reader).
94+
pub fn read_reader_lines_parallel<T: for<'a> Deserialize<'a> + Send>(
95+
reader: impl BufRead + Send,
96+
) -> anyhow::Result<Vec<T>> {
97+
reader
98+
.lines()
99+
.par_bridge()
100+
.map(parse_line::<T>)
101+
.collect::<anyhow::Result<Vec<T>>>()
102+
}
103+
92104
/// Read a file that consists of many JSON blobs, one per line.
93105
pub fn read_file_lines<T: for<'a> Deserialize<'a>>(filename: &Path) -> anyhow::Result<Vec<T>> {
94106
fn f<T: for<'a> Deserialize<'a>>(filename: &Path) -> anyhow::Result<Vec<T>> {

0 commit comments

Comments
 (0)