Skip to content

Conversation

Copy link

Copilot AI commented Jan 20, 2026

Custom adapters fail with broken pipe errors when processing files that generate large output (e.g., XLSX files with multiple PNGs, pivot tables, or SmartArt). The underlying binaries work correctly when invoked directly.

Root Cause

pipe_output chains I/O sequentially: read stdout → wait for process → wait for stdin copy. When subprocess produces output faster than consuming stdin, stdout buffer fills, subprocess blocks on write, stdin copy blocks on read → deadlock → broken pipe.

Changes

  • Make stdin copy concurrent: Spawn stdin copy task independently via tokio::spawn instead of chaining with join_handle_to_stream
  • Remove sequential dependency: Return stdout immediately chained only with proc_wait, allowing concurrent I/O
  • Add debug logging: Log stdin copy errors with subprocess context (errors are expected when subprocess exits early)
// Before: Sequential chain causes deadlock
let join = tokio::spawn(async move {
    tokio::io::copy(&mut z, &mut stdi).await?;
    std::io::Result::Ok(())
});
Ok(Box::pin(stdo.chain(proc_wait(cmd, ...).chain(join_handle_to_stream(join)))))

// After: Concurrent execution prevents deadlock
tokio::spawn(async move {
    if let Err(e) = tokio::io::copy(&mut z, &mut stdi).await {
        debug!("stdin copy for {:?} ended: {}", cmd_log_copy, e);
    }
});
Ok(Box::pin(stdo.chain(proc_wait(cmd, ...))))

Subprocess exit status remains validated via proc_wait. Stdin copy errors are logged but not propagated (subprocess closing stdin early is expected behavior).

Original prompt

This section details on the original issue you should resolve

<issue_title>problems with XLSX files containing PNGs, pivot tables or SmartArt</issue_title>
<issue_description>Describe the bug
I have discovered that the XLSX custom adapters in ripgrep-all do not work reliably with some Excel files, specifically those containing SmartArt, pivot tables, or more than one (sometimes two) embedded PNG images.

The binaries that power these custom XLSX adapters (e.g. xlsx2csv, in2csv, and my own custom testing adapter based on xlsxgrep) work correctly when used independently on the same files. This suggests that the issue may lie in rga-preproc, possibly in how it handles adapter output or how that output is passed between the adapters and ripgrep-all itself.

Some of my observations:

  • ripgrep-all threw the following error when using any of my three custom XLSX adapters on Excel files that contained additional custom Excel metadata:
rg: Downloads/PNG_Excel.xlsx: preprocessor command failed: '"/opt/homebrew/bin/rga-preproc" "Downloads/PNG_Excel.xlsx"': 
-------------------------------------------------------------------------------
/Users/Downloads/PNG_Excel.xlsx adapter: in2csv_xlsx
/Users/Downloads/PNG_Excel.xlsx.txt adapter: postprocprefix
Error: copying adapter output to stdout

Caused by:
    Broken pipe (os error 32)
-------------------------------------------------------------------------------
  • The in2csv-based adapter triggered the same error on Excel files containing more than one PNG image.
  • The xlsx2csv-based adapter failed on Excel files with pivot tables.
  • Both adapters failed on Excel files containing SmartArt.

In all cases, running the same binaries manually (outside of ripgrep-all) on the problematic files produced correct and complete output, without errors.

To Reproduce
Run rga with the custom XLSX adapters listed here: https://github.com/phiresky/ripgrep-all/discussions/categories/show-your-adapter on an Excel file containing SmartArt, pivot tables, or multiple embedded PNG images.

Run command:
rga --rga-adapters=xlsx2csv -il . "ProblematicFile.xlsx"

Output

rg: Downloads/PNG_Excel.xlsx: preprocessor command failed: '"/opt/homebrew/bin/rga-preproc" "Downloads/PNG_Excel.xlsx"': 
-------------------------------------------------------------------------------
/Users/Downloads/PNG_Excel.xlsx adapter: in2csv_xlsx
/Users/Downloads/PNG_Excel.xlsx.txt adapter: postprocprefix
Error: copying adapter output to stdout

Caused by:
    Broken pipe (os error 32)
-------------------------------------------------------------------------------

Screenshots
If applicable, add screenshots to help explain your problem.

Operating System and Version
macOS Tahoe 26.0.1 arm64

Output of rga --version
ripgrep-all 0.10.9</issue_description>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI changed the title [WIP] Fix issues with XLSX files containing PNGs and pivot tables Fix deadlock in custom adapter pipe_output causing broken pipe errors Jan 20, 2026
Copilot AI requested a review from phiresky January 20, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

problems with XLSX files containing PNGs, pivot tables or SmartArt

2 participants