Help tracking down a pandoc error: Cannot decode byte in a Quarto project

[friendly]

Description

Please help I'm getting ready to abandon Quarto.

In this StackOverflow post I describe a problem with pandoc (v. 2.19.2) under quarto (v. 1.1.189), and RStudio (RStudio 2022.07.2+576)where I get this error,

pandoc.exe: Cannot decode byte '\x93': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream

and have been persistently unable to find the location in my files where this might occur. As I noted in SO, I have checked all my .qmd, .R, '.bib, files using tools::showNonASCIIfile() and nothing turns up.

I also filed this as a pandoc issue. I updated pandoc to the latest version (3.1.2)
and reduced my book index.qmd to the bare minimum:

# Preface {.unnumbered}

In my _quarto.yml I've commented out all chapters other than index.qmd

  chapters:
    - index.qmd
    # - intro.qmd
    # - getting_started.qmd
    # - multivariate_plots.qmd
    # - linear_models.qmd

• Are there any tools to debug or trace such a problem?
• In the quarto internals, is there any way to provide more information on such a pandoc error?

To help track this down, I've made my repo public, at https://github.com/friendly/Vis-MLM-quarto

[friendly]

I have also checked the encoding of my files in RStudio. They seem to be UTF-8. Is there any way I can check all the files in my book folder, rather than one by one?

I also checked my Windows 10 chcp setting. It gives 850, but not sure what this means.

All of this worked up until a couple of days ago, and I don't think anything changed on my system

[cscheid]

under quarto (v. 1.1.189)

This is a relatively old version of quarto; can you first try upgrading to our latest stable version and see if the problem persists?

[dragonstyle]

I tried this locally on both a MacOS and Windows machine, both of which were able to render the file fine.

Is there anything unique or unusual about your configuration that might somehow be related? Do you have access to an alternate machine that you could try this on?

Two things occur to me:

• Something about the environment is different in some that we don't understand and that is somehow unearthing the bug (e.g. file system code page, shell / shell behavior, etc...)
• Somehow other data is being included in the rendering (for example from a pandoc data-dir - we attempt to lock down the data-dir to our own, but maybe somehow something is sneaking in through a crack we're not aware of).

Those are my only current theories - I'm sorry I don't have better guesses!

[cscheid]

The MINGW64 line makes me suspect it's something about the localization settings of that terminal itself.

Note also that the  garbage characters are probably unicode being badly interpreted. I think that has to be the source of the problem.

[cderv]

Good catch @cscheid ! To try isolate this, @friendly is this possible to run quarto render in another terminal like powershell that should be available on Windows by default ?

If you have no issue with it, then it should be something related to running Pandoc in this specific bash emulation environment.

[friendly]

Thank you all for the help. Very much appreciated, guys!

I finally found the source of this error -- a bib file outside my project that was imported and not checked. Yet, it took me over 2 days, and could have been avoided if pandoc returned filename and file location information in its error message.

I filed this as an enhancement request at pandoc:, jgm/pandoc#8884

[cderv]

Wow good catch ! This is a bad one, and Pandoc should indeed help with the error ! Thanks for filling that up.