pandoc: Cannot decode byte '\xc3': Data.Text.Encoding: Invalid UTF-8 stream error after upgrade to 1.4.549 (win release) for project started with 1.3.450 if accented character is present in shortcode

[kompre]

Description

Hello,

after upgrading quarto to the stable release 1.4.549, I get this error for a project started with the previous stable release (1.3.450).

pandoc: Cannot decode byte '\xc3': Data.Text.Encoding: Invalid UTF-8 stream

After some investigation, the culprit was an accented character "à" (italian document) present in a shortcode in the text:👍

{{< meta classe_rugosità >}}

with the previous version the presence of accented character was not an issue, and was nice to be able to write a yaml in my natural language. (by the way, the yaml file can contain accented character, the error is raised if they get called in the source file).

Also another change I noticed that use to work in 1.3.450 and not anymore in 1.4.549: in 1.3.450, I could write the label for a code cell (tbl-, fig-) with underscore character (_) between words, but with 1.4.549 it results in invalid latex because quarto escapes the underscore. To be more clear:

• tbl-parametri_neve in source file, get translated in the latex file as:
  • tbl-parametri_neve with 1.3.450 (unchanged, latex can compile without errors)
  • tbl-parametri\_neve with 1.4.549 (latex raise an error, cannot compile)

This escaping behavior is not really even consistent because in the latex file I can see that the labels that are automatically generated by python (e.g. equations labels) or the labels that I write for markdown elements such that contains underscore do not get escaped, they stay as is. It seems that quarto is too eager to escape the stuff it founds in the code cell (e.g. #| label: tbl-some_words)

here a minimal working example:

---
title: "escaping underscore"
format: pdf
keep-tex: true
---

## Example

The following table is generated by code cell

```{python}
#| label: tbl-table_generated
#| tbl-cap: "caption of this table with an underscore (tbl-table_generated)"

from IPython.display import Markdown

Markdown("""
|  a  |  b  |  c  |
| --- | --- | --- |
| 1   | 2   | 3   |
| 4   | 5   | 6   |
"""
)


```

This instead is written in Markdown

|  a  |  b  |  c  |
| --- | --- | --- |
| 1   | 2   | 3   |
| 4   | 5   | 6   |

: caption of this table with an underscore (tbl-table_written){#tbl-table_written}

which result in the following latex with 1.4.549 (only relevant part):

\begin{longtable}[]{@{}lll@{}}

\caption{\label{tbl-table\_generated}caption of this table with an % underscore is escaped
underscore (tbl-table\_generated)}

\tabularnewline

\toprule\noalign{}
a & b & c \\
\midrule\noalign{}
\endhead
\bottomrule\noalign{}
\endlastfoot
1 & 2 & 3 \\
4 & 5 & 6 \\

\end{longtable}

This instead is written in Markdown

\begin{longtable}[]{@{}lll@{}}
\caption{caption of this table with an underscore
(tbl-table\_written)}\label{tbl-table_written}\tabularnewline  % underscore is not escaped
\toprule\noalign{}
a & b & c \\
\midrule\noalign{}
\endfirsthead
\toprule\noalign{}
a & b & c \\
\midrule\noalign{}
\endhead
\bottomrule\noalign{}
\endlastfoot
1 & 2 & 3 \\
4 & 5 & 6 \\
\end{longtable}

[cderv]

Also another change I noticed that use to work in 1.3.450 and not anymore in 1.4.549: in 1.3.450, I could write the label for a code cell (tbl-, fig-) with underscore character (_) between words, but with 1.4.549 it results in invalid latex because quarto escapes the underscore.

Regarding the underscore in label, this is indeed not allowed. It was not a problem to render before but still not advised. Now all labels are used as id, especially when referenced. This creates problem with rendering tools like LaTeX.

This has been discussed in

• Add an error about label containing underscore  #7307

Plan is to add a clear error about the problem and disallow rendering of document with _ in labels.

after upgrading quarto to the stable release 1.4.549, I get this error for a project started with the previous stable release (1.3.450).
pandoc: Cannot decode byte '\xc3': Data.Text.Encoding: Invalid UTF-8 stream
After some investigation, the culprit was an accented character "à" (italian document) present in a shortcode in the text:

Can you open a new issue about this with reproducible example and information on your system setup ?

Pandoc has been updated between 1.3 and 1.4 and could also be the source of the issue. Thanks !

[kompre]

_ was really useful when double click to select. It will be missed

issue for encoding error created at #8485

[mcanouil]

Word separator is modifiable at least in VSCode.

[kompre]

good to know! I'll have a look

[mcanouil]

FYI