latex_macros expansion eats spaces when the macro is followed by a digit

[hendriktews]

Hi,

I believe this is a bug but I am not sure, therefore posting here.

When I process

\newcommand{\name}{A}

Document \name X.

Document \name 5.

with pandoc a.md -o a.pdf with pandoc version 3.1.11.1 I get a PDF with two lines, where \name is replaced by A, as expected. However, in the first line there is a space between A and X while in the second line there is no space between A and 5. The effect is already visible when translating to LaTeX with -t latex. One gets a space between the expanded macro and a following digit when there is a newline between the macro and the digit in the source.

Is it a feature that LaTeX macro expansion deletes all spaces behind a macro when the macro is followed by a digit on the same line?

[massifrg]

The space marks the end of the macro name (this comes from TeX). You can use one of the following:

\name\ X.

or

\name{} X.

[hendriktews]

No, as I wrote, this has nothing to do with TeX, the space is eaten by pandoc, but only if the macro is followed by a digit.

[jgm]

% pandoc -t native
\foo 2
[ RawBlock (Format "tex") "\\foo 2" ]
% pandoc -t native
\foo bar
[ Para [ RawInline (Format "tex") "\\foo " , Str "bar" ] ]

In the first case the numeral is included in the raw tex element. In the second case only the space is.

Honestly, I don't remember why - or if there was a reason for this. I'd have to track it down in the code.
(Until then, I'm not sure if it's a bug or a feature...though if a feature, it's certainly a surprising one)

[jgm]

Same thing with latex reader:

% pandoc -f latex+raw_tex -t native
a \foo 2 e
[ Para
    [ Str "a"
    , Space
    , RawInline (Format "latex") "\\foo 2"
    , Space
    , Str "e"
    ]
]

I think this is the root cause: https://github.com/jgm/pandoc/blob/main/src/Text/Pandoc/Readers/LaTeX/Parsing.hs#L1049-L1050
With raw TeX (unknown commands), we try to parse presumed arguments, including dimensions.
Of course this is always a bit of a guess, because without knowing how \foo was defined, we don't know how many arguments it takes, or what kind.

Now this is a bit odd, because we do have the macro definition and pandoc is expanding it, but the 2 is still captured:

% pandoc -t native
\newcommand{\foo}{Name}

a \foo 2 e
[ RawBlock (Format "tex") "\\newcommand{\\foo}{Name}"
, Para
    [ Str "a"
    , Space
    , RawInline (Format "tex") "Name2"
    , Space
    , Str "e"
    ]
]

Perhaps that can be fixed. It's very delicate going between TeX and Markdown parsing, though. I recommend using explicit raw_attribute for raw content, to avoid any trouble.

[jgm]

Or use {} after the command as suggested above.

[massifrg]

@hendriktews : you're right, I read too fast and inattentively.

I tested with the latest version, 3.9.0.2, and I get the same results you get.

Quoting from the manual:

When latex_macros is disabled, the raw LaTeX and math will not have macros applied.
This is usually a better approach when you are targeting LaTeX or PDF.

And it looks like you're targeting PDF (-o a.pdf) of LaTeX (-t latex).

Disabling the latex_macros extensions with -f markdown-latex_macros, you get no spaces in both cases, so maybe the exception is the space between "A" and "X" and not its absence between "A" and "5".

If I remember well, in TeX/LaTeX, you can't write \nameX (it would be read as a macro whose name is "nameX"), but you can write \name5, because "5" has a different catcode and so TeX knows that the macro token stops just before the "5" digit.
In TeX, when you write \name 5, you get the same result of \name5.