Wrong margin side with rtl page direction in lualatex

[Udi-Fogiel]

I don't think this dependency is correct, left should mean left, with no regard to other parameters.

\documentclass{article}
\usepackage[right=10cm]{geometry}
\pagedirection=1\bodydirection=1
\begin{document}
Hello

\newpage
\pagedirection=0\bodydirection=0
Hello
\end{document}

[Udi-Fogiel]

For compatibility with existing code, maybe adding new options, trueright, trueleft etc., that would mean right and left regardless of \pagedirection or other parameters would be better.

[u-fischer]

my knowledge about bidi and directions is rather scarce, but imho geometry can't do anything here. It basically only sets \oddsidemargin, the kernel code uses then \moveright to move the page, and if you change the page direction \moveright moves to the left and so things go wrong.

Which lead to the question why are you changing \pagedir and \bodydir at all? What is the intended effect?

[Udi-Fogiel]

my knowledge about bidi and directions is rather scarce, but imho geometry can't do anything here. It basically only sets \oddsidemargin, the kernel code uses then \moveright to move the page, and if you change the page direction \moveright moves to the left and so things go wrong.

Which lead to the question why are you changing \pagedir and \bodydir at all? What is the intended effect?

Just a bad example. What I'm looking for is a concise syntax that will work in all engines and with all values of \pagedirection.
For example, if I would like a 10cm right margin, then in the usual case

\documentclass{article}
\usepackage[right=10cm]{geometry}
\begin{document}
Hello
\end{document}

would simply work, but the following will work only in XeLaTeX and pdfLaTeX , while with LuaLaTeX
I'll need to replace right with left.

\documentclass{article}
\usepackage[right=10cm]{geometry}
\usepackage[hebrew,provide=*,bidi=default]{babel}
\begin{document}
Hello
\end{document}

So, to determine what options should I use, I'll need to test for the engine, and the main language being used,
and I find it a bit cumbersome, especially when sometimes I get remarks from members of the team about portability
of code between engines :)

Maybe the geometry package can determine the value of \pagedirection at the beginning of the document and decide upon the value of \oddsidemargin from that? and again, probably a new option will be better to keep everyone happy...

BTW, I don't think this is not really relevant, but maybe will spark different ideas. LuaTeX does provide four new registers like \hoffset which are used according to \pagedirection's value, so simply setting them will theoretically work, but LaTeX does not use \hoffset in the first place... For an example, OpTeX set \pagerightoffset after calculating \hoffset
https://github.com/olsak/OpTeX/blob/86bbcfd3e7f6aaab3a5029668b92df852ff2224a/optex/base/margins.opm#L96-L98

[u-fischer]

while with LuaLaTeX I'll need to replace right with left.

Well yes, because babel changes pagedirection. So we are back to the question why this is done.

Maybe the geometry package can determine the value of \pagedirection at the beginning of the document

That wouldn't work for documents not using geometry but setting \oddsidemargin with other means.

If the \moveright in the output routine is wrong, then probably the kernel should change and not a package like geometry try to work around the problem. But to decide what really must be done, one needs some documentation what the direction commands do and how they are meant to be used. The luatex documentation only points back to some other model without explaining anything at all and I don't know where the original documentation is.

[u-fischer]

I made a few tests in plain. And what I gather is the following (I'm not sure if the ascii art really comes through ...)

By default TeX places the top and left of the page box at the 1in/1in point. So it spills to the right and bottom:

    1in
1in+----------------
     |  text
     |    
     |

If you set pagedirection to 1, the page box is placed at the \pagerightoffset point which is normally 1in from the right of the page.

                          1in
paperwidth - 1in           +----------------
                            |  text
                            |    
                            |

It then normally would go out of the page as given by the mediabox. If you use \moveright on the box then it moves to right and so even more out of the view.

If you set \bodydirection to 1, then the page box is placed with its right side on the reference point and so spills out of the page to the left:

                     1in
-----------1in+   
          text      |  
                      |    
                      |

Additionally the \moveright changes it meaning and moves to the left.

If you set both to 1, then the page is set on the right reference point and spills to left and so is correctly placed on the page:

                                       1in
---------paperwidth - 1in+
                        text          |  
                                        |    
                                        |

but again \moveright has changed its meaning and moves to the left and so breaks LaTeX expectations how the box behaves on the page.

As you want "left margin" to mean "left margin" imho the correct way to handle this not to set these directions at all. It looks wrong to change all reference points and then to add lots of complicated calculations to get the old behaviour back (and I ignored the additional problem of differences between the TeX page dimensions and the PDF page dimensions/mediabox anyway).

Searching a bit around I found that the main reason that \pagedirection is changed seems to be that this has a side effect on \parshape and so changes also list layouts etc (e.g. https://tex.stackexchange.com/a/717219/2388). I would suggest to investigate if one can avoid to touch to the page directions by setting \shapemode instead. According to the documentation one can set it for more than one paragraph by using negative numbers.

[Udi-Fogiel]

Maybe the geometry package can determine the value of \pagedirection at the beginning of the document

That wouldn't work for documents not using geometry but setting \oddsidemargin with other means.

I think that is ok because the page order changes as well in RTL (you open a RTL book the opposite way), so odd pages has the outer margin on the opposite side.

If the \moveright in the output routine is wrong, then probably the kernel should change and not a package like geometry try to work around the problem. But to decide what really must be done, one needs some documentation what the direction commands do and how they are meant to be used. The luatex documentation only points back to some other model without explaining anything at all and I don't know where the original documentation is.

Considering that there is no documentation other than the source code, and that the developers of the code don't use \pagedirection nor \bodydirection, and that these primitives are removed in LuaMetaTeX, there isn't much to do about it.

From I can see \pagedirection is used during shipout to determined which offset to use for page dimensions (\pagerightoffset or \pageleftoffset), which LaTeX does not care about (not using \hoffset): https://gitlab.lisn.upsaclay.fr/texlive/luatex/-/blob/master/source/texk/web2c/luatexdir/pdf/pdfshipout.c#L130-146. It is also used to determine the page reference point (top left or top right corner): https://gitlab.lisn.upsaclay.fr/texlive/luatex/-/blob/master/source/texk/web2c/luatexdir/pdf/pdfshipout.c#L174-189

\bodydirection is a bit more obscured. I think it controls the direction of created boxes that does not specify direction explicitly (i.e. not using the dir keyword). This includes the page box, hence if \bodydirection and \pagedirection are not the same at shipout time the reference point is wrong and the page is positioned at odd places: https://gitlab.lisn.upsaclay.fr/texlive/luatex/-/blob/master/source/texk/web2c/luatexdir/pdf/pdfshipout.c#L98-100

I cannot speak on behave of babel, but I believe the decision to use these two primitives was made to make it possible to use existing code that create some layout design for LTR documents, but for RTL documents, without the need to patch much (like the bidi package does). This is probably the ideal solution for the LaTeX ecosystem as not much packages are written with RTL layout in consideration, but it has some flaws.

• Interface becomes confusing because sometimes right means right but sometimes it means right in LTR and left in LTR (hence this ticket).
• I don't think everything in a layout needs to be reflected.
• I believe using \bodydirection while sometimes help, sometimes it can go against your wishes, for example in math (which babel do patch).

In the long run, I believe having some support in the kernel for bidi layouts (not text necessarily) is the best solution. Having a shared layout direction parameter which package writer can test and users can set, some kind of boxes that their direction is determined by the layout direction specified and boxes that their direction is fixed (for graphics for example), sorting out the left and right meaning so that it will be uniform, for example \raggedright was discussed before (I'm not saying changing \raggedright but having an alternative with fixed direction will be nice, sometimes you want it to be reflected, sometimes you do not).

[Udi-Fogiel]

@u-fischer

That wouldn't work for documents not using geometry but setting \oddsidemargin with other means.

It depends if this is a two sided document I think, but definitely need some thinking.

But to decide what really must be done, one needs some documentation what the direction commands do and how they are meant to be used. The luatex documentation only points back to some other model without explaining anything at all and I don't know where the original documentation is.

I've written technical details in https://codeberg.org/polyglossia/luabidi/src/branch/master/luabidi.tex under the section Notes on \LuaTeX's Direction Model.

I made a few tests in plain. And what I gather is the following (I'm not sure if the ascii art really comes through ...)

This is mostly right, but some minor details are not accurate.

As you want "left margin" to mean "left margin" imho the correct way to handle this not to set these directions at all. It looks wrong to change all reference points and then to add lots of complicated calculations to get the old behaviour back (and I ignored the additional problem of differences between the TeX page dimensions and the PDF page dimensions/mediabox anyway).

The direction of the box being shipped out changes the default location of boxes inside of it, so there might be other consequences (marginpar, \hbox in vertical mode, multicolumn layout) so it might be more complicated to not set the direction of that box,
or it might be more simple, but IMHO testing for \pagedirection is not hard, if that should go into the kernel or be done here that is more up to you :)

I would like to point out that this ticket is less about the output and more about the end user. I don't think the every day user should deal with details about LuaTeX's directional model.

Regardless of different point of views, the fact is that today I often get questions from confused users about the layout, because they don't know they need to replace right with left in Hebrew documents. When such a thing happens often, the design of the interface is questionable.

Searching a bit around I found that the main reason that \pagedirection is changed seems to be that this has a side effect on \parshape and so changes also list layouts etc (e.g. https://tex.stackexchange.com/a/717219/2388). I would suggest to investigate if one can avoid to touch to the page directions by setting \shapemode instead. According to the documentation one can set it for more than one paragraph by using negative numbers.

See latex3/babel#380 and latex3/babel#395.

@jbezos can you jump in the conversation?

[Udi-Fogiel]

The direction of the box being shipped out changes the default location of boxes inside of it, so there might be other consequences (marginpar, \hbox in vertical mode, multicolumn layout) so it might be more complicated to not set the direction of that box, or it might be more simple, but IMHO testing for \pagedirection is not hard, if that should go into the kernel or be done here that is more up to you :)

Actually, now that I think about it, the box being shipped out is not the actual page box, but just a box containing the top margin, and the page box (with \moveright), so maybe simply setting that box direction is the way to go. It is as simple as replacing \sipout\vbox{...} with

\def\bdirz@{}
\def\pagedirz@{}
\ifx\directlua\@undefined\else
  \def\bdirz@{bdir\z@}
  \def\pagedirz@{\pagedirection\z@}
\fi
\pagedirz@
\shipout\vbox\bdirz@{...}

Then the \moveright will mean right, and the page box itself can still be RTL if needed.

[Udi-Fogiel]

One can just use the build/page/reset hook to reset \bodydirection so maybe there isn't any need for changes, but is there some way to set it back inside \shipout\vbox?

[u-fischer]

It is as simple as replacing \sipout\vbox{...} with ... \pagedirection\z@

If setting the \pagedirection back to zero is the right fix, this puts us back to the question why \pagedirection and \bodydirection are changed at all. If I do not touch them your example above works fine

\documentclass{article}
\usepackage[right=10cm]{geometry}
\usepackage{lipsum}
\begin{document}

%\pagedir TRT\bodydir TRT
\pardir TRT \textdir TRT
Hello \lipsum[1]
\end{document}

Sorry but I'm not against improving the RTL support of the LaTeX kernel or geometry. But I do not want to introduce a fix which is only there to work around some patch in luabidi or babel or whatever. Fixes must fix something that shows up without these packages.

[Udi-Fogiel]

If setting the \pagedirection back to zero is the right fix,

It is just as a safety in case some old documents set \pagedirection, not really needed.

this puts us back to the question why \pagedirection and \bodydirection are changed at all. If I do not touch them your example above works fine

\documentclass{article}
\usepackage[right=10cm]{geometry}
\usepackage{lipsum}
\begin{document}

%\pagedir TRT\bodydir TRT
\pardir TRT \textdir TRT
Hello \lipsum[1]
\end{document}

Sorry but I'm not against improving the RTL support of the LaTeX kernel or geometry. But I do not want to introduce a fix which is only there to work around some patch in luabidi or babel or whatever. Fixes must fix something that shows up without these packages.

Around what patch? \bodydirection is the direction of boxes created in vertical mode without an explicit direction specified (without the (b)dir keyword), if you don't want to use it, then you need to use \setbox in horizontal mode, or pass an explicit direction to the boxes. So it basically either use \bodydirection, or start patch allot of things in the kernel and other packages.

For example it affects lists, as they box the labels in vertical mode, and so the glue inside of them will be positioned accordingly:

\documentclass{article}
\usepackage[right=10cm]{geometry}
\usepackage{lipsum}
\begin{document}
 
%\pagedir TRT\bodydir TRT
\pardir TRT \textdir TRT
Hello \lipsum[1]
\begin{itemize}
\item foot
\item bar
\end{itemize}

\bodydir TRT
\begin{itemize}
\item foot
\item bar
\end{itemize}

\bodydir TLT
\end{document}

You also need it for the direction of the boxes of the footer and header etc.

In any case, as I said in #20 (comment) I don't think changes in geometry are needed, the following seems to work:

\documentclass[oneside]{book}
\usepackage[right=10cm]{geometry}
\AddToHook{build/page/reset}{\bodydirection=0\relax}
\AddToHook{build/column/before}{\bodydirection=1\relax} % depending on layout direction
\makeatletter
\def\@begindvi{\bodydirection=1\relax} % depending on layout direction
\begin{document}

\bodydirection=1\textdirection=1\pardirection=1
\section{foo}

Hello
\newpage
\bodydirection=0\textdirection=0\pardirection=0

\section{foo}

Hello

\end{document}

The only thing needed is an actual hook where \@begindvi is used, or better yet right after \shipout\vbox{.
If you think this is the way to go, should I open a ticket in the LaTeX2e repo?

[Udi-Fogiel]

Sorry but I'm not against improving the RTL support of the LaTeX kernel or geometry. But I do not want to introduce a fix which is only there to work around some patch in luabidi or babel or whatever. Fixes must fix something that shows up without these packages.

I don't fully understand that position. What about latex3/babel#397? There isn't something really wrong with what babel is doing, it does not patch anything related, it is just that
LaTeX does not take into account boxes directions, so even if I just set the text direction I get wrong alignment:

\documentclass[fleqn]{article}

%\usepackage{amsmath}
%\usepackage[english,hebrew,provide*=*,bidi=basic]{babel}

\begin{document}
\textdirection=1
\begin{equation}
a+b=c
\end{equation}
\end{document}

In here \textdirection affects the box direction because it is created in horizontal mode. The box should always have the direction of \mathdirection, regardless of babel, luabidi, or what languages are used.

Does this count as shows up without these packages?

[Udi-Fogiel]

In any case, as I said in #20 (comment) I don't think changes in geometry are needed, the following seems to work:

\documentclass[oneside]{book}
\usepackage[right=10cm]{geometry}
\AddToHook{build/page/reset}{\bodydirection=0\relax}
\AddToHook{build/column/before}{\bodydirection=1\relax} % depending on layout direction
\makeatletter
\def@begindvi{\bodydirection=1\relax} % depending on layout direction
\begin{document}

\bodydirection=1\textdirection=1\pardirection=1
\section{foo}

Hello
\newpage
\bodydirection=0\textdirection=0\pardirection=0

\section{foo}

Hello

\end{document}
The only thing needed is an actual hook where \@begindvi is used, or better yet right after \shipout\vbox{. If you think this is the way to go, should I open a ticket in the LaTeX2e repo?

But I think with this the outer and inner keys are misleading now :) As with RTL book the first page is on the left side, not on the right side.

This one can easily be fixed with a package option.

[u-fischer]

For example it affects lists, as they box the labels in vertical mode, and so the glue inside of them will be positioned accordingly:

But the formatting of both list versions is wrong, your \bodydir does not lead to a correct formatting: without it the bullets are in the correct place, only the text is missing a shift, in the second case the text is still placed wrong and now the bullets are additionally in the margin where they shouldn't be:

And yes I know that it works if you change the \shapemode. But then you have to think about what should happen if someone issues a \selectlanguage{english} between two items.

So it basically either use \bodydirection, or start patch allot of things in the kernel and other packages.

Well we are patching a lot anyway for the tagging support ;-).

But generally: as you know luametatex does not know pagedirection and bodydirection and at some time we perhaps want a lualatexmetatex, so if we use the commands we need to be sure that they are needed and we need a plan for an engine which doesn't have them.