Layout broken on latest Speedata version

[Demostenus]

Hello,

I had a layout that I was using, and was working without issues on version 5.1.20
Recently wanted to use the latest version 5.1.31 and the print fails strangely

I was checking differences between the results in the .protocol files:

Before on 5.1.20 I had these lines:

  <entry level="warn" msg="[page 10] replace: one of the arguments is empty"></entry>
  <entry level="warn" msg="[page 10] replace: one of the arguments is empty"></entry>

Now on 5.1.31 the second line is no longer printed and it fails.

  <entry level="warn" msg="[page 10] replace: one of the arguments is empty"></entry>
  <entry level="error" msg="Lua error:" message="attempt to perform arithmetic on a nil value (local &#39;second&#39;)"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw\lua\xpath.lua" line="616" function="eval_addition"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw\lua\xpath.lua" line="720" function="reduce"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw\lua\xpath.lua" line="792" function="eval_argument"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw\lua\xpath.lua" line="807" function="parse_internal"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw\lua\xpath.lua" line="838" function="parse"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="4264" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="1187" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="1187" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="4652" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="4652" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="4644" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="1187" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="4644" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="1187" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="4644" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="4644" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="2169" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="1187" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\commands.lua" line="3822" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="886" function="dispatch"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="1806" function="initialize_luatex_and_generate_pdf"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher.lua" line="1221" function="?"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\spinit.lua" line="518" function="main_loop"></entry>
  <entry level="message" msg="stacktrace" source="@D:\speedata\sw/lua/publisher\spinit.lua" line="529" function="?"></entry>

Do you have any idea what is happening and what should I check?

[pgundlach]

I haven't touched the xpath.lua for a while, but re-implemented the XML parser. So probably I have broken something there. I'd be very interested in getting the layout.xml if it is possible.

I'd hope that if you fix the problem with the second argument with replace, the problem might go away.

You can also try the new XPath parser (yes, I know about #667)

I assume you mean version 5.3.1.

Just in case you are brave enough: you can try sp --option lib=rust which uses a completely different XML parser. But this is untested (it works with all my local tests, but this is very fresh code).

[Demostenus]

Yes, I am using latest version 5.3.1.

Just for info:
I found where the replace issue was happening and fixed it.
Now in the protocol it continues from page 10, up until page 78 where the protocol stops abruptly and is cut like when using lxpath, but I am still using the old luxor parser (which on 5.1.20 is working for me).

 <entry level="debug" msg="PlaceObject" type="Table" col="16" row="269" wd="170" ht="16" page="78"></entry>
 <entry level="debug" msg="PlaceObject" type="Table" col="1" row="1" wd="170" ht="22" page=

Also in Windows Task Manager I can see sdluatex.exe growing in memory usage.
From the moment I noticed the protocol getting cut (4GB usage), to now at 13 GB usage and growing. But no more output in protocol

So I think the new changes broke the old parser also.

[Demostenus]

I left the print process continue and sdluatex.exe kept growing in memory usage until it reached 100% and everything crashed (browser, vscode, everything). Never seen anything like this before.

With the exact same command and exact same data and layouts on 5.1.20 it prints 280 pages.
Cannot understand what to look for, to make it work on latest version.

[pgundlach]

I don't have any idea what goes wrong. I will try to pack the Windows package with a different LuaTeX binary, let's see if this helps. (Will take some time).

[Demostenus]

I was trying to find the exact version, or github commit that created these issues for me. Like I said before the last version that was working for me without any changes on my side, was 5.1.20.

I tried to build from source (github) version 5.1.24 and it wasn't working, then tried 5.1.22 and it was working, next tried 5.1.23 and it was also working.
So the issue started betwen 5.1.23 and 5.1.24.
Next I tried to see which commit exactly, and it was exactly the first one after 5.1.23: read_attribute new type rawstring w/o {} escape

• Commit e5d28ae
  All versions after this commit no longer work for me.

And I can see there are not that many changes done in this commit, so I decided just to revert these changes on 5.1.24 (which wasn't working) and then built the executable and it worked! So obviously these are the changes that break the processing on my layouts.

I then tried to do the same fix (reverting these changes) on the latest source repo 5.3.2, but I cannot manage to build the executable due to the rust build part. Maybe you can check if these changes are really necessary, but according to my tests, these are the exact changes that break my layouts.

[Demostenus]

Update, I managed to build the executable for 5.3.2 with the 2 changes mentioned above reverted as mentioned (pre read_attribute new type rawstring w/o {}) and now publishing finishes as before.

So, these are the changes affecting my layouts. Maybe you can check if there is a different way to include them if they are really needed.

[pgundlach]

I will probably remove the rust part in the next version (it is an experiment, and not used in production).

Thank you for the comment, I will examine it.

[pgundlach]

Do you have any Value elements with the select attribute containing curly braces?

<Value select="...{ ... } .." />

[Demostenus]

Yes, I am using curtly braces in many places, both in Value select and other places where normally you cannot put XPath or variables

Is this issue affecting only Value select? or other places also?

And what should I do? Simply remove the curly braces?

[pgundlach]

Only problematic in <Value select=".." />. If in doubt, post some examples. You can still use curly braces in the select statement, but there is no special handling of those anymore.

[Demostenus]

I was only using curly braces inside select of Value like this:

<Value select="${concat('ParaNumber', $lvl)}" />

To dynamically create variables, so I changed to

<Value select="sd:variable('ParaNumber', $lvl)" />

And now I get this error spammed:
E: XPath function unknown (fname=variable,prefix=threshold/thresholdValue, ' ', sd)

which is strange because I was using sd:variable in other places.

But I feel since this feature of raw strings is breaking code that used to work, maybe it shouldn't be the default.

[pgundlach]

My tests with sd:variable and the old xpath parser look fine. The old xpath parser gets into some strange state where it thinks that the prefix of the function is not sd but threshold/thresholdValue, ' ', sd. No idea why.

I will try to keep the curly brace syntax for the old XPath parser, but not for the new one (which is the default now), since this is not valid XPath syntax and the manual does not guarantee this behaviour.

[pgundlach]

Version 5.3.3 has the old syntax (hopefully) and builds without the rust library, just if you want to try.

[Demostenus]

Yes, it seems to work out of the box, without any changes from my side.