Using third-party Lua libraries in XMake, and support for standard Lua

[lasagnaphil]

TLDR: We should have standard Lua 5.1 support in XMake, so we can use the wide ecosystem of third-party libraries, along with a better user experience overall from existing Lua users.

I'm currently needing to write a code generator that takes inputs as JSON files and outputs C++ files as part of the build process. My first instinct was to use Lua from XMake with a third-party template engine (https://github.com/bungle/lua-resty-template). However, soon I discovered that XMake's Lua doesn't really conform to standard Lua in many respects, and almost all third-party Lua libraries are not compatible with it. The differences that I've found:

• no require(), has its own module system using import()
• no pcall/xpcall, instead there's a custom try/catch function
• no metaprogramming facilities like loadstring() and setmetatable()
• some core functions from Lua 5.1+ like select() are missing

I'm guessing that these restrictions/modifications on the Lua runtime are as intended, since XMake uses Lua as more of a DSL for a build system and isn't trying to provide a full-fledged Lua runtime like NodeJS or Luvit. (And I know that substantial modifications are often done on the runtime when embedding Lua in C/C++ applications, in particular game engines) However, it's useful to do tons of general-purpose stuff with XMake's Lua, since you then don't need to introduce an additional dependency on users like requiring them to install a standalone interpreter for Python/Lua/etc. And the LuaJIT runtime in XMake is incredibly fast and is well suited even for computationally intensive tasks, so why not use it?

So my proposal is that we should have standard Lua 5.1 support in XMake, so we can use the wide ecosystem of third-party libraries, along with a better user experience overall from existing Lua users. In terms of security issues that this might bring (like allowing loadstring()), XMake already runs arbitrary system commands right now so I don't think it's a concern. And maybe the pcall() functions can coexist with try/catch (since try/catch is already implemented with pcall(), if I'm correct? And personally I prefer using require() instead of import(), since it makes it harder to pollute your global namespace and get weird errors. Regardless of preference for standard Lua or XMake's conventions, I think the two can coexist without too many problems.

[waruqi]

xmake supports lua 5.1, 5.4 and luajit as its runtime and is completely user-aware. Even if xmake switches from lua 5.4 to lua5.1/luajit as its runtime, users will still be able to compile their projects without any modifications to the xmake.lua file.

This is why I have disabled all lua native api's that have compatibility issues and will not consider opening them up.

In xmake you can only use import to import modules and install the xmake specification to write configuration files, which provides maximum versatility and better readability and consistency.

Due to the flexibility of lua, without any restrictions, the xmake.lua configuration style for each project would be very messy, which would not be good for the user.

This is because xmake is just a build tool, providing only the generic modules needed to build, and not for the lua runtime to load any lua programs.

And the LuaJIT runtime in XMake is incredibly fast and is well suited even for computationally intensive tasks, so why not use it?

xmake is not computationally intensive task, luajit is very unstable, and it does not support some of the newer architectures such as riscv and loongarch. The authors are no longer actively maintaining updates to luajit.

Alternatively, you can continue to use luajit as a runtime for xmake, just recompile xmake with luajit enabled

cd xmake_sourcedir
make RUNTIME=luajit

Previously xmake used luajit as the default runtime and encountered too many stability issues, many of which could not be fixed and the luajit authors would not fix them.

Also, I have tested lua 5.4/luajit and, at least for xmake, there is no impact on compilation performance, and lua 5.4 is much more memory efficient and the binary file are much smaller.

[lasagnaphil]

Reading your comment and doing some additional research I now understand that there are some issues with LuaJIT. Maybe if it doesn't bring the performance benefits it's fine to just drop support for LuaJIT and just stick to standard Lua 5.4.

Anyway, thanks for explaining the philosophy underlying your choices. Although they are different from mine. I think it's an understandable design decision. (Personally I think a meta-build system like XMake should give as much power to the user possible by providing a general-purpose language. For example, like how languages like Nim, Zig, and Jai are doing where you can write build scripts using the same language as the one you're building for...)

[waruqi]

The user can still use 90% of the common api and modules of the lua language, such as: io, os, table, string, pairs, coroutine and all the basic lua syntax. On top of this, a large number of useful built-in modules and extensions are available.

Only a few interfaces with compatibility problems have been disabled, such as setfenv, etc., and alternative generic modules are provided, using import instead of require and try/catch instead of pcall/xpcall.

So we can still use lua for anything in the scripting domain.

no metaprogramming facilities like loadstring() and setmetatable()

Also, setmetatable/getmetatable can still be used, and we can call it with debug.setmetatable/debug.getmetatable.

and lua5.4 has not loadstring, it is also an incompatible interface.