Notes from using xmake in a custom RTOS project

[davidchisnall]

I've spent some time over the last couple of weeks trying to move our RTOS from CMake to xmake. This was motivated by a couple of things, largely:

• CMake has a strong split between built-in and customisable things and, if your model doesn't look exactly like something that exists already, this can cause friction.
• CMake doesn't have a good way of passing structured data around.

I've now got a working proof-of-concept (thanks to @waruqi for answering so many questions!) that we can evaluate. Consumers who want to build a firmware image can write something like this:

includes({wherever they have put our code})
add_rules("mode.release", "mode.debug")
-- This is a bit annoying.  -O0 generates code that's *way* too big for most
-- embedded platforms and it would be nice if our code could customise the default 
-- optimisation levels for release and debug builds from the included file.
set_optimize("Oz")

-- These are annoying, I don't know why they aren't
-- picked up when set at the global scope in the included file.
set_toolchains("cheri-mcu-clang")
set_defaultarchs("cheri-mcu")
set_defaultplat("cheri-mcu")
set_languages("clatest", "cxx20")


-- Look, we can define a library!
library("lib")
    add_files({a load of files here})

-- And compartments!
compartment("example")
    add_files("example/example.c")

compartment("example2")
    add_files("example2/example.c")

compartment("uart")
    add_files("uart/main.cc")

-- Now we can combine them into a firmware image.
firmware("example-firmware")
    add_deps("lib", "example", "example2", "uart")
    -- This works, but it would be really nice if we could provide an `add_threads` 
    -- method on a target from our code and not rely on consumers adding an `on_load`
    -- and remembering to set `{expand = false}` at the end.
    on_load(function(target)
        target:values_set("threads", {
            {
                compartment = "example",
                priority = 1,
                entry_point = "entry_point",
                stack_size = 0x400,
                trusted_stack_frames = 2
            }, {
                compartment = "example2",
                priority = 2,
                entry_point = "entry_point",
                stack_size = 0x400,
                trusted_stack_frames = 2
            }, {
                compartment = "uart",
                priority = 31,
                entry_point = "entry_point",
                stack_size = 0x400,
                trusted_stack_frames = 2
            }
        }, {expand = false})
    end)

As you can see, there are a few cosmetic issues, but everything basically works. I have a few notes on my experience using xmake:

The good

There were quite a few positive things from working with xmake.

• The overall model seems to map to almost anything that I want to do.
• The buildcmds monad is really flexible and powerful.
• Having a mature rich scripting language to work in is so much nicer than CMake's macro language.
• Adding new rules lets me customise pretty much any part of the build.
• I was able to create something that actually works and (mostly) has a decent UI for consumers.

The bad

I am going to spend a lot more time on the bad, because the good things are already good and xmake is already sufficiently good that I'd like to see it improve.

Documentation

The docs are the first thing that anyone coming to xmake encounters and they're really lacking. Almost everything that I'm using in xmake, I learned about either by asking questions here or by reading the code. This makes me pretty nervous about asking users of our system to use xmake: if they need anything that we're not providing then they're going to have problems.

The first omission is a high-level abstract model. I believe, from reading the code, that xmake has a map-fold model, where each target defines a set of inputs, a map operation that's applied to each input (usually a compile step, can be the identity function), a fold operation that combines the outputs of the map into a single output (usually a link step. This can be omitted sometimes? Object targets omit it, as do phony ones, but I don't know how general this is). What lives in .xmake? What lives in build? What lives in /tmp/? What do I need to rm -rf to guarantee a completely clean build? How are dependencies calculated? What are implicit arcs in the dependency graph and when (and how) do I need to add them?

The batchcmds monad (which I really like) has no documentation. A few examples use it, but there's no docs section explaining what it is, how it works, why it exists, and so on. Even in the code, most of the methods have a single sentence fragment as a comment and I don't have enough context to know what they mean.

Most of the description-scope things have equivalents in script scope, but this is not documented. These are all exposed in the code via three layers of indirection, so figuring out exactly what is happening here is not easy from the code until you've spent a day or two getting familiar with the internals of xmake.

The fact that I can define functions at description scope was not something I learned from the docs, but was very useful. I didn't learn how to define a function in a file that I can use in script scope in multiple rules / targets.

Many of the script-scope things behave differently depending on the type of thing that they're called with and this is not documented. A few them take extra parameters and these are also not documented. For example, I have no idea how I learned that I could pass table.join(opt, {stdout = {path to output}) as the third argument to batchcmds:vrunv but it wasn't the docs and it's really useful.

Building from object directories is flaky.

xmake likes me to run the build commands from my source directory. There are good reasons why other modern build systems have moved away from that and have strong separation between source and build paths:

• The destination directory is a useful bit of state that can be used to store build configuration.
• The source directory may be read-only (often, in secure build environments).
• The destination may be in a fast filesystem (e.g. a memory disk or a FS with sync turned off), so writes are much faster than anywhere in the source tree.
• Storing anything in the source tree interacts poorly with revision control systems (they must be explicitly excluded) and with search (same problem).

Driving xmake from the object directory is really flaky. Even using the dev release, it periodically forgets that I don't want to put things in the source tree. Dependency resolution seems to sometimes break in exciting ways. To be honest, this is my biggest reservation about xmake: it promotes an antipattern and seems to be very poorly tested in cases where people try to avoid it.

Some things are just strings, some things are objects

set_kind just sets a string. You can set this to anything, but the codebase is then scattered with things that will fail because they'll compare the string against some known values and not find anything. If this is necessary at all (I'm not convinced that it is, add_rule("binary") ought to be able to express anything set_kind("binary") can), it would be great if there were an API for adding new kinds of target and all of the things that do if target:get("kind") == "whatever instead called methods on the object returned from target:kind(). Some of these may be querying properties of the kind (e.g. target:kind().is_linker_input()).

There were a few other places with this kind of pattern. CMake has a two-tier system of built-in things that are completely immutable, other things are in the script. For the most part, xmake lacks this and it's fantastic that I can extend it in almost any place, so exceptions to that make me sad. Anything like set_kind adds a point where users can't hook into the code and extend it for their uses.

Some things are easy to use from script scope, some are not

Our firmware rule needs to construct a linker script based on the libraries and compartments that the firmware target depends on. add_configfiles is easy to do in description scope, I have no idea how to do it in script scope. Fortunately, I could work around that because users will call the firmware() function that I define at description scope and that will add the config file. I then set the config vars in script scope in my rule (once we've collected all of the dependencies and can inspect them). I learned how to do this by reading the code for configfiles. This means that I'm relying on an implementation detail, which makes me nervous.

Creating a new target from script scope is not supported. Modifying a target other than the one that the rule is processing works (and I need to do it) but is apparently not recommended.

Composition is not well defined

I learned that a rule can depend on another rule. It's still not clear to me exactly how this composition works. This feels like a form of inheritance or delegation but there are several kinds of things that I'd want to do there:

• Have my on_* / after_* hooks run before the ones of the rule that I depend on.
• Have my hooks run after the ones of the rule that I depend on.
• Have my hooks run instead of the ones of the rule that I depend on.

I believe that add_depends makes mine run after the rule that I depend on. I don't know how to do the other forms of composition.

Dependency calculation seems to not always work

I hit two bugs in dependency calculation that are either bugs in my xmake.lua or in xmake itself. The fact that I can't identify the root cause of either makes me nervous:

• Sometimes, the firmware link happens before the compartments on which it depends have been created. This seems not to happen when my build directory is in my source directory. At some point, this stopped happening but I don't know why (and therefore I don't know whether it will start happening again).
• If I change the .in file for my linker script, it is not regenerated. I have added the output file as an explicit dependency of my link job with batchcmds:add_depfiles, but the dependency of the output on the .in file seems to be lost. The configfile thing does seem to be rerun if xmake.lua changes.

Paths are treated as strings

The library provides some nice path-manipulation utilities, but all of the APIs treat paths as strings. This was one of the best features of Fabrique: it had a type for files and expanded them to strings only when invoking other tools. This made it possible to differentiate between arguments that were file references and ones that weren't, which meant that you could do things like set up sandboxing policies for tools based on the files that they were supposed to be able to access. It also meant that string operations on paths just didn't work, so you didn't end up in the mess CMake found itself where so many people write "${foo}/${bar}" that you need to handle "/" as a path separator on Windows.

It may be too late to fix this without it being a breaking change, but I'd love to see xmake add an explicit file type that things like runv could handle as arguments and convert to strings, but which were never implicitly converted to strings elsewhere.

[waruqi]

Sorry, English is not my first language and I usually have to use google translate to understand your feedback, including updates to the documentation.

As a result the documentation is not perfect in many places.

Also, I would like you to break up the discussion into individual topics so that I can understand and respond better. The text is too long for me to understand and there is a risk of missing some points.

I learned that a rule can depend on another rule. It's still not clear to me exactly how this composition works. This feels like a form of inheritance or delegation but there are several kinds of things that I'd want to do there:

The dependencies between the rules are simply binding a batch of rules and their on_xx/after_xx scripts are all executed at the same time.

However, in the dev version I have added support for controlling the order of dependencies.

see #2817

xmake/xmake/rules/qt/moc/xmake.lua

Line 23 in 4a6e3f2

add_deps("qt.ui", {order = true})

Dependency calculation seems to not always work

It should work and if there is a problem I need reproducible examples to be able to analyse the cause of the problem in detail.

Paths are treated as strings

you can use local p = path("/tmp/file") to new a path object, then use some methods, e.g. p:directory(), p1 .. p2, p:absolute()

[davidchisnall]

Sorry, English is not my first language and I usually have to use google translate to understand your feedback, including updates to the documentation.

No problem. Tell me if Google translate produces something confusing and I'll try to rephrase it.

My post probably sounds very negative and I want to be clear: Once I figured out how to make things work, I really liked most of what I found with xmake. There were quite a few rough edges where things could be nicer but I think this is the first build system where I've done something moderately complex and not ended up hating it.

As a result the documentation is not perfect in many places.

Once I am sure that I understand things, I hope I can start contributing updates to the docs.

Also, I would like you to break up the discussion into individual topics so that I can understand and respond better. The text is too long for me to understand and there is a risk of missing some points.

Feel free to start separate reply threads for the individual bits.

However, in the dev version I have added support for controlling the order of dependencies.

Thanks, I'll take a look. I don't actually need this currently, but this is something I've needed when splitting bits of other build systems up.

This also reminds me that you are not often using Fixes #12345 in commits and so tying bugs to the commits that fix them is hard. A few of the bugs that I saw had a comment from you saying 'this is fixed' and closing them, but it's really useful if these are linked to the commit so that someone who has a similar bug can go and see what needed to be changed to fix the issue.

It should work and if there is a problem I need reproducible examples to be able to analyse the cause of the problem in detail.

I've really struggled with this because most of the failures are intermittent. I had one that happened all of the time, but then it stopped for no obvious reason.

It would be great to have a testing mode that printed all of the information that's being used to schedule jobs. In particular:

• Statically, for each target, what needs to be ready before each of its build steps.
• Dynamically, what checks were done (and passed) to allow a job to start running.

If I had something like this then I could probably create a test. Is there some testing infrastructure that I should look at? Ideally something that mocks the mtime interfaces and lets me force a build step with explicitly set modification times.

you can use local p = path("/tmp/file") to new a path object, then use some methods, e.g. p:directory(), p1 .. p2, p:absolute()

In my ideal build system, the only place where you construct paths from strings would be in reading command-line arguments or the output from other tools, the only place where you convert paths to strings would be when invoking other tools, everything else would treat paths as explicit types.