Feedback report from trying to use this with niche tools

[IcedQuinn]

I spent several hours today looking in to adopting xmake but the documentation to add custom targets appears to be the most minimum it could be. I am coming in from systems like Plan 9 Mk, PyDoIt and Ninja where it is easy to set up arbitrary rules for obscure tools.

Some of these will probably need to be split in to their own separate issues.

• on_build examples looks like markdown but it just copies and renames a file, doesn't even show an example of os.exec and how one is intended to run say pandoc.
• it took a bit to hunt down os.exec is what you call to actually get files built
• pulling up the source code there are references to functions likes autogenfile which are never mentioned in the documentation
• there is almost no docs for on_build_files but then a huge example for lex on buildcmd, which also brings in concepts like tracking the extra dependencies generated by the rule, but the on_build sections never told you how to do this at all
• is on_build deprecated for on_buildcmd? it would appear to be since one is documented and the other is not, and the source code for things like protobuf, lex and capnproto use buildcmd
• how does one deal with jobs where many files are specified as inputs but only one or two of those actually are part of the command line?

two examples of order only dependencies here:

• in nim you compile similarly to pascal where you only specify the name of the application unit but you may still say its dependencies are required; so i say nim c foo but in mk or ninja i may say that this has order only dependencies to all of the other nim (or .pas) files.
• in imatix GSL you have to specify some number of script files and model files to run; i managed to reverse engineer how to use an 'object' target and a custom rule to do this and generate an output, but xmake doesn't associate the inputs and outputs together and always rebuilds it. i also tried not using separate targets but this just merged all the .gsl and .xml files together via add_file and didn't do what it should.

for example i was able to dig out a way to get a target to actually run what it should:

# xmake.lua
rule('gsl')

on_build_files(function (target, sourcebatch, opt)
    local script = target:values('gsl.script')
    for _, sourcefile in ipairs(sourcebatch.sourcefiles) do
        if path.extension(sourcefile) == '.xml' then
            os.exec('gsl -script:%s %s', script, sourcefile)
        end
    end
    os.exec('touch potato.script')
end)

target('potato')
set_kind('object')
set_values('gsl.output', 'potato.script')
set_values('gsl.script', 'potato.gsl')
set_filename('potato.script')
add_files('potato.gsl', 'potato.xml', {rule='gsl'})

but since i used the objects kind i lose access to being able to set a filename which is also aggravating. so i have to use custom values to set that and xmake will always try to rebuild these.

i could fiddle around with writing some documentation but i have no idea how this is supposed to work (short of just copying the protobuf code and changing some strings)

for example in plan 9 mk the job is just

potato.script: potato.gsl potato.xml
    gsl -script:potato.gsl potato.xml > $target

in please.build there's just a simple seven lines to make a new rule for something like this

def word_count(name:str, file:str) -> str:
    return genrule(
        name = name,
        srcs = [file],
        outs = [f"{name}.wc"],
        cmd = "wc $SRC > $OUT",
    )

i'm not yet sure if xmake is just not suitable for jobs other than bog standard cmake like builds, or if it's 👍 but just needs someone to write docs.

[waruqi]

some docs:

https://xmake.io/#/manual/custom_rule?id=ruleon_buildcmd_file
https://xmake.io/#/manual/custom_rule?id=ruleon_buildcmd_files
https://xmake.io/#/manual/project_target?id=targeton_build
https://xmake.io/#/manual/project_target?id=targeton_build_files

on_build examples looks like markdown but it just copies and renames a file, doesn't even show an example of os.exec and how one is intended to run say pandoc.

see https://xmake.io/#/manual/builtin_modules?id=osexecv

is on_build deprecated for on_buildcmd? it would appear to be since one is documented and the other is not, and the source code for things like protobuf, lex and capnproto use buildcmd

Both of these can be used and have not been discarded. But using on_buildcmd_file will be simpler, and it can handle dependency update issues more conveniently. If you use on_build_file, you can also use depend.on_changed to handle them, but it's a little more complicated.

xmake/xmake/rules/utils/symbols/export_all/export_all.lua

Line 41 in 0b09f8f

depend.on_changed(function ()

on_build_file is more flexible, but on_buildcmd_file is simpler. For novices, I recommend using on_buildcmd_file

related issue: #1246

for example i was able to dig out a way to get a target to actually run what it should:

rule('gsl')
    set_extensions(".xml")
    on_buildcmd_file(function (target, batchcmds, sourcefile, opt)
        local script = target:values('gsl.script')
        local outputfile = --? we need get output file path
        batchcmds:execv("gsl", {"-script:" .. script, sourcefile}, {stdout = outputfile})

        batchcmds:add_depfiles(sourcefile)
        batchcmds:set_depmtime(os.mtime(outputfile))
        batchcmds:set_depcache(target:dependfile(outputfile))
    end
end)

or

rule('gsl')
    set_extensions(".xml")
    on_build_file(function (target, sourcefile, opt)
        import("core.project.depend")
        local script = target:values('gsl.script')
        local outputfile = --? we need get output file path
        depend.on_changed(function ()
             os.execv("gsl", {"-script:" .. script, sourcefile}, {stdout = outputfile})
        end, {dependfile = target:dependfile(outputfile), files = sourcefile, lastmtime = os.mtime(outputfile)})
    end
end)

[waruqi]

more examples:

https://github.com/xmake-io/xmake/blob/master/xmake/rules/lex_yacc/lex/xmake.lua
https://github.com/PucklaMotzer09/VulkanKraft/blob/96a9dcfe9bc6d9ef3ba228c03488b7d040e9d3b4/xmake.lua#L6

[xq114]

I do agree that the documentation for rules is too simple. It lacks examples for many useful cases. For me I usually read the built-in rules under the folder https://github.com/xmake-io/xmake/tree/master/xmake/rules (but I can seldom find the exactly same case as what I need) and get help from github discussions. In my port of suitesparse https://github.com/xq114/SuiteSparse-xmake/blob/main/xmake.lua I felt that even with detailed help from @waruqi (much thanks!), writing rules is still not a easy thing to do, at least for now. I wish if there was a more detailed documentation about the target object, the option object, and more examples of rules explained in detail about the usage, design procedure and key points.

[waruqi]

The examples in the document really cannot meet the needs of all users. I suggest you can directly read https://github.com/xmake-io/xmake/tree/master/xmake/rules or https://github.com/xmake-io/xmake/tree/master/tests/projects

Of course, if I have time, I will consider adding more examples to the document.

It takes a lot of time and work to complete the documentation, and my current fragmentation time can only barely satisfy the resolution of issues and the development of some new features.

I will update part of the documentation before publishing new version, and I will also consider adding documentation for the option and target instance interface later.

[waruqi]

About rules, I have used batchcmd to simplify them before. I think it should not be too complicated to implement a rule by referring to the existing rules with batchcmd.

Of course, I haven't used batchcmd to simplify some of the built-in rules of xmake, so it seems to be more complicated, and I will continue to update them later.

[davidchisnall]

The batchcmds thing looks really nice, but the on_linkcmd on targets is not documented neither is batchcmd at all, the only way that you can find what the methods are is to look in the source code (xmake/modules/private/utils/batchcmds.lua) and, even there, most methods have no comments and so you need to read the implementation to understand what they do and then try to figure out how they're supposed to be used. It would be fantastic if batchcmds could have the same kind of per-method documentation that targets have today.

[waruqi]

There are some here, but they are incomplete. https://xmake.io/#/manual/custom_rule?id=custom-batch-compile-script-process-one-source-file-at-a-time

Because I have so little free time, but if I have time later, I will consider adding them.

You can refer directly to some of the rules already implemented within xmake, or ask directly in discussions.

[IcedQuinn]

i'll go through some of these some time tonight and start trying to break them down in to individual issues.

the ones that come to mind are:

• rewriting the markdown example to one that actually processes markdown (ex. runs the input through multimarkdown or pandoc) so that example can be copy/pasted and also teaches the reader about os.exec and setting up the dependency tracking
• going through the links from here to set up examples for:
  • one input one output (ex. one markdown -> one html)
  • one input multiple outputs (ex. one lex file becomes one .c file which then becomes one compiled dependency)
  • multiple inputs one output (ex. creating a zip)
  • multiple inputs multiple outputs (ex. a GSL script which runs both a template file and a file to fill the template's data and outputs one or more generated code files)
• idiomatic way for complex generated files; for example whether these should be set up as their own targets and depended on by others or with custom rules and set_values
• something to do about custom outputs; probably something like the 'objects' kind but still has access to filename attributes. this would let you have something like the project's texinfo document or pdf or epub compiled as a target which is currently possible with 'objects' except you have to use set_values to name the output because target:filename() returns nothing for 'objects' kinds.

goal would be getting these templates documented so you just open the docs for custom rules, pick the one closest to what you need, copy/paste it and change out some text. ex. please.build and meson shows off how genrule works pretty close to this.

[IcedQuinn]

Ok. These have been broken down in to separate issues. I'll try to get through the links from the thread and see which ones I can PR.