Allow named args in depends-on

[bollwyvl]

Problem description

per #3533 (comment):

Also, the list-based depends-on.*.args list seems like it might be challenging to reason about above a certain level of complexity, and might benefit from allowing explicitly naming the args being set:

[tasks.foo-one--]
args = [{ "arg" = "bar" }]
cmd = "echo built {{ bar }} > bar/{{ bar }}.txt"
inputs = ["foo/{{ bar }}/"]
outputs = ["bar/{{ bar }}.txt"]

[tasks.foo-all]
depends-on = [
  { task = "foo-one--", args = {bar="baz"} },
  { task = "foo-one--", args = {bar="boo"} },
]

[Hofer-Julian]

We should make sure that non-default values are still required to be set then and give a good error message if they are not. Apart from that, the proposal sounds good to me.

[bollwyvl]

Sure: as JSON Schema (or the nested schema-like field descriptions in #3535) can't validate either the argument count or parameter names directly in an editor, with either approach they should be validated as early as possible, and not deferred until actually running a misconfigured task dependency.

I feel like when authoring a task graph (or merging authored changes) the named approach would likely be more robust and readable, and allows for having multiple optional values which can be set independently without double-encoding the n-1th params to set optional param n.

The array form might have the slight advantage, typographically, for long args, due to TOML whitespace requirements, as it is pretty awful to write lists-inside-objects-inside-of-lists.

[theavey]

I think there is some further oddness with how this is currently implemented. Taking an example from the docs

[tasks.base-task]
args = [
  { "arg" = "arg1", "default" = "default1" },  # First argument with default
  { "arg" = "arg2", "default" = "default2" },  # Second argument with default
]
cmd = "echo Base task with {{ arg1 }} and {{ arg2 }}"

[tasks.partial-override]
# Only override the first argument
depends-on = [{ "task" = "base-task", "args" = ["override1"] }]

running base task works as expected

>>> pixi run base-task
Base task with default1 and default2

>>> pixi run base-task cli1
Base task with cli1 and default2

>>> pixi run base-task cli1 cli2
Base task with cli1 and cli2

>>> pixi run base-task cli1 cli2 cli3
Error:   x task 'base-task' received more arguments than expected

However, I don't think there is a way to give arguments to partial-override that get passed to base-task

>>> pixi run partial-override
Base task with override1 and default2

>>> pixi run partial-override cli1
Base task with override1 and default2

but what's worse, any arguments you pass to partial-override are just silently ignored

>>> pixi run partial-override cli1 cli2 cli3
Base task with override1 and default2

If you change the partial override to now accept args, it will at least throw an error if you provide more arguments than expected, but I have not found a way to actually use those arguments to run base-task (I suppose other than just calling pixi run)

[tasks.partial-override]
depends-on = [{ "task" = "base-task", "args" = ["override1"] }]
args = [
  { "arg" = "arg2", "default" = "newdefault2" },  # doesn't actually do anything
]

[tasks.partial-override-with-args]
cmd = "pixi run base-task override1 {{ arg2 }}"
args = [
  { arg = "arg2", default = "newdefault2" },
]

[lucascolley]

I have not found a way to actually use those arguments to run base-task

You can use the args inside the depends-on section, see e.g. scipy/xsf#41. I think this should work:

[tasks.partial-override-with-args]
depends-on = [{ "task" = "base-task", "args" = ["default1", "{{ arg2 }}"] }]
args = [
  { arg = "arg2", default = "newdefault2" },
]

I guess this issue would allow us to remove the inclusion of "default1".

[theavey]

@lucascolley beautiful, thank you for that! Solves my immediate use case

I guess this brings up three thoughts:

1. This behavior of how to pass through args to dependent tasks should be documented, imo (opened docs: add example of passing arg to depended-on task #4030)
2. I still think this original enhancement is a good idea: named args would add convenience and clarity
3. Also, the silent ignoring of extra arguments doesn't seem great to me. I now realized that if a task does not list any args, any arguments given will be appended to the command (which I don't recall having seen documented anywhere). But in the case where there is no cmd (as is the case with the partial-override task above, when it is just a wrapper for other tasks via depends-on), the command line arguments are accepted and ignored. This does not seem like an immediate or urgent issue, but I imagine as pixi becomes more widely used (hopefully!!), less aware users would find this behavior unexpected

[lucascolley]

I now realized that if a task does not list any args, any arguments given will be appended to the command (which I don't recall having seen documented anywhere).

This is very useful behaviour but I agree it should be documented better.

in the case where there is no cmd (as is the case with the partial-override task above, when it is just a wrapper for other tasks via depends-on), the command line arguments are accepted and ignored.

We should raise an error here 👍

[lucascolley]

@Hofer-Julian could you assign this issue to me? We have a use-case for this at scipy/xsf#41.