`pueue restart` stashed jobs

[alexheretic]

A detailed description of the feature you would like to see added.

pueue restart doesn't currently work with stashed tasks: Couldn't restart tasks: [0].

It could be made to with behaviour, say for stashed task 33:

• pueue restart 33 => 33 is enqueued at the back of the work queue. 33 is removed. Equivalent to re-adding an identical task and removing 33.
• pueue restart -s 33 => 33 is stashed at the back of the work queue. 33 is removed.
• pueue restart -i 33 => 33 is enqueued in place. Same behaviour as enqueue.

The behaviour seems fairly consistent to existing behavour.

Explain your usecase of the requested feature

When managing tasks I may have a bunch of stashed ones followed by later ones I started. I sometimes want earlier stashed tasks to run after later already queued ones. In general it is a bit of a pain currently to reorder tasks with switch.

With the proposed behaviour I could use pueue restart to enqueue stashed task(s) to run after everything else I already queued, or to move stashed task(s) to the back while keeping them stashed.

Alternatives

Using switch... please don't make me.

Additional context

No response

[Nukesor]

Hey :D

The restart feature also sounds a bit like a workaround around the actual feature you want: A convenient way to re-organize non-running tasks in the queue.

So, let's brainstorm a bit. I see the usecase and I feel the pain of switch. What would be a convenient way of re-ordering tasks on the cli (and/or editor).

1. An interactive TUI would of course be awesome and best suited. I'm not experienced with interactive TUIs, so I'm not sure if we can always ensure to have the capability of an interactive TUI? That tui would also not be scriptable, but I guess such a reorder command would never be scripted, but rather always be an interactive operation.
2. Something like pueue edit comes to mind. For people that use vim/emacs/etc. it would probably be convenient to shift around lines, others might find it pretty painful :D. I'm also not sure if we get all of the important info on a single line.
3. I'm very open to other ideas and input :)

Things we have to handle nicely:

1. While re-ordering, we have to somehow ensure that the existing order of tasks that already run (and thereby cannot be moved) and we have to properly convey this info to the user. Although, now that I think of it, it might even be possible to refactor pueue to make it so that Ids are only their queue position without being their identifier. That would decouple the actual task logic from the queue logic and make it possible to switch around running tasks... Although, that's a bit offtopic.
2. While re-ordering, we have to "lock" a whole group or even the whole daemon. I.e. we have to prevent tasks from being processed/added/removed. (Except we make an interactive TUI version, that can handle such events live and only performs "switch/move" commands under the hood.)

[alexheretic]

The restart feature also sounds a bit like a workaround around the actual feature you want: A convenient way to re-organize non-running tasks in the queue.

Yes I think that's fair 😅. I guess the restart behaviour proposition is a "minimal change" kinda thing which can be an easier sell.

So, let's brainstorm a bit. I see the usecase and I feel the pain of switch. What would be a convenient way of re-ordering tasks on the cli (and/or editor).

I actually didn't realise you could already re-assign ids using pueue edit. This is more helpful than switch combined with a decent text editor, so a useful result 🙂. Though it does has some issues (which maybe should be in their own ticket?):

• edit can re-assign ids, but only to ids of other tasks you are editing. I should be able to assign to a non-existing id, which would allow moving tasks to the front/back (with limitations).
• These isn't a convenient way to edit all editable tasks: pueue edit --all (or maybe argless pueue edit should do this?)
• Somewhat error prone if and id is used twice I think tasks can be lost.

I think those can probably be improved.

But I also think a non-TUI command for moving around queued/stashed tasks could exist that's more general than switch.
e.g. move 1 or more task ids to the front or back leaving other tasks order otherwise unaffected.

• pueue move 12 13 14 --start (or fail and do nothing if there is no space / tasks are not pending etc).
• pueue move 12 13 14 --end

Usage example

$ pueue
───────────────────────────────────────────────────────
 Id   Status    Command       Path         Start   End 
═══════════════════════════════════════════════════════
 11   Stashed   echo task a   /home/alex               
───────────────────────────────────────────────────────
 12   Stashed   echo task b   /home/alex               
───────────────────────────────────────────────────────
 13   Stashed   echo task c   /home/alex               
───────────────────────────────────────────────────────
 14   Stashed   echo task d   /home/alex               
───────────────────────────────────────────────────────
 15   Stashed   echo task e   /home/alex               
───────────────────────────────────────────────────────

$ pueue move 12 13 14 --start
task 12 moved to task: 8
task 13 moved to task: 9
task 14 moved to task: 10

$ pueue
───────────────────────────────────────────────────────
 Id   Status    Command       Path         Start   End 
═══════════════════════════════════════════════════════
  8   Stashed   echo task b   /home/alex               
───────────────────────────────────────────────────────
  9   Stashed   echo task c   /home/alex               
───────────────────────────────────────────────────────
 10   Stashed   echo task d   /home/alex               
───────────────────────────────────────────────────────
 11   Stashed   echo task a   /home/alex               
───────────────────────────────────────────────────────
 15   Stashed   echo task e   /home/alex               
───────────────────────────────────────────────────────

I think this could be implemented. However, pueue move ... --start has a serious limitation in a world where ids must be unsigned ints. This would perhaps benefit from switching to i64 ids, but continuing to start from 0. This way it'll essentially always be possible to move tasks to the front maybe using negative ids.

Rework ids / order

decouple the actual task logic from the queue logic

I think this is an interesting point. Currently ids are both:

• id number to reference to task
• an order value for execution

This coupling is perhaps not ideal if we consider moving tasks around. The former would benefit from being essentially immutable, while the latter we want to be able to mutate freely.

It may be worth considering a world where ids were immutable and unconnected to execution order. Pueue could have an additional order parameter for each task (or use a vec-index as equivalent).

Considering the usage of this it seems quite attractive to me. It resolves the "can't move in front of id:0" issue and it would make it fairly easy to implement pueue move 1 2 3 --after 12 / pueue move 12 13 --before 5 since we may freely re-order tasks now.

It also seems easier to understand since the actual id of a task never changes. Mutating task ids around can be confusing sometimes meaning we have to read the command itself to be sure of what is what.

Immutable id + detached order usage example

• switch still works but doesn't change ids only order.
• proposed move command has fewer fail cases and can freely re-order stuff.

$ pueue
───────────────────────────────────────────────────────
 Id   Status    Command       Path         Start   End 
═══════════════════════════════════════════════════════
 11   Stashed   echo task a   /home/alex               
───────────────────────────────────────────────────────
 12   Stashed   echo task b   /home/alex               
───────────────────────────────────────────────────────
 13   Stashed   echo task c   /home/alex               
───────────────────────────────────────────────────────
 14   Stashed   echo task d   /home/alex               
───────────────────────────────────────────────────────
 15   Stashed   echo task e   /home/alex               
───────────────────────────────────────────────────────

$ pueue switch 11 15
Tasks have been switched

$ pueue
───────────────────────────────────────────────────────
 Id   Status    Command       Path         Start   End 
═══════════════════════════════════════════════════════
 15   Stashed   echo task e   /home/alex               
───────────────────────────────────────────────────────
 12   Stashed   echo task b   /home/alex               
───────────────────────────────────────────────────────
 13   Stashed   echo task c   /home/alex               
───────────────────────────────────────────────────────
 14   Stashed   echo task d   /home/alex               
───────────────────────────────────────────────────────
 11   Stashed   echo task a   /home/alex               
───────────────────────────────────────────────────────


$ pueue move 12 13 14 --start

$ pueue
───────────────────────────────────────────────────────
 Id   Status    Command       Path         Start   End 
═══════════════════════════════════════════════════════
 12   Stashed   echo task b   /home/alex               
───────────────────────────────────────────────────────
 13   Stashed   echo task c   /home/alex               
───────────────────────────────────────────────────────
 14   Stashed   echo task d   /home/alex     
───────────────────────────────────────────────────────
 15   Stashed   echo task e   /home/alex                 
───────────────────────────────────────────────────────
 11   Stashed   echo task a   /home/alex                       
───────────────────────────────────────────────────────

pueue edit could perhaps work as you suggest in this world. You can literally re-order the lines in a text editor to change the order.

[Nukesor]

I actually didn't realise you could already re-assign ids using pueue edit.

Uhh. You cannot actually change the ids :D. You're just edit all the values to that of another task. Tbh, I never thought about the fact that one could use it that way!

Edit was never really designed to manipulate the order of the queue. The similar behavior is just a byproduct of being able to edit multiple tasks at the same time 😅 .

I think this could be implemented. However, pueue move ... --start has a serious limitation in a world where ids must be unsigned ints. This would perhaps benefit from switching to i64 ids, but continuing to start from 0. This way it'll essentially always be possible to move tasks to the front maybe using negative ids.

Negative ints are really annoying to use, as they often lead to confusion when using those IDs on the CLI. -1 is interpreted as an argument. Negative indices get a hard no from my end :D. However, when Id behavior is reworked, it would be doable to shuffle around IDs.

Considering the usage of this it seems quite attractive to me. It resolves the "can't move in front of id:0" issue and it would make it fairly easy to implement pueue move 1 2 3 --after 12 / pueue move 12 13 --before 5 since we may freely re-order tasks now.

Yep. My first thought was to give task ids a UUIDs. Using something like that, instead of another integer, has the benefit that there's no ambiguity on which ID is being used.
Associating two integers with a single task will lead to confusion and frustration. People will enter the wrong ids or have trouble memorizing it, it's just not a good UX.

By default, I wouldn't change anything on the current behavior. But by adding a new unique non-ambiguous task_id, script writers can get a persistent "handle" to a specific task, which is already worth a lot.
And internally, that ID allows a lot of operations that are currently not possible.
For the normal user, nothing really has to change, except that more features are available.

However, having two IDs also entails that almost all commands and arguments must be capable of accepting both, the "queue position" or an "task_id" as argument.
While scripting, one will most likely use the unique task_id.
On the CLI, though, users will default to the queue position. Using the queue position is also what I still want to be the default, as it's the 99% way of how people use Pueue today, and it works out for pretty much everybody + it's super intuitive.

pueue move 12 13 14 --start

I kind of like the semantic of move.
But I'm not too happy about using edit for moving tasks around. It was never designed to be used that way, and depending on the edit approach (when using file instead of toml editing), it also cannot really be used that way.

Also, using edit for that is really not a good UI experience.
The optimal way of doing this would be to have something that would represent the final queue afterwards, so the user can quickly validate the changes.
Editing integers in a large pueue edit screen is pretty messy. Everything is all over the place and one has to keep the order in mind instead of having it right there on the screen.

I think I need to ponder a bit more on this topic :D

Thanks for the suggestions and your findings on edit!

[alexheretic]

Negative ints are really annoying to use, as they often lead to confusion when using those IDs on the CLI. -1 is interpreted as an argument.

Good point, ok agreed.

Yep. My first thought was to give task ids a UUIDs. Using something like that, instead of another integer, has the benefit that there's no ambiguity on which ID is being used.

I was suggesting ids stay as short ints, but they no longer relate to execution order. The order would be a separate thing that may not even need to be enumerated for the user? Wouldn't this mean usage was essentially the same, using int ids. They would just no longer ever change and, say, 12 could end up being ordered before 0.

uuids are great for generating unique ids in a distributed setting, they might be overkill here and can be a bit long. Though I suppose it would be possible to support the docker-style "first n digits of id/hash if unique" thing to allow shorthand usage in cli.

Anyway thanks for responding and considering these improvements 🙂

[Nukesor]

uuids are great for generating unique ids in a distributed setting, they might be overkill here and can be a bit long. Though I suppose it would be possible to support the docker-style "first n digits of id/hash if unique" thing to allow shorthand usage in cli.

Agreed, having docker-style names for tasks or short-hand hashes would be neat and probably a lot more user friendly.

I was suggesting ids stay as short ints, but they no longer relate to execution order. The order would be a separate thing that may not even need to be enumerated for the user? Wouldn't this mean usage was essentially the same, using int ids. They would just no longer ever change and, say, 12 could end up being ordered before 0.

I'm currently thinking about usecases. Specifically: when would the internal id be used?
In my mind, the queue position is very intuitive and it will be the first/main number that will be visible in pueue status, as that's the number the queue is sorted by.

So I expect users to continue using that queue position.

The internal id will probably be used by power users that want to have more/scripted control over tasks and their positions. Especially for scripting, internal IDs will be very convenient, as they represent a persistent handle to a task. But for scripting, the internal id doesn't need to be simple.

What are the usecases you have in mind for the internal id?