server : use std::move whenever possible #12936
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -1552,11 +1552,11 @@ struct server_queue { | |||||
| std::condition_variable condition_tasks; | ||||||
|
|
||||||
| // callback functions | ||||||
| std::function<void(server_task&&)> callback_new_task; | ||||||
| std::function<void(void)> callback_update_slots; | ||||||
|
|
||||||
| // Add a new task to the end of the queue | ||||||
| int post(server_task && task, bool front = false) { | ||||||
| std::unique_lock<std::mutex> lock(mutex_tasks); | ||||||
| GGML_ASSERT(task.id != -1); | ||||||
| // if this is cancel task make sure to clean up pending tasks | ||||||
|
|
@@ -1565,16 +1565,16 @@ struct server_queue { | |||||
| } | ||||||
| QUE_DBG("new task, id = %d, front = %d\n", task.id, front); | ||||||
| if (front) { | ||||||
| queue_tasks.push_front(task); | ||||||
| } else { | ||||||
| queue_tasks.push_back(task); | ||||||
|
||||||
| } | ||||||
| condition_tasks.notify_one(); | ||||||
| return task.id; | ||||||
| } | ||||||
|
|
||||||
| // multi-task version of post() | ||||||
| int post(std::vector<server_task> && tasks, bool front = false) { | ||||||
| std::unique_lock<std::mutex> lock(mutex_tasks); | ||||||
| for (auto & task : tasks) { | ||||||
| if (task.id == -1) { | ||||||
|
|
@@ -1596,10 +1596,10 @@ struct server_queue { | |||||
| } | ||||||
|
|
||||||
| // Add a new task, but defer until one slot is available | ||||||
| void defer(server_task && task) { | ||||||
| std::unique_lock<std::mutex> lock(mutex_tasks); | ||||||
| QUE_DBG("defer task, id = %d\n", task.id); | ||||||
| queue_tasks_deferred.push_back(task); | ||||||
| condition_tasks.notify_one(); | ||||||
| } | ||||||
|
|
||||||
|
|
@@ -1611,7 +1611,7 @@ struct server_queue { | |||||
| } | ||||||
|
|
||||||
| // Register function to process a new task | ||||||
| void on_new_task(std::function<void(server_task&&)> callback) { | ||||||
ngxson marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
| callback_new_task = std::move(callback); | ||||||
| } | ||||||
|
|
||||||
|
|
@@ -1660,7 +1660,7 @@ struct server_queue { | |||||
| lock.unlock(); | ||||||
| break; | ||||||
| } | ||||||
| server_task task = std::move(queue_tasks.front()); | ||||||
| queue_tasks.pop_front(); | ||||||
| lock.unlock(); | ||||||
|
|
||||||
|
|
@@ -2004,7 +2004,7 @@ struct server_context { | |||||
|
|
||||||
| slot.reset(); | ||||||
|
|
||||||
| slots.push_back(std::move(slot)); | ||||||
| } | ||||||
|
|
||||||
| default_generation_settings_for_props = slots[0].to_json(); | ||||||
|
|
@@ -2105,7 +2105,7 @@ struct server_context { | |||||
| return true; | ||||||
| } | ||||||
|
|
||||||
| bool launch_slot_with_task(server_slot & slot, const server_task && task) { | ||||||
|
||||||
| bool launch_slot_with_task(server_slot & slot, const server_task && task) { | |
| bool launch_slot_with_task(server_slot & slot, server_task && task) { |
There was a problem hiding this comment.
We also move the params and prompt_tokens from task to slot, so I think this is currently not possible
There was a problem hiding this comment.
What is not possible? With the const present, the moves of params and prompt_tokens are actually copies. Removing the const from the argument will make them proper moves.
There was a problem hiding this comment.
The problem is that the prompt_tokens will soon contains image tokens as unique_ptr, so it will require a proper move.
And even without image (the current case where we have only text tokens), I think a proper move still make sense here. This funtion is the last place where the task want to go (i.e. we expect task to be destroyed here)
There was a problem hiding this comment.
Ah now looking back at this, I realized that I'm thinking the reverted (I thought that you want to add the const) ; Sorry for the confusion 🥲
Outdated
There was a problem hiding this comment.
The task should not be referenced after being moved. We can make this safer like this:
const auto id = ctx_server.queue_tasks.get_new_id();
{
server_task task(SERVER_TASK_TYPE_METRICS);
task.id = id;
ctx_server.queue_results.add_waiting_task_id(task.id);
ctx_server.queue_tasks.post(std::move(task), true); // high-priority task
}There was a problem hiding this comment.
Thanks, that's a good idea. I also spotted some places where task is used after the move. It should be fixed in my last commit: 9487165
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.