Skip to content

RSDK-10643: condition var refresh thread #428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

RSDK-10643: condition var refresh thread #428

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
05b21d8
condition var refresh thread
lia-viam May 1, 2025
a2c8bf2
include cond var
lia-viam May 1, 2025
a78291f
concise refresh
lia-viam May 1, 2025
7198e15
const lock guard
lia-viam May 2, 2025
4fd7e13
Merge branch 'main' of github.com:viamrobotics/viam-cpp-sdk into robo…
lia-viam May 5, 2025
1a81c32
Merge branch 'robot-client-cond-var' of github.com:lia-viam/viam-cpp-…
lia-viam May 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions src/viam/sdk/robot/client.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,11 @@ RobotClient::~RobotClient() {
}

void RobotClient::close() {
should_refresh_.store(false);
if (should_refresh_) {
Copy link
Member

@acmorrow acmorrow May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If should_refresh_ is changing state, it should do so under the lock I think.

Copy link
Collaborator Author

@lia-viam lia-viam May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is not changing state-- should_refresh_ is only mutated in this method and in the named constructor which sets it in the first place, so there may be a concurrent read happening here but there's no way for it to be read while being mutated

Copy link
Member

@acmorrow acmorrow May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll argue that this is one of those cases where even if it is safe in practice, it is still better to do the check under the lock. It'll be annoying if someday we stand up TSAN and it whines about a leak here, and this isn't a hot path, so trying to eliminate the lock doesn't buy much of anything.

Copy link
Member

@acmorrow acmorrow May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean race of course, not leak, since I'm talking about TSAN not ASAN.

Copy link
Collaborator Author

@lia-viam lia-viam May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

Copy link
Member

@acmorrow acmorrow May 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd even move the read of should_referesh_ under the lock, or even eliminate the check entirely. It should be fine in close to just unconditionally set should_refresh_ = false (under the lock), and then notify. If there is no refresh thread, it is just a big no-op.

std::unique_lock<std::mutex> lk{refresh_lock_};
should_refresh_ = false;
refresh_cv_.notify_one();
}

if (refresh_thread_.joinable()) {
refresh_thread_.join();
Expand Down Expand Up @@ -220,16 +224,21 @@ void RobotClient::refresh() {
}

void RobotClient::refresh_every() {
while (should_refresh_.load()) {
try {
std::this_thread::sleep_for(refresh_interval_);
refresh();
std::unique_lock<std::mutex> lk{refresh_lock_};

} catch (std::exception&) {
while (true) {
Copy link
Member

@acmorrow acmorrow May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be while (should_refresh_)?

if (refresh_cv_.wait_for(lk, refresh_interval_) == std::cv_status::timeout) {
Copy link
Member

@acmorrow acmorrow May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

condvars can (in theory) have spurious wakeups (the reasons are somewhat historically [interesting] (https://stackoverflow.com/a/8594964)). Potentially, this could use wait_until to reenter the wait state should that happen. However, I'm not sure the complexity is warranted. Is there a risk to doing an early refresh in the event of a spurious wakeup?

Copy link
Collaborator Author

@lia-viam lia-viam May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one is interesting and relates to my answer about your other comments regarding the if and while conditions. spurious wakeups are the reason condvar code often needs an extra flag variable--if there's a wakeup we can tell if it's spurious or not by checking the value of bool no_i_really_meant_to_wake_you_up. all of the condition_variable::wait[_preposition] functions have an overload taking a predicate argument, and per cppreference (and, eg, libcxx) this is equivalent to while(!pred) wait(lk).

if we don't mind doing an extra refresh during a spurious wakeup, which I agree is probably harmless, it occurred to me that this whole function could just be

void RobotClient::refresh_every()
{
    std::unique_lock<std::mutex> lk{refresh_lock_}
    refresh_cv_.wait_for(lk, refresh_interval_, [this]{
        if (should_refresh_) refresh();
        return !should_refresh_;
    });
}

regarding the optional suggestion below though this would make me feel a bit weird about turning refresh_interval_ into an optional because I think that gets a little harder to reason about the validity of reading from refresh_interval_ or having to handle it being a nullopt

try {
refresh();
} catch (const std::exception& e) {
VIAM_SDK_LOG(warn) << "Refresh thread terminated with exception: " << e.what();
break;
Copy link
Member

@acmorrow acmorrow May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can refresh fail for legit but transient reasons like network being down? If so, is it actually correct to permanently shut down the refresh loop?

Copy link
Collaborator Author

@lia-viam lia-viam May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm good question, I was just copying the semantics from the previous implementation and adding a log statement. I imagine distinguishing types of errors could be done by catching GRPException separately and inspecting it

}
} else if (should_refresh_ == false) {
Copy link
Member

@acmorrow acmorrow May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe can be part of the loop condition?

break;
}
}
};
}

RobotClient::RobotClient(ViamChannel channel)
: viam_channel_(std::move(channel)),
Expand Down
6 changes: 4 additions & 2 deletions src/viam/sdk/robot/client.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
/// @brief gRPC client implementation for a `robot`.
#pragma once

#include <atomic>
#include <condition_variable>
#include <string>
#include <thread>

Expand Down Expand Up @@ -186,7 +186,9 @@ class RobotClient {
void refresh_every();

std::thread refresh_thread_;
std::atomic<bool> should_refresh_;
std::mutex refresh_lock_;
std::condition_variable refresh_cv_;
bool should_refresh_;
Copy link
Member

@acmorrow acmorrow May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be possible to eliminate the bool by making refresh_interval_ an optional.

Copy link
Collaborator Author

@lia-viam lia-viam May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think you're right, we're storing a T data; bool valid; which is basically just what an optional is under the hood.

(we could just have refresh_interval_ alone do all the work, but it's a little less self-documenting, although there are compact_optional types that allow this sort of thing)

Copy link
Member

@acmorrow acmorrow May 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is probably worth doing. There's just less state to manage, which means less risk of things decohering. It'd also simplify RobotClient::with_channel:

From

    robot->refresh_interval_ = std::chrono::seconds{options.refresh_interval()};
    robot->should_refresh_ = (robot->refresh_interval_ > std::chrono::seconds{0});
    if (robot->should_refresh_) {
        robot->refresh_thread_ = std::thread{&RobotClient::refresh_every, robot.get()};
    }

To

    if (options.refresh_interval() != 0) {
        robot->refresh_interval_.reset(options.refresh_interval());
        robot->refresh_thread_ = std::thread{&RobotClient::refresh_every, robot.get()}
    }

std::chrono::seconds refresh_interval_;

ViamChannel viam_channel_;
Expand Down
Loading