Support client RPC operation level timeout

[shrijeet]

Client RPC calls like get and put, if can support configurable timeouts, will be helpful in enforcing SLAs.

[tsuna]

Version 1.1 support synchronous timeouts (i.e. block a thread to wait for an RPC to complete for a certain amount of time, and give up waiting if the RPC is taking too long). Asynchronous timeouts need to be implemented.

For synchronous timeouts, just call join(timeout) on the Deferred of the RPC.

[shrijeet]

Asynchronous timeouts need to be implemented.

Can you give me a brain dump on this? I wish to work on it.

[tsuna]

Sure. For asynchronous timeouts, we need a timer thread that takes care of all the timeouts. This should be implemented with Netty's HashedWheelTimer. Every single RPC that is sent out would schedule a new TimerTask and save the Timeout for this RPC, probably along with its Deferred. Then for each response that arrives from the network, we'd take the Timeout object and cancel() it. If a timeout occurs, we'd callback the Deferred with an exception, say a RpcTimeoutException (that would inherit from RecoverableException).

There are a couple subtleties involved, such as properly handling the cases where the RPC times out and then immediately after that you get the response back from the network. In this case the response should just be silently ignored as no processing / handling of it can occur, because we already gave a RpcTimeoutException to the Deferred.

The HBase RPC protocol doesn't support timeouts and doesn't support canceling RPCs, so once something has been sent to the wire, we can't take it back.

[shrijeet]

This is helpful. Let me see where I can get with this.

[shrijeet]

@tsuna , finally I sat down and created a raw patch of what I understood you intended this to be done as.

I broke down your words for my understanding as follows:

• "For asynchronous timeouts, we need a timer thread that takes care of all the timeouts"
  • We don't need to create a new timer. Reuse existing 'timer' (already of type HashedWheelTimer) instance in HBaseClient.
• "Every single RPC that is sent out would schedule a new TimerTask and save the Timeout for this RPC, probably along with its Deferred"
  • Create a new field in HBaseRpc class, rpc_timeout of type TimeOut.
• "Then for each response that arrives from the network, we'd take the Timeout object and cancel() it."
• In decode method of RegionClient we receive responses. Once received response, we will call rpc.cancelTimeout method (a new method to be written in HBaseRpc) . This method will call timeout.cancel()
• "If a timeout occurs, we'd callback the Deferred with an exception, say a RpcTimeoutException (that would inherit from RecoverableException)."
  • We can create a timer task with delay set as value of the desired rpc timeout. I am thinking this could be per rpc timeout value. TimerTask executes once that delay is passed. In run method we will throw the exception RpcTimeoutException. In the world of deferred this means do a callback on deferred with argument as an instance of RpcTimeoutException which inherits RecoverableException.

Here is the rough patch (by no means it is close ;-) ) . Also for now only Get is supporting rpc timeout.

Patch : https://gist.github.com/2950832

[tsuna]

Please send a pull request, it's going to make it easier to chat about the code than with a Gist.

[shrijeet]

Will do.