Calling RServe using ThreadPool

[sopko16]

I have been looking this over and I cannot seem to figure out if this is possible:

1.) Can I call RServe rapidly with different threads from C# ThreadPool? When I try to do it, I seem to be getting this error result:
"Additional information: Recieved an unknown command parameter type from the server:16."

2.) Is there another way I should do this? Should I attempt to launch a few RServe daemons and then send multiple requests to each?

I'm attempting to send multiple lists of a window of 200 numbers to do the same calculation. Basically I save 210 numbers, and then want to pass data with these indices to perform the same calculation (sliding 200 length window of data points): I send these "in parallel"-
[0:199]
[1:200]
[2:201]
[3:202]
etc.

*IF I do these calculations "in line" where the first blocks the rest (without the ThreadPool or using Threading of some sort) my calculation might be too slow. I appreciate your help.

Thanks,
Mike P. Sopko

[sopko16]

Sorry. I am fairly certain that I have fixed this issue. I adapted this example for C# and it seems to work:
http://www.studytrails.com/R/RServe/R-Java-multi-thread-using-RServe-Unix.jsp

Hopefully this helps anyone else who has a similar issue. It might also be worthwhile to provide a sample code of something like multithreaded requests. If you would like to do this, I might be able to contribute some of my demo example code to yours (mine uses a threadpool). Closing for now and let me know what you think.

Thanks,
Mike

[SurajGupta]

I think there are two issues here:

1. Thread-safety of RConnection
   I don't think RConnection is thread safe. If multiple threads are making calls into the same RConnection then you'll garble your messages to the server. It's fairly easy to make RConnection thread-safe; I could just lock() around all the methods that send messages to the server. That's probably a good thing for me to do so that the consumer (you) doesn't have to manage the lock.
2. Multiple RConnection to the same server
   I'd have to investigate this some more. I've forgotten whether the server allows multiple client connections. If it does out-of-the-box, then you could just create an RConnection per thread. I'd have to dig deeper.

Of course, you could just launch multiple servers with one RConnection to each server and it would definitely be thread-safe. But then you'd incur the startup cost of each server and the additional memory overhead because you'll have multiple R processes running.

[sbryfcz]

So does this mean that this can not be used in an C# MVC environment? I was planning on using this library in webservice layer. Calls to the webservice would make calls to an RServer.

[SurajGupta]

This has nothing to do with C# MVC. MVC is a framework. Like all things .net you can make many choices. If you need to talk to R in a multi-threaded way then you need to select a path that won't break. I've outlined 3 options above. the first is a fix I an make in the code. the second I would need to research. the third is guaranteed thread-safe but requires more overhead. It's up to you to decide which approach is appropriate for your situation. If you tell me more about what you're trying to do in MVC perhaps I can help if it's simple/quick for me to understand.

[sbryfcz]

Sure. I am going to have a simple WebApi Controller/Endpoint. Something like:

[HttpPost]
double Predict(Data d)
{
    //create a new RConnection
    using (var r = new RConnection(...))
    {
        //load data into r
        r["d"] = d;
        //eval some r code an return the results
        return r.Eval("predict(model, d)");
    }
}

Just pseudo code, but I think you get the idea.

I apologize for not knowing more. I'm trying to learn more about MVC and how it handles multiple requests. It seems that MVC requests use thread pools. When a new request comes in, a thread is grabbed from the thread pool. This thread is used to complete the request and then the thread is returned to the pool. So if multiple requests come in, multiple RConnections (on different threads) would be created. My understanding was that RServe (on unlix/linux) would fork a process for each connection. I've done some simple testing and things seems ok....but I'm still learning about the pitfalls.

Any advice on which approach you think would be appropriate would be great. Thanks!

[sopko16]

It looks like when I run my sample code, I am seeing this:
*** stack smashing detected ***: /usr/lib64/R/library/Rserve/libs//Rserve terminated

It looked like RServe would launch multiple processes and the answers that I was receiving back were not garbage. But I need to confirm with more test code. I was running a long running R function and sending it the thread number (to R). so that way, if it had an answer from the calculation and the thread number looked correct, I could be fairly certain it was spawing multiple servers.

I also ran Linux "top" command and was able to see multiple RServe(s) entering and exiting. The problems I am now seeing are related to this message:

*** stack smashing detected ***: /usr/lib64/R/library/Rserve/libs//Rserve terminated
And I cannot seem to get my same sample code to work (it looks like my Linux machine is killing the processes when the "smashing" happens). I'll keep playing around with it and might be able to email over some sample code here soon.

[sopko16]

I have a sample code of this actually working. I have it down to as simple as I believe is possible at this point. I am launching 4 threads on a 4 core machine using ThreadPool and ManualResetEvent. Suraj, would you be interested in me emailing you the solution? If so, my email is at sopko.16@googlemail.com

I think this could help you a bit. Then, you can look in the "top" command output and see the RServe processes, etc.

I'm still getting a (different) failure (at around 30 seconds in):
"An unhandled exception of type 'RserveCLI2.RserveException' occurred in RserveCLI2.dll
Additional information: Rserve connection was closed by the remote host"

I'm wondering if it is my hosting provider (Linode and DigitalOcean) but not yet sure. This seems to happen on and off now and again.

EDIT: Right now I do not see this error on DigitalOcean (using the same code). I DO see it on Linode. I'm not sure what is causing it to happen yet.
I tried editing this on Linode but still issues:

echo 30> /proc/sys/net/ipv4/tcp_keepalive_intvl
http://www.tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/

*I've also seen keep.alive option here, but not sure if that is the issue and cannot get it to work yet...
https://github.com/s-u/Rserve/wiki/rserve.conf
Last comment here:
s-u/Rserve#2

[SurajGupta]

I think if this is working on your box and on DigitalOcean, then perhaps Linode doesn't provide enough resources to make this scenario work. I don't know enough about these clouds to be helpful.

If I were trying to do what you are doing, I wanted to maximize performance for the end-user and you are OK creating multiple RServers (so you can afford the memory/CPU resources), then I would do the following:

1. create a Static or Singleton class. When the class is constructed, it launches X instances of the R server and creates one RConnection per RServer and stores that info in an instance variable. Actually, encapsulate this in a method and call that method the first time the method I'm about to describe is run (that way your MVC app won't take a bunch of time to startup)
2. create a method that does the math with all the parameters it needs to do that. That method will basically "check-out" one of the available RConnections, use it to do the math, "check-in" the RConnection and return to the user (obviously put a try/catch/finally so that no matter what you "check-in" the RConnection so that it's available for other calls. Use lock() to make sure that only one thread at a time can "check-out" an RConnection. Also, catch RServeException and whenever that happens re-create the RConnection, retry the math, and then check-in the re-created one, not the original one. That will ensure that anytime the connection between the client and server is broken, that you refresh it with a new connection. If there are no connections available to "check-out", then just wait and retry.

something like this (I just quickly blasted out this code, so obviously you need to think some more about it make sure it's correct)...

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;

/// <summary>
/// TODO: Fill out the description.
/// </summary>
public class RManager
{
    private const int MaxReservationAttempts = 10;
    private object setupLock = new object();
    private bool isSetup = false;
    private List<Reservation> reservations;
    private object reservationLock = new object();

    public void Setup()
    {
        if (!this.isSetup)
        {
            lock (this.setupLock)
            {
                if (!this.isSetup)
                {
                    // for each of X servers you want to setup, launch the server, 
                    // create an RConnection, store all that data in Reservation and then add to this.reservations
                }

                this.isSetup = true;
            }
        }
    }

    public void DoMyMath(IEnumerable<double> myInputs)
    {
        Setup();
        var reservation = MakeReservation();
        try
        {
            // use reservation.RConnection to do your math    
        }
        catch (RServeException ex)
        {
            // rebuild reservation.RConnection and try again
        }
        finally
        {
            ReleaseReservation(reservation);
        }
    }

    public Reservation MakeReservation()
    {
        int attempts = 0;
        do
        {
            lock (reservationLock)
            {
                var reservation = reservations.FirstOrDefault(_ => !_.IsReserved)
                if (reservations != null)
                {
                    return reservation;
                }
            }
            attempts++;
            Thread.Sleep(1000);                
        }
        while ( attempts <= MaxReservationAttempts )
        throw new InvalidOperationException("can't make a reservation.");
    }

    public void ReleaseReservation(Reservation)
    {
        // ...
    }

}

internal class Reservation
{
    public int Port { get; set; }
    public int RConnection { get; set; }
    public bool IsReserved { get; set; }
}

1. Make your Controller super thin - it just calls the method that does your math as described above and returns the result.

[sopko16]

Nice! Thanks for providing the code. I will look at your code soon here as well to see how I can also improve mine.

Sorry for the confusion with regard to my Linux boxes. That was not a "real issue," I was able to get mine to work: the issue was that I was attempting to launch too many threads for the number of cores that I have. With my Linux boxes and the threadpool, it looks like I can safely add n and n+1 threads (4 core machine can have up to 5 threads). I can now do about 390 runs per minute of a code that takes one second each time to run with 7 threads.

When I have been doing this successfully, I have been creating an individual RServe server for each dedicated thread number. Without that, I was experiencing many more issues. Thank you for your help here. I'll keep you up to date; I can potentially share code here in the future if you would like.

[SurajGupta]

closing this out and adding: #32