GGML RPC Server Security

[struct]

I've read this pull request that states the insecurity of the RPC server is well understood and until a plan is in place to harden the server the work should happen in the open, and not go through the advisory process. This makes sense and I appreciate the transparency.

It currently lacks authentication, contains memory address disclosure vulnerabilities and has no fuzzer targeting the client or server (although I did put one together here). Is there currently a plan to improve the security of the RPC server? If I were to write up what security controls are needed would that be helpful?

I wrote down quick thoughts up after spending a few hours looking at the implementation. Fixing these issues is entirely possible and I am willing to put in some time to implement the hardening measures required.

cc @rgerganov @CISC

[CISC]

If there is interest, it might make sense to apply to the GitHub Secure Open Source Fund (before August 12th) for this.

[struct]

Access to funding is good for the project long term, but I recommend you first put together a list of what security controls are missing, and a rough idea of how you'd like to implement them. Some of them I called out above and implementing them is fairly straight forward.

[rgerganov]

The main challenge is finding a proper way for fuzzing the graph_compute method. On the server side we re-create tensors by directly setting ggml_tensor fields instead of using public APIs. This may create all kind of invalid inputs that will crash ggml_backend_graph_compute(). Doing tensor validation in the RPC or other backends is not a great option as discussed here.

We need a better way for re-creating the tensors on the server side instead of doing ggml_new_tensor_4d() and setting all the fields.

[thevilledev]

ideally the project should define where the security boundaries are, or future security efforts on the server are not always going to make sense

Thanks for raising this issue @struct. Share the sentiment, having worked with a few RPC related security issues in the project. Well documented security boundaries and security model would greatly help contributing to project and align efforts.

[rgerganov]

What is the intended long term goals for the RPC server? Is it serving clients of various and separate privilege levels? Or is it intended to serve a single user on a closed network?

Serving clients with separate privilege levels is certainly out of scope. The RPC server is serving one client at a time and users can think of it as remote GPU fully dedicated to their workload. There is also an option to start multiple servers on the same host and split the available VRAM between them.

It may be more productive to focus on things like access control and encryption of the communications.

As @struct said this can be handled with other means like wireguard, openssh, etc. I don't see much value of adding this to the RPC server.

[rgerganov]

I can work on targeting that method with a more graph aware fuzzing harness.

If there is no way to add common input validation that can be reused in all backends that means that we have to fuzz the server with each and every backend which is not feasible IMO.

[struct]

Serving clients with separate privilege levels is certainly out of scope.

Thanks. I can understand all users being at the same privilege level, and there being no granularity in the authorization required to call individual RPC commands (aside from being able to authenticate to the rpc server itself via some SSH tunnel). But is it reasonable to assume a use case where someone is hosting e.g. 8 GPU's and wants to make them individually available to 8 different users? If 1 of those users compromises the RPC server they would have access to the other users RPC tensors.

In this use case the server should be sandboxed, RPC commands validated, and individual implementation errors fuzzed. If this use case isn't what you are designing for then that makes a big difference in which security efforts to prioritize.

If there is no way to add common input validation that can be reused in all backends that means that we have to fuzz the server with each and every backend which is not feasible IMO.

I agree that different verifiers per backend isn't feasible, ideally verification of the graph happens at the RPC layer similar to @thevilledev's previous PR. Is there an opportunity to revisit that?

[rgerganov]

But is it reasonable to assume a use case where someone is hosting e.g. 8 GPU's and wants to make them individually available to 8 different users? If 1 of those users compromises the RPC server they would have access to the other users RPC tensors.

With NVIDIA you can start 8 servers and use CUDA_VISIBLE_DEVICES to select a separate GPU for each server. In this case you will get some kind of isolation provided by the CUDA runtime.

I agree that different verifiers per backend isn't feasible, ideally verification of the graph happens at the RPC layer similar to @thevilledev's previous PR. Is there an opportunity to revisit that?

Feel free to experiment if you want. I think that it will be very hard to capture all implicit assumptions made by all kernels in all backends, so that memory corruptions are prevented.