Integrate Raft library into a lock service

[marc-hessenauer]

Hello everybody! I'm writing my master's thesis on distributed consensus for lock services and would appreciate some thoughts on my design concept.
Here is my use case: I would like to integrate the a Raft library like etcd/Raft into a lock service. The idea is similar to Google’s Chubby lock service, which is a proprietary service that I obviously can’t use.
The lock service my company provided me with runs on a single instance currently. I have to employ consensus to make it run in a cluster. Is etcd/Raft a good fit? Alternatives are Dragonboat Raft and HashiCorp Raft.
Any questions, ideas, resources, production experience stories are welcome

[serathius]

I would not start from building your own lock service, maybe look at the existing options like etcd for example?

As for picking raft library, while the etcd/Raft might be the most stable and popular in production due to etcd/Kubernetes popularity, it's not under active development.

[marc-hessenauer]

Thank you for the advice! To add some context: I'm writing the thesis in my company and the task I was given is to take the lock service a senior engineer wrote and make it distributed (short version). So this is a hard requirement. The guy who developed the service says he tried etcd and it was way too slow and complicated for his use case. I didn't see the code yet, so I can't analyze what the issue might have been.

Stable and popular sounds good for a library, but "easy to integrate" and "good performance" are valid criteria aswell - do you have any insights on which lib is most suitable here? I know I'm asking in an etcd forum and have no problem with hearing biased opinions ;)

[serathius]

The guy who developed the service says he tried etcd and it was way too slow and complicated for his use case.

Too slow, maybe. But, you should start from your performance requirements and benchmarks. Too complicated, don't know if writing your own distributed system is simpler, such projects require much more work around the project than just coding the service. I work on etcd correctness, and I can share that just validating correctness of distributed system is a large project.

While etcd is not the simplest system and has some bloat, it's easy to underestimate complexity of when starting from scratch. Maybe the guy is right, depending on your requirements and trade-offs, maybe you can build system simpler than etcd, but in my experience most systems become complicated because they need to solve the problem. Still, from time to time projects make a wrong trade-off and get rewritten.

[marc-hessenauer]

Very interesting thoughts, that helps a lot already. I'll share my thoughts on the topics you mentioned aswell and I'll use separate answers so we can discuss in separate threads, if you like.

[serathius]

Stable and popular sounds good for a library, but "easy to integrate" and "good performance" are valid criteria aswell - do you have any insights on which lib is most suitable here?

Don't know as I haven't used other libraries, I don't think you will find people that used all three. Maybe talk to authors about why they didn't use the other implementation. However based on fact that there are so many raft implementation, usually the answer to question why implemented their own is "Why not?". You need to do your own research, define requirements, which features you need, build a PoC and see which one is best suited.

[marc-hessenauer]

Makes sense! It's good to hear from someone with experience that there is no silver bullet here. I'm currently trying my luck with Dragonboat since it claims to be the fastest - let's see how it goes...

[marc-hessenauer]

Performance Requirements: I was told that 50ms for a lock or unlock request are the upper limit. 200 requests per second are the current average of the system we are supporting. In the light of the etcd benchmarks in the documentation, that doesn't sound overly ambitious to me. Do you agree?

[serathius]

This is pretty small, I have seen etcd on PC handle over 20k QPS lease refreshes. One thing to node, there is no lock, unlock operations in etcd nor distributed systems. Etcd exposes lower level tools like keys and leases, that can be used to create many different kinds of distributed system primitives like lock. Those primitives are implemented on client side like Mutex https://github.com/etcd-io/etcd/blob/main/client/v3/concurrency/mutex.go. This means that to lock, unlock operations translate to multiple lower level operations.

[marc-hessenauer]

Good to know that 20k QPS are realistic in a small setup aswell! And thanks for the hint with how lock operations are constructed, we're using leases and Go Mutexes in our system aswell.

[marc-hessenauer]

Simplicity: I'll differentiate between simplicity in the API used by the developer of the client application and the simplicity of the backend managed by my colleague and me.
Our goal is to develop a service specifically for locking purposes with an API that is as simple as possible, but that still provides read and write locking as well as hierarchical locking. That part should remain simple, also in a distributed version.
Simplicity in the API might still mean that we need a rather complicated backend. This is not desired, but a downside we would be willing to accept, at least to a certain extent. So far, my assumption was that the system would not be too complicated, since I'm merging two existing components: the lock service and a Raft library. Doesn't sound like I'm reinventing the wheel, but I might be wrong of course.
My question to you would be: How do you test etcd for correctness (a link or a short description would do)? What might be potential areas where a lock service might need extra work that is not easily spotted in the beginning of the design process?

[serathius]

Correctness is an continuous effort.

Framework https://github.com/etcd-io/etcd/tree/main/tests/robustness
How it works: https://www.youtube.com/watch?v=IIMs0EjQZHg
Weekly meeting: https://docs.google.com/document/d/1idZ_7tV6F18v223LyQ0WVUn9gXLSKyeLwYTdAgbjxpw/edit?usp=sharing

[serathius]

Note, our correctness effort is still focused on etcd KV API and Watch, we didn't have enough time to cover leases or higher level primitives like distributed lock.

[marc-hessenauer]

Very insightful, thanks for sharing your resources! I'll have to implement something similar for distributed locks in my case once I'm done with the early stages of the POC. If I have questions or results that might be relevant to you, I will notify you - probably over a private message. I'll close this thread for now, since my question has been answered.

[marc-hessenauer]

Final note: The feedback and resources shared by Marek were very helpful. Further discussion will take place outside of this thread. Thanks a lot for your time! @serathius

[serathius]

No problem, if you can share I would be interested in your conclusions about raft implementations.