Evaluation

Disclamer: This page contains our thoughts about the evaluation part of the pluto project. We are excited to discuss promising directions for pluto. Feel free to leave a comment or an issue to get in touch. Also note, that this reflects initial ideas and challenges. Elements might change or get dropped completely as the semester progresses.

Ambition & Purpose

• The evaluation is supposed to convey the advantages of weakly-consistent replication models for responsive and resilient geo-replicated services of planetary scale.
• It shall supplement the (anticipated, see Verification) correctness verification in terms of real world measures.
• Therefore, huge IMAP workloads and long-running test scenarios are needed to come closer to realistic use cases.

Setup

We need to evaluate (and possibly, benchmark) pluto against a production-grade IMAP service. For our first evaluation in terms of response times we chose Dovecot. It is the most used IMAP service in the world and thus, I would recommend to stick with Dovecot as the real world standard. As far as I know, Dovecot has no native feature comparable to pluto's weakly-consistent replication among more than two nodes. Therefore, we have to set up Dovecot in a way that allows for global synchronization. My initial proposal is to try to set up the following three systems:

• pluto, in its highest optimized version
• Dovecot A, using dsync between server pairs, configured to pair-wise sync all master/master replicas
• Dovecot B, using some mechanism or underlying system to achieve replication (GlusterFS? Dovecot Object Store? Replicated database?)

All of the setups will be deployed as kubernetes applications and the Google Compute Engine will be our public cloud of choice. Each setup is supposed to be spanning the "whole world", therefore 3 or more data centers shall be used for replication (e.g. Belgium, Taiwan, Oregon).