Achieving 11M IOPS & 66 GiB/s IO on a Single ThreadRipper Workstation | Tanel Poder Consulting

[giscus[bot]]

Achieving 11M IOPS & 66 GiB/s IO on a Single ThreadRipper Workstation | Tanel Poder Consulting

TL;DR Modern disks are so fast that system performance bottleneck shifts to RAM access and CPU. With up to 64 cores, PCIe 4.0 and 8 memory channels, even a single-socket AMD ThreadRipper Pro workstation makes a hell of a powerful machine - if you do it right!
Update 1: Since 2021, networks have gotten faster and now it is possible to have high-speed reliable remote I/O even in cloud VMs! - Linux, Oracle, SQL performance tuning and troubleshooting - consulting & training.

https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation/

[npodbielski]

I am thinking about building such machine for my self host lab.
Any new insights after 4 years?

[tanelpoder]

I guess not new insights , but key points:

1. With PCIe 5.0, just a quad (PCIe5 capable) adapter and 4 SSDs is good enough (you'd get like 58 GiB/s with 4 SSDs, that each give you 14.5 GiB/s or so. Plus your motherboard would have 1-2 slots for "boot disks" that might be PCIe4.

2. Since the consumer SSDs give you great throughput, but not great sync write latency, find a way to add one enterprise-grade SSD to your setup too (not into the quad adapter probably, but separate). Depends on what you're gonna run and test of course, but if you plan to stress test database workloads, then you'd need something with low write latency for your database WAL/redo logs, etc. 4 years ago, Intel Optane disks were the best choice, but Intel has discontinued them... I have also some Micron 7400s as I just mentioned in my latest blog:

https://tanelpoder.com/posts/using-pg-test-fsync-for-testing-low-latency-writes/

3. A consumer/gaming machine might be fine if you don't plan to use a discrete GPU, but if you plan to use a GPU too (that takes x16 PCIe lanes) or multiple quad SSD cards like my setup, then you'd want to use AMD ThreadRipper Pro CPUs (that are based on EPYC server CPUs). The consumer ones have only like 24-56 PCIe lanes going into the CPU directly, so the PCIe transport may become a bottleneck. My ThreadRipper back then had 128 PCIe lanes :-)

4. As far as new lessons learned by now - I was hoping to run so many more (higher level) tests with this kit, different database engines, etc, but never got to it. So choose your investment wisely! :-)

[npodbielski]

Thanks very much!