Performance issues on servers

[PatrikTheDev]

Hey, I've been deploying this project on different servers from different providers and I've been seeing wildly different performance between them. When running on my personal rig (Intel i7-14700KF (28) @ 5.500GHz, 4080 Super), I'm able to get ~140it/s and utilize the GPU fully. I'm able to replicate this decent performance on some 4090 based servers but some servers are only able to get around 40it/s with 20-30% GPU utilization (never goes above) and one CPU core being pegged to 100% usage. I can replicate this performance issue across many servers with weaker CPUs (server currently has an AMD EPYC 7402P (24) @ 2.800GHz)

Is there any way to squeeze out more performance or is this strictly CPU limited in my case? I want to be able to leverage services like vast.ai and others but I've been unable to for this reason.