Per our message on reddit, I am having the occasional stuttering issue. It sounds like you have looked into it a bit and determined it is not related to the load on the pi's CPU.
My initial thought was that there was too much data for my wireless network so I was going to decrease the size/resolution of the video and see if that had an impact on this issue.