Figure out where the bulk of garbage collection is occurring and see if it is possible to add efficiencies.
Further, determine if there is a way to keep it from periodically locking up the draw calls (ex: minimum framerate regardless of logical lag).