Add a "refresh avatar" button next to the avatar that's enabled if a GitHub account is connected. This will fire off the job in the background so at least users can manually refresh their avatar if they want to. Perhaps throttle to 1 time a minute or so.