This is probably down to compiling numpy. I see from numpy/numpy#5479 that the builds shouldn't really be used.
We should probably download wheels from http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy, perhaps placing into a numpy-wheels repository which the appveyor build can check out.