I couldn't get PyFFTW3 installed on my machine, but pyFFTW is an alternative wrapper for FFTW that appears better maintained.
You can see a speed comparison of the two wrappers here, they're more or less equivalent:
mperrin/webbpsf#10
I'll submit a pull request in <1 week to get it working.