[NFC] Remove spirv-patches folder and add it to gitignore (#15754)

- remove leftover
- Add to gitignore

Author: jsji

.gitignore

@@ -30,6 +30,9 @@
 /build*
 !buildbot
 
+#pulldown temp folder
+/spirv-patches-*
+
 #==============================================================================#
 # Explicit files to ignore (only matches one).
 #==============================================================================#