Rollup merge of #143406 - scottmcm:did-we-need-that-unsafe, r=compiler-errors

Remove some unnecessary `unsafe` in VecCache

I'm pretty sure, but until perf confirms,
r? ghost

Author: matthiaskrgr