Remove unnecessary `GEM_PATH` modification #568

Job	Run time
cargo clippy	50s
make (yjit-bench, --enable-yjit=dev, --yjit-stats, true)	5m 42s
cargo test	1m 3s
make (yjit-bindgen, To fix: use patch in logs, --with-gcc=clang-14 --enable-yjit=dev, /usr/lib/ll...	4m 30s
make (check, RUSTC='rustc +1.58.0', 1.58.0)	8m 43s
make (test-bundled-gems, --enable-yjit=dev)	20m 41s
make (check, --enable-yjit=dev)	12m 20s
make (check, --enable-yjit=dev, --yjit-call-threshold=1 --yjit-verify-ctx --yjit-code-gc, -T --yj...	18m 12s
YJIT Ubuntu result	4s
	1h 12m 5s

Provide feedback