After a horrible two days of debugging, I finally found the problem for my input issues.
Problem
When using gj or <C-d> on a blank line at the bottom of the document, the cursor would wrap around to the top. Only a minor inconvenience, but it triggered my perfectionism.
Workaround
Pin @codemirror/view at version 6.38.8. Everything above that will result in the described behavior.
Fix
I'm not sure if it's a regression in codemirror or if it's something codemirror-vim should fix.
Just happy I finally found the problem.