It is really weird, but when I open a crafting table and put an itemstack in the upper or middle row, it is acting normal (item moved from crafting table to backpack when scrolling down), but in the lower row, it inverted (item moved from crafting table to backpack when scrolling up).
I'm not sure how the function works, but I've come up with a fix
decide slot picked -> decide slot for item to be moved to -> get Y coords for both slots (on screen) -> which one's bigger -> decide what "scrolling up" should do
I suppose this method will be able to fix this bug on complicated inventories