When setselect is enabled for mouse selections, clicking anywhere after making a selection doesn't exit selection mode, but instead changes the endpoint of the selection. The same happens when a drag is started, instead of a new selection the endpoint of the existing selection is moved around.
With empty setselect, visual mode is used instead of select mode. This doesn't exhibit the same broken behavior, i.e the breakage is specific to select mode.
> nvim --version
NVIM v0.10.2
Build type: RelWithDebInfo
LuaJIT 2.1.1724512491
Run "nvim -V1 -v" for more info