how row split works

[Kas1o]

in my guess, it will split weight matrix to multiple matrix by row. copy each of them to diffrent device and with same complete input.
then, It should be able to increase the work rate during multi-GPU running. and decrease the decode latency.
but after test, it even worse than normal layer split. my 2080ti * 2 only get 3.11 tps output.

run command:

./llama-server -m model_path -fa -cb --ctx-size 32768 -ngl 99 -np 1 -ts 1,1 -sm row