rpc : report actual free memory

[rgerganov]

Start tracking the free memory on every device and report it appropriately.

Start reporting the free memory on every device instead of using fixed values. Now llama-cli users can get a nice memory breakdown when using RPC devices.

[rgerganov]

@slaren I am thinking to drop the -m option of the rpc-server which overrides the reported total_mem. It is very confusing as it is not enforced and users can achieve the same with --tensor-split on the client side. What do you think?

[slaren]

@slaren I am thinking to drop the -m option of the rpc-server which overrides the reported total_mem. It is very confusing as it is not enforced and users can achieve the same with --tensor-split on the client side. What do you think?

Yes, I think that would be a good change.