Update the docs on -t --threads

[takasurazeem]

It's a documentation update for more clear wording on what -t parameter actually means.

[ericcurtin]

Are we sure this is correct? I was chatting with @doringeman about this recently. I tested this is the past and the default was certainly low, 4 threads (unless something changed in the meantime):

llama.cpp/ggml/include/ggml.h

Line 228 in 3a59971

#define GGML_DEFAULT_N_THREADS 4

It was super apparent in the past when testing on an Ampere system with 100+ cores

[takasurazeem]

Are we sure this is correct? I was chatting with @doringeman about this recently. I tested this is the past and the default was certainly low, 4 threads (unless something changed in the meantime):

llama.cpp/ggml/include/ggml.h

Line 228 in 3a59971

#define GGML_DEFAULT_N_THREADS 4

It was super apparent in the past when testing on an Ampere system with 100+ cores

Oh, my bad, must have been an oversight, I will go through the code and confirm. In any case the docs could benefit from more descriptive wording, thanks for the review.

[adhusch]

Are we sure this is correct? I was chatting with @doringeman about this recently. I tested this is the past and the default was certainly low, 4 threads (unless something changed in the meantime):

llama.cpp/ggml/include/ggml.h

Line 228 in 3a59971

#define GGML_DEFAULT_N_THREADS 4

It was super apparent in the past when testing on an Ampere system with 100+ cores

I can confirm that it now by default uses 100+ threads when you have 100+ cores. As this is likely not desired (but on the other hand the -1 default likely makes sense for the majority of users) the improved documentation is very valuable.

[ngxson]

This table is auto-generated, changes here will be discarded. Make your changes to arg.cpp instead.

llama.cpp/tools/server/README.md

Line 25 in eba9734

<!-- Note for contributors: The list below is generated by llama-gen-docs -->

[takasurazeem]

This table is auto-generated, changes here will be discarded. Make your changes to arg.cpp instead.

llama.cpp/tools/server/README.md

Line 25 in eba9734

<!-- Note for contributors: The list below is generated by llama-gen-docs -->

Addressed and added to correct file.

[slaren]

The default is not to use all available, the logic is more complex than that.

[takasurazeem]

Ok, I have taken out the "use all available" part, but kept the CPU because it makes it clearer. I will take a look at the logic and put up another PR with more descriptive message.

[ericcurtin]

@slaren I was thinking the same, saw it being set as 4 on an Ampere ARM machine with a bazillion cores in the past

[slaren]

There is some logic to avoid using logical cores (e.g. from SMT), but it may not work well in the Ampere CPU.