loader: refactor tensor weights storage

[kylo5aby]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[ngxson]

IMO this will be quite redundant, because:

• get_weights is only used when loading model tensors, so it's not worth optimizing it
• Total number of tensors is usually small (usually less than of 1000?), so a linear search is ok in most cases

[slaren]

I think the goal of the change is good, the way we use this list of tensors is effectively $O(N^2)$, which may not be a problem right now, but it doesn't take a lot to make an algorithm with this complexity a serious performance problem. However, duplicating the list of weights as a map is very error-prone and will make it harder to work with this code in future. A more ambitious refactor that completely replaces the vector of weights with a map would be welcome.

[kylo5aby]

updated. PTAL

[slaren]

The side effect of using a unordered_map was that it caused weights to appear in a random order when quantizing a model, which may be confusing. I changed it to a map and added a custom comparator that sorts the weight by layer. As a side benefit, this may reduce memory usage when using mmap since it ensures that the tensors used by the CPU are in a contiguous block at the beginning of the file, which allows unmapping the weights offloaded to a GPU cleanly.