Dot product example on p 58 fig 3.8

[richlevinson-design-2]

I bought this book based on reviews and believe it is highly regarded.
I have an MS in Physics and know what a dot product is.
On p 58 fig 3.8 there are 3 dot products shown x1.x2, x2.x2, x3.x2.
I cannot get any of them to match the result.
For example x1.x2 should = .4x.5 + .1x.8 + .8x.6 = .20 + .08 + .48 = .76 = w21
but answer in fig 3.8 says w21=0.9, which does not equal 0.76.

I know this is a nit, but I am new to LLM's and trying to understand this stuff for the first time and if I can't understand how the dot products are calculated it puts me on a very unfirm foundation for continuing to read the material.

Thanks,
Rich

[rasbt]

Hi Rich,
thanks for your interest in the book.

The quick answer here is that the numbers in the figures are truncated so that it could all fit into the figure with a reasonable font size.

I remember considering / trying 3 different options:

1. The full numbers (figures were too crammed and font size was too small)
2. Truncating (has the issue that numbers in the figures can only be used for referencing the code, but you get different results if you manually multiply those, like what you observed in your case)
3. Rounding (it made it harder to cross-reference the numbers and also had the same issue as the truncation)

I hope this helps, and please let me know in case you have any follow-up questions.

[richlevinson-design-2]

Thank you so much for getting back to me on this.
I am extremely interested on learning how these LLM's work.
I will spend some time reviewing your reply and try to get some python code going to repeat the examples and see if I can get the attention weights to come out (and try to understand how the normalization process works).
What really threw me was that I couldn't get any of the 3 dot products to work, and I did read on p57 about the rounding and couldn't get that to work on x1.x2 even w rounding, so I was very unsure if I was understanding correctly.
Again, thank you very much for your quick response.
I will get back to you after I have a chance to work through this a little more.

Rich

[rasbt]

Happy to help!

So the key tip here is, if you want to calculate the dot products by hand in this chapter, don't use the values shown in the figure to do the calculations. Instead, use the numbers in the text. The numbers in the figure are just showing the truncated values and would give you the wrong results (due to the truncation) if you multiply and add them.

Anyways, I hope you enjoy the book!

[richlevinson-design-2]

Hi Sebastian (I assume it's you :) ),

Thanks for your tips on the dot product in your book (I was reading ahead
to try to get a quick take on the "attention" concepts).
I have begun ch 2 and it's time to get serious about the python.

I have begun reading App A, and kind of got stuck on whether to do the
CPU or GPU version of PyTorch. I have an older pc (~2017, I think), but
I did go to task manager and found that it has on the Performance: GPU 0
page that it has an "Intel HD Graphics 620" card, which shows:

"Shared GPU/Memory" = 0.3/15.9 GB w utilization = 1%.
Also my python version is 3.13.9, Oct 14 2025.

Is that enough info for you to suggest that I use the GPU version of PyTorch,
as it indicates you recommend on top of p 255 in App A?

Thanks,
Rich

[rasbt]

Hi Rich,

yes, that's me (Sebastian) :).

Regarding your case, that would definitely be the CPU version. The PyTorch GPU version is actually pretty specific to NVIDIA CUDA GPUs. (Although, AMD cards are now also getting better support.)

I.e., the PyTorch GPU version includes a bunch of CUDA code and CUDA itself (which is why it's almost 1 GB in size). Since CUDA code runs only on NVIDIA GPUs, downloading the PyTorch GPU version would still work on your computer (it would just run in the default CPU mode), but it would be a bit wasteful in terms of storage space.