Module 02: Overflow in Sigmoid unit test, slightly confusing warning message

[asgalon]

This showed up when running the test_unit_sigmoid cell:

That is a bit of a mixed message there. Looking at the test, the assertion holds for the Tensor([-1000,1000]), so the warning for overflow is probably expected. Perhaps it would be a good idea to add a comment saying so?

Same happens in the module integration test below, and in the Aha moment output

[adityamulik]

I faced this issue too, I think the np.exp() function with float32 dtype is not able to handle larger numbers. I tried fixing this.
Below was the claude response:

Based on the below, 88.72 is max else it will overflow.

float32:  np.exp(x) overflows when x > 88.72
float64:  np.exp(x) overflows when x > 709.78

Workaround for the Sigmoid forward function:

exp_neg_abs = np.exp(-np.abs(x.data))  # always safe, always tiny
result = np.where(
    x.data >= 0,
    1.0 / (1.0 + exp_neg_abs),           # positive x → standard formula
    exp_neg_abs / (1.0 + exp_neg_abs)    # negative x → rearranged formula
)
return Tensor(result)

[asgalon]

I think that the overflow is expected here. The problem is merely that the warning is a bit confusing because it looks like something that should be fixed, while the test explicitly checks that the result is within the expected range when exp overflows. So either this warning should be suppressed, or explicitly advertised as expected. The red color does not really help here. The assertion passes and looks useable for the expected use cases of Sigmoid, so the test result for tensor values that cause exp to overflow is correct. There is no need to complicate the algorithm, just to get rid of the confusing warning, I'd say.

[asgalon]

I think the correct way to deal with this is to set

np.warnings.filterwarnings('ignore', category=RuntimeWarning)

just not sure where is the best place. On the other hand, this hides the fact that an overflow happens, so from an educational standpoint it may be preferable to just put a note in before it happens that this warning is expected. That is more a question of style, perhaps.