[wip] fix some hackable level 1 problems

[bkal01]

As stated in this issue and this PR by @doux-jy, it's possible to hack 94: MSE via the statistical properties of the input.

96: Huber Loss and 99: Hinge Loss also face the same problem and we can fix these issues the same way, by sampling inputs from a heavy-tailed distribution.

Huber Loss

For the Huber Loss, given some targets $y$ and predictions $\hat{y}$, we compute:

$$ L(y, \hat{y} = \begin{cases} 0.5(y-\hat{y})^2 & \text{if} |y - \hat{y}|\leq 1 \\ |y - \hat{y}| - 0.5& \text{otherwise} \end{cases} $$

If $y, \hat{y}, s \sim \text{Unif}(0,1)$, where $s$ is the scale, then $|y - \hat{y}|$ can never exceed 1, implying that the Huber Loss problem is functionally just computing the MSE Loss. Since we know this can be hacked via direct expectation computation, the Huber Loss problem is also vulnerable.

Hinge Loss

For the Hinge Loss, given some targets $y$ that are $\pm 1$ and predictions $\hat{y}$, we compute:

$$ L(y_i, \hat{y_i}) = \text{max}(0, 1 - y_i\cdot \hat{y}_i) $$

averaged across all indices. Given that $\hat{y}\sim\text{Unif}(0,1)$ and $y_i$ is $\pm 1$ with equal probability, we can say:

$$ L(y_i, \hat{y_i}) = \begin{cases} \mathbb{E}[1 - \hat{y_i}] & y_i = 1 \\ \mathbb{E}[1 + \hat{y_i}] & y_i = -1 \end{cases} $$

So,

$$\mathbb{E}[L | y] = \frac{0.5n_{\text{pos}} + 1.5n_{\text{neg}}}{n}$$ $$= 1.5 - \frac{n_{\text{pos}}}{n}$$

which we get by substituting $n_{\text{neg}} = n - n_{\text{pos}}$.

Finally, by construction of $y$:

$$\frac{1}{n}\sum_{i=1}^{n}y_i = \frac{n_{\text{pos}} - n_{\text{neg}}}{n}$$ $$ = \frac{2n_{\text{pos}}}{n} - 1$$

This means we can use the mean of the targets to compute the expected value of the output (1.0 - 0.5*targets.mean())

[doux-jy]

Thanks for your PR addressing the MSELoss validation issue. I noticed you used Pareto(0.01, 0.15) for the fix.

A quick technical note: with $\alpha = 0.15$, the distribution's first moment(mean) doesn't converge. This means MSE's expected value becomes mathematically unstable, potentially undermining the validation's reliability.

For Pareto distributions:
$\alpha \le 1$: first moment diverges
$1 < \alpha \le 2$: first moment converges, second moment diverges ✓
$\alpha > 2$: both moments converge

We need $1 < \alpha \le 2$ to ensure MSE grows with sample size while keeping expectations stable. I'd suggest using $\alpha = 1.5$ (as in my original proposal) or another value in $(1, 2]$. This ensures the validation properly detects partial computations.

Happy to discuss further!

[bkal01]

whoops you are totally right! mistake on my part moving the decimal point up.

[simonguozirui]

Changes make sense to me and good that we are improving this on the problem definition side as people might just take the problems for their use case(we can do more on the eval fuzzing side too).

Since now KenrelBench supports more precision (bf16 and fp16 in addition to the original fp32), let's make sure this doesn't cause trouble with overflowing, etc.

[doux-jy]

Changes make sense to me and good that we are improving this on the problem definition side as people might just take the problems for their use case(we can do more on the eval fuzzing side too).

Since now KenrelBench supports more precision (bf16 and fp16 in addition to the original fp32), let's make sure this doesn't cause trouble with overflowing, etc.

My replication in Issue 97 may help.