updating chapter on squashingscaler

Author: rcap107

content/chapters/05_feat_eng_numerical.qmd

@@ -10,21 +10,84 @@ format:
 ---
 
 Now that we can apply selections to any column we want thanks to `ApplyToCols` and
-the selectors, it is time to s
+the selectors, it is time to scale numerical features safely.
 
+## Numerical features with outliers
 
-1. Numerical features with outliers
-2. Regular scalers
-3. SquashingScaler
-    - SquashingScaler can't invert the transformation
+When dealing with numerical features that contain outliers (including infinite values), standard scaling methods can be problematic. Outliers can dramatically affect the centering and scaling of the entire dataset, causing the scaled inliers to be compressed into a narrow range.
 
+Consider this example:
 
-## Exercise: comparing scalers
-Scale the following data with the scikit-learn StandardScaler and RobustScaler,
-and then with the SquashingScaler.
+```{python}
+from helpers import (
+    generate_data_with_outliers,
+    plot_feature_with_outliers
+)
 
-Answer the following questions:
-- Mean/std of the transformed data
-- Max/min of the transformed data
-- Mean/std of the inliers (use provided iqr function)
+values = generate_data_with_outliers()
 
+plot_feature_with_outliers(values)
+
+```
+
+In this case, most of the values are in the range `[-2, 2]`, but there are some 
+large outliers in the range `[-40, 40]` that can cause issues when the feature 
+needs to be scaled. 
+
+
+### Regular scalers and their limitations
+
+The **StandardScaler** computes mean and standard deviation across all values.
+With outliers present, these statistics become unreliable, and the scaling factor
+can become too small, squashing inlier values.
+
+The **RobustScaler** uses quantiles (typically the 25th and 75th percentiles)
+instead of mean/std, which makes it more resistant to outliers. However, it
+doesn't bound the output values, so extreme outliers can still have very large
+scaled values.
+
+### SquashingScaler: A robust solution
+
+The `SquashingScaler` combines robust centering with smooth clipping to handle
+outliers effectively. It works in two stages:
+
+### Stage 1: Robust Scaling
+- Centers the median to zero
+- Scales using quantile-based statistics (by default, the interquartile range)
+- For columns where quantiles are equal, uses a custom MinMaxScaler
+- For columns with constant values, fills with zeros
+
+### Stage 2: Soft Clipping
+- Applies a smooth squashing function:
+$x_{\text{out}} = \frac{z}{\sqrt{1 + (z/B)^2}}$
+- Constrains all values to the range
+$[-\texttt{max\_absolute\_value}, \texttt{max\_absolute\_value}]$ (default: 3)
+- Maps infinite values to the corresponding boundaries
+- Preserves NaN values unchanged
+
+### Key advantages of `SquashingScaler`
+The `SquashingScaler` has various advantages over traditional scalers: 
+
+- It is **Outlier-resistant**: Outliers don't affect inlier scaling, unlike the
+`StandardScaler`.
+- It has **Bounded output**: All values stay in a predictable range, ideal for 
+neural networks and linear models.
+- It **Handles edge cases**: The scaler works with infinite values and constant 
+columns.
+- It **Preserves missing data**: NaN values are kept unchanged. 
+
+A disadvantage of the `SquashingScaler` is that it is**Non-invertible**: 
+The soft clipping function is smooth but cannot be exactly inverted. 
+
+## Comparison with other scalers
+
+When compared on data with outliers:
+- **StandardScaler** compresses inliers due to large scaling factors
+- **RobustScaler** preserves relative scales but allows extreme outlier values
+- **SquashingScaler** keeps inliers in a reasonable range while smoothly bounding all values
+
+If we plot the impact of each scaler on the result, this is what we can see: 
+```{python}
+from helpers import scale_feature_and_plot
+scale_feature_and_plot(values)
+```

content/chapters/helpers/__init__.py

@@ -0,0 +1,2 @@
+from .generate_synthetic_data import *
+from .plot_squashing_scaler import *

notebooks/generate_synthetic_data.py …pters/helpers/generate_synthetic_data.pynotebooks/generate_synthetic_data.py renamed to content/chapters/helpers/generate_synthetic_data.py notebooks/generate_synthetic_data.py renamed to content/chapters/helpers/generate_synthetic_data.py

@@ -0,0 +1,105 @@
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.preprocessing import RobustScaler, StandardScaler
+from skrub import SquashingScaler
+
+
+def generate_data_with_outliers():
+    np.random.seed(0)  # for reproducibility
+    values = np.random.rand(100, 1)
+    n_outliers = 15
+    outlier_indices = np.random.choice(values.shape[0], size=n_outliers, replace=False)
+    values[outlier_indices] = np.random.rand(n_outliers, 1) * 100 - 50
+    return values
+
+
+def plot_feature_with_outliers(values):
+    """Plot a feature with outliers and annotate it."""
+    x = np.arange(values.shape[0])
+    fig, axs = plt.subplots(1, layout="constrained", figsize=(6, 4))
+
+    axs.plot(x, values)
+    _ = axs.set(title="Feature with outliers", ylabel="value", xlabel="Sample ID")
+    axs.axhspan(-2, 2, color="gray", alpha=0.15)
+
+    x_data, y_data = [30, 2]
+    desc = "Data is mostly\nin [-2, 2]"
+    axs.annotate(
+        desc,
+        xy=(x_data, y_data),
+        xytext=(0.15, 0.8),
+        textcoords="axes fraction",
+        arrowprops=dict(arrowstyle="->", color="red"),
+    )
+
+    x_outlier, y_outlier = np.argmax(values), np.max(values)
+    desc = "There are large\noutliers throughout."
+    _ = axs.annotate(
+        desc,
+        xy=(x_outlier, y_outlier),
+        xytext=(0.6, 0.85),
+        textcoords="axes fraction",
+        arrowprops=dict(arrowstyle="->", color="red"),
+    )
+
+
+def scale_feature_and_plot(values):
+
+    squash_scaler = SquashingScaler()
+    squash_scaled = squash_scaler.fit_transform(values)
+
+    robust_scaler = RobustScaler()
+    robust_scaled = robust_scaler.fit_transform(values)
+
+    standard_scaler = StandardScaler()
+    standard_scaled = standard_scaler.fit_transform(values)
+
+    x = np.arange(values.shape[0])
+    fig, axs = plt.subplots(1, 2, layout="constrained", figsize=(8, 5))
+
+    ax = axs[0]
+    ax.plot(x, sorted(values), label="Original Values", linewidth=2.5)
+    ax.plot(x, sorted(squash_scaled), label="SquashingScaler")
+    ax.plot(x, sorted(robust_scaled), label="RobustScaler", linestyle="--")
+    ax.plot(x, sorted(standard_scaled), label="StandardScaler")
+
+    # Add a horizontal band in [-4, +4]
+    ax.axhspan(-4, 4, color="gray", alpha=0.15)
+    ax.set(title="Original data", xlim=[0, values.shape[0]], xlabel="Percentile")
+    ax.legend()
+
+    ax = axs[1]
+    ax.plot(x, sorted(values), label="Original Values", linewidth=2.5)
+    ax.plot(x, sorted(squash_scaled), label="SquashingScaler")
+    ax.plot(x, sorted(robust_scaled), label="RobustScaler", linestyle="--")
+    ax.plot(x, sorted(standard_scaled), label="StandardScaler")
+
+    ax.set(ylim=[-4, 4])
+    ax.set(title="In range [-4, 4]", xlim=[0, values.shape[0]], xlabel="Percentile")
+
+    # Highlight the bounds of the SquashingScaler
+    ax.axhline(y=3, alpha=0.2)
+    ax.axhline(y=-3, alpha=0.2)
+
+    fig.suptitle(
+        "Comparison of different scalers on sorted data with outliers", fontsize=20
+    )
+    fig.supylabel("Value")
+
+    desc = "The RobustScaler is\naffected by outliers"
+    axs[0].annotate(
+        desc,
+        xy=(0, -70),
+        xytext=(0.4, 0.2),
+        textcoords="axes fraction",
+        arrowprops=dict(arrowstyle="->", color="red"),
+    )
+
+    desc = "The SquashingScaler is\nclipped to a finite value"
+    _ = axs[1].annotate(
+        desc,
+        xy=(0, -3),
+        xytext=(0.4, 0.2),
+        textcoords="axes fraction",
+        arrowprops=dict(arrowstyle="->", color="red"),
+    )