Merge pull request #276 from Haleshot/haleshot/PReLU

New problem: PReLU

Author: moe18

Problems/98_PReLU/learn.md

@@ -0,0 +1,39 @@
+### Understanding the PReLU (Parametric ReLU) Activation Function
+
+The PReLU (Parametric Rectified Linear Unit) is an advanced variant of the ReLU activation function that introduces a learnable parameter for negative inputs. This makes it more flexible than standard ReLU and helps prevent the "dying ReLU" problem.
+
+#### Mathematical Definition
+
+The PReLU function is defined as:
+
+$$
+PReLU(x) = \begin{cases}
+x & \text{if } x > 0 \\
+\alpha x & \text{otherwise}
+\end{cases}
+$$
+
+Where:
+- $x$ is the input value
+- $\alpha$ is a learnable parameter (typically initialized to a small value like 0.25)
+
+#### Key Characteristics
+
+1. **Adaptive Slope**: Unlike ReLU which has a zero slope for negative inputs, PReLU learns the optimal negative slope parameter ($\alpha$) during training.
+
+2. **Output Range**: 
+   - For $x > 0$: Output equals input ($y = x$)
+   - For $x \leq 0$: Output is scaled by $\alpha$ ($y = \alpha x$)
+
+3. **Advantages**:
+   - Helps prevent the "dying ReLU" problem
+   - More flexible than standard ReLU
+   - Can improve model performance through learned parameter
+   - Maintains the computational efficiency of ReLU
+
+4. **Special Cases**:
+   - When $\alpha = 0$, PReLU becomes ReLU
+   - When $\alpha = 1$, PReLU becomes a linear function
+   - When $\alpha$ is small (e.g., 0.01), PReLU behaves similarly to Leaky ReLU
+
+PReLU is particularly useful in deep neural networks where the optimal negative slope might vary across different layers or channels.

Problems/98_PReLU/solution.py

@@ -0,0 +1,35 @@
+def prelu(x: float, alpha: float = 0.25) -> float:
+    """
+    Implements the PReLU (Parametric ReLU) activation function.
+    
+    Args:
+        x: Input value
+        alpha: Slope parameter for negative values (default: 0.25)
+    
+    Returns:
+        float: PReLU activation value
+    """
+    return x if x > 0 else alpha * x
+
+def test_prelu() -> None:
+    # Test positive input (should behave like regular ReLU)
+    assert prelu(2.0) == 2.0, "Test failed for positive input"
+    
+    # Test zero input
+    assert prelu(0.0) == 0.0, "Test failed for zero input"
+    
+    # Test negative input with default alpha
+    assert prelu(-2.0) == -0.5, "Test failed for negative input with default alpha"
+    
+    # Test negative input with custom alpha
+    assert prelu(-2.0, alpha=0.1) == -0.2, "Test failed for negative input with custom alpha"
+    
+    # Test with alpha = 0 (behaves like ReLU)
+    assert prelu(-2.0, alpha=0.0) == 0.0, "Test failed for ReLU behavior"
+    
+    # Test with alpha = 1 (behaves like linear function)
+    assert prelu(-2.0, alpha=1.0) == -2.0, "Test failed for linear behavior"
+
+if __name__ == "__main__":
+    test_prelu()
+    print("All PReLU tests passed.")