Merge pull request #320 from Haleshot/haleshot/softplus

Add New Problem 99: Softplus

Author: moe18

Problems/99_Softplus/learn.md

@@ -0,0 +1,42 @@
+### Understanding the Softplus Activation Function
+
+The Softplus activation function is a smooth approximation of the ReLU function. It's used in neural networks where a smoother transition around zero is desired. Unlike ReLU which has a sharp transition at x=0, Softplus provides a more gradual change.
+
+### Mathematical Definition
+
+The Softplus function is mathematically defined as:
+
+$$
+Softplus(x) = \log(1 + e^x)
+$$
+
+Where:
+- $x$ is the input to the function
+- $e$ is Euler's number (approximately 2.71828)
+- $\log$ is the natural logarithm
+
+### Characteristics
+
+1. **Output Range**: 
+   - The output is always positive: $(0, \infty)$
+   - Unlike ReLU, Softplus never outputs exactly zero
+
+2. **Smoothness**:
+   - Softplus is continuously differentiable
+   - The transition around x=0 is smooth, unlike ReLU's sharp "elbow"
+
+3. **Relationship to ReLU**:
+   - Softplus can be seen as a smooth approximation of ReLU
+   - As x becomes very negative, Softplus approaches 0
+   - As x becomes very positive, Softplus approaches x
+
+4. **Derivative**:
+   - The derivative of Softplus is the logistic sigmoid function:
+   $$
+   \frac{d}{dx}Softplus(x) = \frac{1}{1 + e^{-x}}
+   $$
+
+### Use Cases
+- When smooth gradients are important for optimization
+- In neural networks where a continuous approximation of ReLU is needed
+- Situations where strictly positive outputs are required with smooth transitions

Problems/99_Softplus/solution.py

@@ -0,0 +1,40 @@
+import math
+
+def softplus(x: float) -> float:
+    """
+    Compute the softplus activation function.
+    
+    Args:
+        x: Input value
+        
+    Returns:
+        The softplus value: log(1 + e^x)
+    """
+    # To prevent overflow for large positive values
+    if x > 100:
+        return x
+    # To prevent underflow for large negative values
+    if x < -100:
+        return 0.0
+    
+    return math.log(1.0 + math.exp(x))
+
+def test_softplus():
+    # Test case 1: x = 0
+    assert abs(softplus(0) - math.log(2)) < 1e-6, "Test case 1 failed"
+    
+    # Test case 2: large positive number
+    assert abs(softplus(100) - 100) < 1e-6, "Test case 2 failed"
+    
+    # Test case 3: large negative number
+    assert abs(softplus(-100)) < 1e-6, "Test case 3 failed"
+    
+    # Test case 4: positive number
+    assert abs(softplus(2) - 2.1269280110429727) < 1e-6, "Test case 4 failed"
+    
+    # Test case 5: negative number
+    assert abs(softplus(-2) - 0.12692801104297272) < 1e-6, "Test case 5 failed"
+
+if __name__ == "__main__":
+    test_softplus()
+    print("All Softplus tests passed.")