Update binary_logistic_regression_torch_with_hidden_layers.ipynb

Author: fs446

binary_logistic_regression_torch_with_hidden_layers.ipynb

@@ -27,12 +27,12 @@
    "id": "a6fe247f",
    "metadata": {},
    "source": [
-    "# Binary logistic regression model with hidden layers and a sigmoid output layer\n",
+    "# PyTorch Model for Binary Regression Using Hidden Layers and Sigmoid Output Layer\n",
     "\n",
-    "- we use **PyTorch** to train the model and to make predictions\n",
     "- we use scikit-learn for data synthesis and split\n",
+    "- we use PyTorch for model / training / prediction\n",
     "- we use scikit-learn for statistical measures\n",
-    "- see [binary_logistic_regression_tf_with_hidden_layers.ipynb](binary_logistic_regression_tf_with_hidden_layers.ipynb) for a TensorFlow implementation of the same problem"
+    "- see [binary_logistic_regression_tf_with_hidden_layers.ipynb](binary_logistic_regression_tf_with_hidden_layers.ipynb) for a TensorFlow implementation of a similar problem"
    ]
   },
   {
@@ -42,8 +42,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import numpy as np\n",
     "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
     "import sklearn\n",
     "import torch\n",
     "\n",
@@ -64,7 +64,7 @@
    "id": "696e1590",
    "metadata": {},
    "source": [
-    "## Synthesis of Data"
+    "## Data"
    ]
   },
   {
@@ -96,7 +96,13 @@
     ")\n",
     "X_train, X_test, Y_train, Y_test = train_test_split(\n",
     "    X, Y, train_size=train_size, random_state=None\n",
-    ")"
+    ")\n",
+    "M_train = X_train.shape[0]\n",
+    "M_test = X_test.shape[0]\n",
+    "print(\"M_train\", M_train)\n",
+    "print(\"X train dim\", X_train.shape, \"Y train dim\", Y_train.shape)\n",
+    "print(\"M_test\", M_test)\n",
+    "print(\"X test dim\", X_test.shape, \"Y test dim\", Y_test.shape)"
    ]
   },
   {
@@ -123,7 +129,7 @@
     "         X[Y == 0, 1],\n",
     "         \"o\", color='dodgerblue', ms=1)\n",
     "plt.axis(\"square\")\n",
-    "plt.title(\"data 1 and 0 \" + str(X.shape))\n",
+    "plt.title(\"data '1' red and '0' blue\" + str(X.shape))\n",
     "plt.xlabel(\"feature 1\")\n",
     "plt.ylabel(\"feature 2\")\n",
     "plt.axis([-10, 10, -10, 10])\n",
@@ -134,7 +140,7 @@
     "         X[Y == 1, 1],\n",
     "         \"o\", color='orangered', ms=1)\n",
     "plt.axis(\"square\")\n",
-    "plt.title(\"data 1 \" + str(X.shape))\n",
+    "plt.title(\"data '1' red \" + str(X.shape))\n",
     "plt.xlabel(\"feature 1\")\n",
     "plt.ylabel(\"feature 2\")\n",
     "plt.axis([-10, 10, -10, 10])\n",
@@ -145,7 +151,7 @@
     "         X[Y == 0, 1],\n",
     "         \"o\", color='dodgerblue', ms=1)\n",
     "plt.axis(\"square\")\n",
-    "plt.title(\"data 0 \" + str(X.shape))\n",
+    "plt.title(\"data '0' blue \" + str(X.shape))\n",
     "plt.xlabel(\"feature 1\")\n",
     "plt.ylabel(\"feature 2\")\n",
     "plt.axis([-10, 10, -10, 10])\n",
@@ -160,6 +166,15 @@
     "## Learning Parameters"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "2be2665e",
+   "metadata": {},
+   "source": [
+    "- set up hyper parameters\n",
+    "- in practice we do hyper parameter tuning, see the upcoming exercises"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -168,8 +183,8 @@
    "outputs": [],
    "source": [
     "batch_size = 2**3\n",
-    "num_epochs = 5\n",
-    "learning_rate = 0.01"
+    "num_epochs = 2**2\n",
+    "learning_rate = 0.1"
    ]
   },
   {
@@ -187,8 +202,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "Y_train = Y_train[:, np.newaxis]\n",
-    "Y_test = Y_test[:, np.newaxis]\n",
+    "Y_train = Y_train[:, None]\n",
+    "Y_test = Y_test[:, None]\n",
+    "\n",
+    "print(\"X train dim\", X_train.shape, \"Y train dim\", Y_train.shape)\n",
+    "print(\"X test dim\", X_test.shape, \"Y test dim\", Y_test.shape)\n",
     "\n",
     "data_train = TensorDataset(torch.FloatTensor(X_train),\n",
     "                           torch.FloatTensor(Y_train))\n",
@@ -279,9 +297,14 @@
    "outputs": [],
    "source": [
     "print('mps available? ', torch.backends.mps.is_available())\n",
-    "# device = torch.device('mps')\n",
+    "# device = torch.device('mps')  # Apple's Metal Performance Shaders (MPS)\n",
     "device = torch.device('cpu')\n",
     "model = model.to(device)\n",
+    "\n",
+    "# serious projects need hardware-specific compile and load\n",
+    "# model = torch.compile(model)\n",
+    "# we go for uncompiled to be hardware agnostic\n",
+    "\n",
     "print(next(model.parameters()).device)\n",
     "# check some model weights\n",
     "model.linear1.weight, model.linear1.bias"
@@ -350,10 +373,12 @@
    "outputs": [],
    "source": [
     "with torch.no_grad():\n",
+    "\n",
     "    er = empirical_risk(model.forward(\n",
     "        torch.tensor(X_train, dtype=torch.float32).to(device)),\n",
     "        torch.tensor(Y_train, dtype=torch.float32).to(device))\n",
     "    print('final train loss', er)\n",
+    "\n",
     "    er = empirical_risk(model.forward(\n",
     "        torch.tensor(X_test, dtype=torch.float32).to(device)),\n",
     "        torch.tensor(Y_test, dtype=torch.float32).to(device))\n",
@@ -399,7 +424,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "print('train')\n",
+    "print('train data')\n",
     "print(confusion_matrix(\n",
     "    y_true=Y_train,\n",
     "    y_pred=Y_pred_train,\n",
@@ -408,7 +433,7 @@
     "    y_true=Y_train,\n",
     "    y_pred=Y_pred_train,\n",
     "    normalize='all')*100)\n",
-    "print('\\n test')\n",
+    "print('\\n test data')\n",
     "print(confusion_matrix(\n",
     "    y_true=Y_test,\n",
     "    y_pred=Y_pred_test,\n",
@@ -434,12 +459,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "p, r, f, s = precision_recall_fscore_support(y_true=Y_train,\n",
-    "                                             y_pred=Y_pred_train)\n",
-    "print('train', p, r, f, s)\n",
-    "p, r, f, s = precision_recall_fscore_support(y_true=Y_test,\n",
-    "                                             y_pred=Y_pred_test)\n",
-    "print('test', p, r, f, s)"
+    "p, r, f, s = precision_recall_fscore_support(\n",
+    "    y_true=Y_train, y_pred=Y_pred_train)\n",
+    "print('train data', p, r, f, s)\n",
+    "p, r, f, s = precision_recall_fscore_support(\n",
+    "    y_true=Y_test, y_pred=Y_pred_test)\n",
+    "print('test data', p, r, f, s)"
    ]
   },
   {
@@ -459,16 +484,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "a = accuracy_score(y_true=Y_train,\n",
-    "                   y_pred=Y_pred_train)\n",
-    "ba = balanced_accuracy_score(y_true=Y_train,\n",
-    "                             y_pred=Y_pred_train)\n",
-    "print('train:', a, ba)\n",
-    "a = accuracy_score(y_true=Y_test,\n",
-    "                   y_pred=Y_pred_test)\n",
-    "ba = balanced_accuracy_score(y_true=Y_test,\n",
-    "                             y_pred=Y_pred_test)\n",
-    "print('test', a, ba)"
+    "a = accuracy_score(\n",
+    "    y_true=Y_train, y_pred=Y_pred_train)\n",
+    "ba = balanced_accuracy_score(\n",
+    "    y_true=Y_train, y_pred=Y_pred_train)\n",
+    "print('train data:', a, ba)\n",
+    "a = accuracy_score(\n",
+    "    y_true=Y_test, y_pred=Y_pred_test)\n",
+    "ba = balanced_accuracy_score(\n",
+    "    y_true=Y_test, y_pred=Y_pred_test)\n",
+    "print('test data', a, ba)"
    ]
   },
   {
@@ -486,64 +511,70 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "levels = [0.0, 0.05, 0.1, 0.37, 0.5, 0.63, 0.9, 0.95, 1]\n",
+    "if N == 2:  # 2D plot of data and classification (curved) line\n",
+    "    levels = [0.0, 0.05, 0.1, 0.37, 0.5, 0.63, 0.9, 0.95, 1]\n",
     "\n",
-    "f1, f2 = np.arange(-10, 10, 0.1), np.arange(-10, 10, 0.1)\n",
-    "xv, yv = np.meshgrid(f1, f2)\n",
-    "xv_tmp = np.reshape(xv, (xv.shape[0]*xv.shape[1], 1))\n",
-    "yv_tmp = np.reshape(yv, (yv.shape[0]*yv.shape[1], 1))\n",
-    "X_tmp = torch.tensor(np.hstack([xv_tmp, yv_tmp]),\n",
-    "                     dtype=torch.float32).to(device)\n",
-    "with torch.no_grad():\n",
-    "    Y_tmp = model.predict_class(X_tmp).cpu().detach().numpy()\n",
-    "tmp = np.reshape(Y_tmp, (xv.shape[0], -1))\n",
-    "# hard decision boundary:\n",
-    "# tmp = (tmp>=0.5) * 1\n",
-    "\n",
-    "plt.figure(figsize=(10, 10))\n",
-    "plt.subplot(2, 2, 1)\n",
-    "plt.plot(X_train[Y_train[:, 0] == 1, 0],\n",
-    "         X_train[Y_train[:, 0] == 1, 1],\n",
-    "         \"o\", color='orangered', ms=1)\n",
-    "plt.contourf(f1, f2, tmp, levels=levels, cmap=\"RdBu_r\")\n",
-    "plt.axis(\"equal\")\n",
-    "plt.colorbar()\n",
-    "plt.title(\"training \" + str(X_train.shape))\n",
-    "plt.xlabel(\"feature 1\")\n",
-    "plt.ylabel(\"feature 2\")\n",
+    "    f1, f2 = np.arange(-10, 10, 0.1), np.arange(-10, 10, 0.1)\n",
+    "    xv, yv = np.meshgrid(f1, f2)\n",
+    "    xv_tmp = np.reshape(xv, (xv.shape[0]*xv.shape[1], 1))\n",
+    "    yv_tmp = np.reshape(yv, (yv.shape[0]*yv.shape[1], 1))\n",
+    "    X_tmp = torch.tensor(np.hstack([xv_tmp, yv_tmp]),\n",
+    "                         dtype=torch.float32).to(device)\n",
+    "    with torch.no_grad():\n",
+    "        ygrid = model.forward(X_tmp).cpu().detach().numpy()\n",
+    "        # probability 0...1\n",
     "\n",
-    "plt.subplot(2, 2, 2)\n",
-    "plt.plot(X_train[Y_train[:, 0] == 0, 0],\n",
-    "         X_train[Y_train[:, 0] == 0, 1],\n",
-    "         \"o\", color='dodgerblue', ms=1)\n",
-    "plt.contourf(f1, f2, tmp, levels=levels, cmap=\"RdBu_r\")\n",
-    "plt.axis(\"equal\")\n",
-    "plt.colorbar()\n",
-    "plt.title(\"training \" + str(X_train.shape))\n",
-    "plt.xlabel(\"feature 1\")\n",
-    "plt.ylabel(\"feature 2\")\n",
+    "    # hard decision boundary:\n",
+    "    # ygrid = (ygrid >= 0.5) * 1  # binary classes {0,1}\n",
     "\n",
-    "plt.subplot(2, 2, 3)\n",
-    "plt.plot(X_test[Y_test[:, 0] == 1, 0],\n",
-    "         X_test[Y_test[:, 0] == 1, 1],\n",
-    "         \"o\", color='orangered', ms=1)\n",
-    "plt.contourf(f1, f2, tmp, levels=levels, cmap=\"RdBu_r\")\n",
-    "plt.axis(\"equal\")\n",
-    "plt.colorbar()\n",
-    "plt.title(\"test \" + str(X_test.shape))\n",
-    "plt.xlabel(\"feature 1\")\n",
-    "plt.ylabel(\"feature 2\")\n",
+    "    # reshape to plane\n",
+    "    ygrid = np.reshape(ygrid, (xv.shape[0], -1))\n",
     "\n",
-    "plt.subplot(2, 2, 4)\n",
-    "plt.plot(X_test[Y_test[:, 0] == 0, 0],\n",
-    "         X_test[Y_test[:, 0] == 0, 1],\n",
-    "         \"o\", color='dodgerblue', ms=1)\n",
-    "plt.contourf(f1, f2, tmp, levels=levels, cmap=\"RdBu_r\")\n",
-    "plt.axis(\"equal\")\n",
-    "plt.colorbar()\n",
-    "plt.title(\"test \" + str(X_test.shape))\n",
-    "plt.xlabel(\"feature 1\")\n",
-    "plt.ylabel(\"feature 2\")"
+    "    plt.figure(figsize=(12, 5))\n",
+    "    plt.subplot(1, 2, 1)  # left plot for training data set\n",
+    "    plt.plot(X_train[Y_train[:, 0] == 0, 0],\n",
+    "             X_train[Y_train[:, 0] == 0, 1],\n",
+    "             \"o\", color='dodgerblue', ms=1)\n",
+    "    plt.plot(X_train[Y_train[:, 0] == 1, 0],\n",
+    "             X_train[Y_train[:, 0] == 1, 1],\n",
+    "             \"o\", color='orangered', ms=1)\n",
+    "    plt.contourf(f1, f2, ygrid, cmap=\"RdBu_r\", levels=levels)\n",
+    "    plt.colorbar()\n",
+    "    plt.axis(\"square\")\n",
+    "    plt.xlim(-6, 6)\n",
+    "    plt.ylim(-6, 6)\n",
+    "    plt.title(\"training: \" + str(X_train.shape))\n",
+    "    plt.xlabel(\"feature 1\")\n",
+    "    plt.ylabel(\"feature 2\")\n",
+    "\n",
+    "    plt.subplot(1, 2, 2)  # right plot for test data set\n",
+    "    plt.plot(X_test[Y_test[:, 0] == 0, 0],\n",
+    "             X_test[Y_test[:, 0] == 0, 1],\n",
+    "             \"o\", color='dodgerblue', ms=1)\n",
+    "    plt.plot(X_test[Y_test[:, 0] == 1, 0],\n",
+    "             X_test[Y_test[:, 0] == 1, 1],\n",
+    "             \"o\", color='orangered', ms=1)\n",
+    "    plt.contourf(f1, f2, ygrid, cmap=\"RdBu_r\", levels=levels)\n",
+    "    plt.colorbar()\n",
+    "    plt.axis(\"square\")\n",
+    "    plt.xlim(-6, 6)\n",
+    "    plt.ylim(-6, 6)\n",
+    "    plt.title(\"test: \" + str(X_test.shape))\n",
+    "    plt.xlabel(\"feature 1\")\n",
+    "    plt.ylabel(\"feature 2\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "43218100",
+   "metadata": {},
+   "source": [
+    "## Nice to think about\n",
+    "\n",
+    "- Instead of using `activation='tanh'` in the dense layers, we could experience that `activation='relu'` yields a more piece-wise linear classification boundary line.\n",
+    "- Please have a look at the classification boundary line and guess how many coefficients a polynomial or spline curve would need to create such a curve. A optimum model should exhibit about same parameter number to create this classification curve. Certainly, thousands of model parameters for this data example is too much. We see: mathematical thinking and and creating expectations for the given problem, rather than just playing around with models (and by that wasting energy), is a valuable, mandatory human skill.\n",
+    "- For more than two features `N` we cannot conveniently plot the data sets and boundary lines anymore. Hence, instead of having visual contact to the data and classification, we heavily rely on the performances measures. So, we make sure that we fully understand the given numbers.\n",
+    "- How to choose the best number of features? How to find the best model? When is a model nicely trained? These are the important questions for real applications. We soon learn about hyper parameter tuning, regularization."
    ]
   },
   {