Update Theory_02.ipynb

Co-Authored-By: Ranuga <[email protected]>

Author: Programmer-RD-AI

Theory_02.ipynb

@@ -5,32 +5,82 @@
    "id": "bb5eee40-7a7b-4d5c-ab73-800776e96510",
    "metadata": {},
    "source": [
-    "# Crross Entropy Loss\n"
+    "# Crross Entropy Loss\n",
+    "\n",
+    "https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "49dd39b1-221c-40fa-a34c-a548b7acca8d",
    "metadata": {},
    "source": [
-    "# Dropout"
+    "# Dropout\n",
+    "\n",
+    "Dropout is used to make the model not over fit and it reduces the number of nodes used or essentially deactivates them\n",
+    "\n",
+    "By dropping a unit out, we mean temporarily removing it from the network, along with all its incoming and outgoing connections\n",
+    "\n",
+    "Dilution and dropout are regularization techniques for reducing overfitting in artificial neural networks by preventing complex co-adaptations on training data. They are an efficient way of performing model averaging with neural networks. Wikipedia\n",
+    "\n",
+    "https://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "311e5114-dd67-4d9d-a837-f6d8a7a53e2c",
    "metadata": {},
    "source": [
-    "# Batch normalization"
+    "# Batch normalization\n",
+    "\n",
+    "Another way to prevent model overfitting\n",
+    "\n",
+    "Normalization is a data preprocessing too used\n",
+    "\n",
+    "When we normalise the model we can generalize its weights and biases\n",
+    "\n",
+    "The batch normalization is kind of a filter where it blends up all of the information\n",
+    "\n",
+    "https://arxiv.org/pdf/1502.03167.pdf"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "id": "05e944ee-5b08-4ff8-9cfb-25fd73eed387",
    "metadata": {},
    "outputs": [],
    "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "2ab2057f-a717-4a50-9245-d1c530ff4bf0",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(tensor([-0.7186, -0.2342,  1.0638,  1.7499,  0.0266,  0.2970, -0.0000, -0.4345,\n",
+       "         -0.0000,  0.0000, -2.3886,  0.0000, -0.2310,  2.1409, -0.9096, -0.7719]),\n",
+       " tensor([-0.5749, -0.1873,  0.8511,  1.3999,  0.0213,  0.2376, -0.6571, -0.3476,\n",
+       "         -0.0300,  0.8420, -1.9109,  1.0163, -0.1848,  1.7127, -0.7276, -0.6175]))"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0cb61562-a5c7-445c-8218-4752a52d24c1",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {