Skip to content

Commit 9316eb7

Browse files
ebattyebatty
committed
Created using Colaboratory
1 parent 595aae8 commit 9316eb7

File tree

1 file changed

+75
-50
lines changed

1 file changed

+75
-50
lines changed

‎Week10/Week10Tutorial2.ipynb‎

Lines changed: 75 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -4,27 +4,14 @@
44
"metadata": {
55
"colab": {
66
"provenance": [],
7-
"collapsed_sections": [],
8-
"toc_visible": true,
9-
"authorship_tag": "ABX9TyOxj4lg4u8IZYP0RgSz5Ion",
10-
"include_colab_link": true
7+
"authorship_tag": "ABX9TyNk3lwH1I5UBvtUCfCy6G+K"
118
},
129
"kernelspec": {
1310
"name": "python3",
1411
"display_name": "Python 3"
1512
}
1613
},
1714
"cells": [
18-
{
19-
"cell_type": "markdown",
20-
"metadata": {
21-
"id": "view-in-github",
22-
"colab_type": "text"
23-
},
24-
"source": [
25-
"<a href=\"https://colab.research.google.com/github/ebatty/MathToolsforNeuroscience/blob/jupyterbook/Week10/Week10Tutorial2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
26-
]
27-
},
2815
{
2916
"cell_type": "markdown",
3017
"metadata": {
@@ -63,7 +50,7 @@
6350
"import matplotlib as mpl\n",
6451
"from sklearn.linear_model import LinearRegression, Lasso, Ridge\n"
6552
],
66-
"execution_count": null,
53+
"execution_count": 1,
6754
"outputs": []
6855
},
6956
{
@@ -140,7 +127,7 @@
140127
" axes[1].plot(reg_validation_r2s, '-ob')\n",
141128
" axes[1].set(title='Validation data', ylabel='R2', xlabel='alpha', xticks = np.arange(0, len(alphas)), xticklabels= alphas);"
142129
],
143-
"execution_count": null,
130+
"execution_count": 2,
144131
"outputs": []
145132
},
146133
{
@@ -168,7 +155,7 @@
168155
" with open(fname, \"wb\") as fid:\n",
169156
" fid.write(r.content)\n"
170157
],
171-
"execution_count": null,
158+
"execution_count": 3,
172159
"outputs": []
173160
},
174161
{
@@ -204,7 +191,7 @@
204191
"run_test = dat['run'][2250:2500, :]\n",
205192
"\n"
206193
],
207-
"execution_count": null,
194+
"execution_count": 4,
208195
"outputs": []
209196
},
210197
{
@@ -221,7 +208,11 @@
221208
{
222209
"cell_type": "code",
223210
"metadata": {
224-
"id": "0dr0odDsd_Z8"
211+
"id": "0dr0odDsd_Z8",
212+
"outputId": "9f8abf66-c4e4-42d4-a0aa-5816dd8cdb69",
213+
"colab": {
214+
"base_uri": "https://localhost:8080/"
215+
}
225216
},
226217
"source": [
227218
"print(resps_train.shape)\n",
@@ -233,8 +224,21 @@
233224
"print(resps_test.shape)\n",
234225
"print(run_test.shape)"
235226
],
236-
"execution_count": null,
237-
"outputs": []
227+
"execution_count": 5,
228+
"outputs": [
229+
{
230+
"output_type": "stream",
231+
"name": "stdout",
232+
"text": [
233+
"(2000, 11983)\n",
234+
"(2000, 1)\n",
235+
"(250, 11983)\n",
236+
"(250, 1)\n",
237+
"(250, 11983)\n",
238+
"(250, 1)\n"
239+
]
240+
}
241+
]
238242
},
239243
{
240244
"cell_type": "markdown",
@@ -251,16 +255,18 @@
251255
"id": "6YtJ0FWFlbzg"
252256
},
253257
"source": [
254-
"First, we'll examine a standard linear regression model and examine our prediction quality on held out data. We fit this model using sklearn. \n",
258+
"First, we'll examine a standard linear regression model and examine our prediction quality on held out data. We fit this model using sklearn.\n",
255259
"\n"
256260
]
257261
},
258262
{
259263
"cell_type": "code",
260264
"source": [
261265
"from sklearn.linear_model import LinearRegression\n",
266+
"from sklearn.preprocessing import StandardScaler\n",
267+
"from sklearn.pipeline import make_pipeline\n",
262268
"\n",
263-
"no_reg_model = LinearRegression(normalize=True)\n",
269+
"no_reg_model = make_pipeline(StandardScaler(with_mean=False), LinearRegression())\n",
264270
"\n",
265271
"no_reg_model.fit(resps_train, run_train)\n",
266272
"\n",
@@ -269,18 +275,35 @@
269275
"R2 = no_reg_model.score(resps_train, run_train)"
270276
],
271277
"metadata": {
272-
"id": "89RmUIjlYOSS"
278+
"id": "89RmUIjlYOSS",
279+
"outputId": "e9ec40c5-3235-40f1-e75f-721f700c5946",
280+
"colab": {
281+
"base_uri": "https://localhost:8080/",
282+
"height": 246
283+
}
273284
},
274-
"execution_count": null,
275-
"outputs": []
285+
"execution_count": 1,
286+
"outputs": [
287+
{
288+
"output_type": "error",
289+
"ename": "NameError",
290+
"evalue": "ignored",
291+
"traceback": [
292+
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
293+
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
294+
"\u001b[0;32m<ipython-input-1-75e9c709344c>\u001b[0m in \u001b[0;36m<cell line: 7>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mno_reg_model\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmake_pipeline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mStandardScaler\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwith_mean\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mLinearRegression\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mno_reg_model\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresps_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrun_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0my_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mno_reg_model\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresps_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
295+
"\u001b[0;31mNameError\u001b[0m: name 'resps_train' is not defined"
296+
]
297+
}
298+
]
276299
},
277300
{
278301
"cell_type": "markdown",
279302
"source": [
280303
"There's four steps in the code above:\n",
281304
"\n",
282-
"- We *initialized* the model. \n",
283-
"We have `normalize = True` so the model normalizes the input data (the neural activities) before fitting.\n",
305+
"- We *initialized* the model.\n",
306+
"We have a standard scalar as the first step in our pipeline so the model standardizes the input data (the neural activities) before fitting.\n",
284307
"\n",
285308
"- We *fit* the model by passing it the training data\n",
286309
"- We *predicted* y (running speed) given resps_train.\n",
@@ -351,7 +374,7 @@
351374
},
352375
"source": [
353376
"\n",
354-
"We can see that we are predicting the training data very accurately (high R2/overlapping traces) but are not predicting the validation data as well. Seems like we're overfitting! We can try some regularization to fix this. \n",
377+
"We can see that we are predicting the training data very accurately (high R2/overlapping traces) but are not predicting the validation data as well. Seems like we're overfitting! We can try some regularization to fix this.\n",
355378
"\n",
356379
"(We are still fitting the validation data fairly well so we can decode running speed quite well from the neural activity!)"
357380
]
@@ -360,7 +383,7 @@
360383
"cell_type": "markdown",
361384
"source": [
362385
" An even better choice would have been to not have separate validation data. Instaed, we could use cross-validation to fit and evaluate on the training + validation data so we use it all. However, this takes too long for the purposes of this tutorial.\n",
363-
" \n",
386+
"\n",
364387
"Just so you know though, we can do cross-validation easily easily in one line of code using the magic of scikit-learn. See the cell below for evaluating the model using 5 folds cross validation"
365388
],
366389
"metadata": {
@@ -371,8 +394,8 @@
371394
"cell_type": "code",
372395
"source": [
373396
"from sklearn.model_selection import cross_validate\n",
374-
"scores = cross_validate(LinearRegression(normalize=True), resps_train, run_train,\n",
375-
" cv = 5, return_train_score = True, return_estimator=True) \n",
397+
"scores = cross_validate(make_pipeline(StandardScaler(with_mean=False), LinearRegression()), resps_train, run_train,\n",
398+
" cv = 5, return_train_score = True, return_estimator=True)\n",
376399
"\n",
377400
"print(f\"Cross-validated training score is {scores['train_score'].mean()}\")\n",
378401
"print(f\"Cross-validated validation score is {scores['test_score'].mean()}\")"
@@ -397,7 +420,7 @@
397420
{
398421
"cell_type": "code",
399422
"source": [
400-
"plot_weights({\"No regularization\": no_reg_model})"
423+
"plot_weights({\"No regularization\": no_reg_model[1]})"
401424
],
402425
"metadata": {
403426
"id": "pLudOSrgbsw1"
@@ -431,11 +454,11 @@
431454
"In sklearn, we fit L2 regularized linear regression models as:\n",
432455
"\n",
433456
"```\n",
434-
"model = Ridge(alpha = 1, normalize = True)\n",
457+
"model = make_pipeline(StandardScaler(with_mean=False), Ridge(alpha = 1))\n",
435458
"model.fit(resps_train, run_train)\n",
436459
"```\n",
437460
"\n",
438-
"Again, the `alpha` input tells the weighting of the regularization term in the loss function. "
461+
"Again, the `alpha` input tells the weighting of the regularization term in the loss function."
439462
]
440463
},
441464
{
@@ -463,7 +486,7 @@
463486
"ridge_train_r2s = np.zeros((len(l2_alphas)))\n",
464487
"ridge_validation_r2s = np.zeros(len(l2_alphas))\n",
465488
"\n",
466-
"# Your code here to loop over different values of alpha, fit a Ridge regression model on the training data, \n",
489+
"# Your code here to loop over different values of alpha, fit a Ridge regression model on the training data,\n",
467490
" # store the resulting R2 score on the training and validation data in the correct entry of `ridge_train_r2s` and\n",
468491
" # `ridge_validation_r2s`\n",
469492
" # Save the fit model for a given alpha in the dictionary `ridge_models` - the key should be the value of alpha\n",
@@ -484,7 +507,7 @@
484507
"id": "9Zlx82yv6xQJ"
485508
},
486509
"source": [
487-
"We can see that by increasing regularization, we decrease performance on training data. Here, we increase performance (for some values of regularization) on validation data more. For high values of regularization, our model does very badly on both train and validation data - we have to pick the regularization amount carefully! "
510+
"We can see that by increasing regularization, we decrease performance on training data. Here, we increase performance (for some values of regularization) on validation data more. For high values of regularization, our model does very badly on both train and validation data - we have to pick the regularization amount carefully!"
488511
]
489512
},
490513
{
@@ -512,10 +535,11 @@
512535
"@widgets.interact\n",
513536
"def plot_observed(alpha = l2_alphas):\n",
514537
" models = {\n",
515-
" \"No regularization\": no_reg_model,\n",
516-
" \"Regularization\": ridge_models[alpha]\n",
538+
" \"No regularization\": no_reg_model[1],\n",
539+
" \"Regularization\": ridge_models[alpha][1]\n",
517540
" }\n",
518-
" plot_weights(models)"
541+
" plot_weights(models)\n",
542+
" plt.show()"
519543
],
520544
"execution_count": null,
521545
"outputs": []
@@ -544,7 +568,7 @@
544568
"In sklearn, we fit L1 regularized linear regression models as:\n",
545569
"\n",
546570
"```\n",
547-
"model = Lasso(alpha = 1, normalize=True)\n",
571+
"model = make_pipeline(StandardScaler(with_mean=False), Lasso(alpha))\n",
548572
"model.fit(resps_train, run_train)\n",
549573
"```\n",
550574
"\n",
@@ -572,7 +596,7 @@
572596
"lasso_train_r2s = np.zeros((len(l1_alphas)))\n",
573597
"lasso_validation_r2s = np.zeros(len(l1_alphas))\n",
574598
"\n",
575-
"# Your code here to loop over different values of alpha, fit a Lasso regression model on the training data, \n",
599+
"# Your code here to loop over different values of alpha, fit a Lasso regression model on the training data,\n",
576600
" # store the resulting R2 score on the training and validation data in the correct entry of `lasso_train_r2s` and\n",
577601
" # `lasso_validation_r2s`\n",
578602
" # Save the fit model for a given alpha in the dictionary `lasso_models` - the key should be the value of alpha\n",
@@ -593,7 +617,7 @@
593617
"id": "ujEewQWG4kel"
594618
},
595619
"source": [
596-
"We can see that with increasing values of alpha, the performance on the training data decreases. On validation data, we get slightly better performance for certain values of alpha (0.001) as compared to the model with no regularization. For high values of regularization, our model does very badly on both train and validation data - we have to pick the regularization amount carefully! "
620+
"We can see that with increasing values of alpha, the performance on the training data decreases. On validation data, we get slightly better performance for certain values of alpha (0.001) as compared to the model with no regularization. For high values of regularization, our model does very badly on both train and validation data - we have to pick the regularization amount carefully!"
597621
]
598622
},
599623
{
@@ -620,10 +644,11 @@
620644
"@widgets.interact\n",
621645
"def plot_observed(alpha = l1_alphas):\n",
622646
" models = {\n",
623-
" \"No regularization\": no_reg_model,\n",
624-
" \"Regularization\": lasso_models[alpha]\n",
647+
" \"No regularization\": no_reg_model[1],\n",
648+
" \"Regularization\": lasso_models[alpha][1]\n",
625649
" }\n",
626-
" plot_weights(models)"
650+
" plot_weights(models)\n",
651+
" plt.show()"
627652
],
628653
"execution_count": null,
629654
"outputs": []
@@ -643,14 +668,14 @@
643668
"# Section 4: Choosing regularizers\n",
644669
"\n",
645670
"\n",
646-
"When should you use $L_1$ vs. $L_2$ regularization? Both penalties shrink parameters, and both will help reduce overfitting. However, the models they lead to are different. \n",
671+
"When should you use $L_1$ vs. $L_2$ regularization? Both penalties shrink parameters, and both will help reduce overfitting. However, the models they lead to are different.\n",
647672
"\n",
648673
"In particular, the $L_1$ penalty encourages *sparse* solutions in which most parameters are 0. Let's unpack the notion of sparsity.\n",
649674
"\n",
650675
"A \"dense\" vector has mostly nonzero elements:\n",
651676
"$\\begin{bmatrix}\n",
652-
" 0.1 \\\\ -0.6\\\\-9.1\\\\0.07 \n",
653-
"\\end{bmatrix}$. \n",
677+
" 0.1 \\\\ -0.6\\\\-9.1\\\\0.07\n",
678+
"\\end{bmatrix}$.\n",
654679
"A \"sparse\" vector has mostly zero elements:\n",
655680
"$\\begin{bmatrix}\n",
656681
" 0 \\\\ -0.7\\\\ 0\\\\0\n",
@@ -702,7 +727,7 @@
702727
"\n",
703728
"When is it OK to assume that the parameter vector is sparse? Whenever it is true that most features don't affect the outcome. One use-case might be decoding low-level visual features from whole-brain fMRI: we may expect only voxels in V1 and thalamus should be used in the prediction.\n",
704729
"\n",
705-
"**WARNING**: be careful when interpreting $\\theta$. Never interpret the nonzero coefficients as *evidence* that only those voxels/neurons/features carry information about the outcome. This is a product of our regularization scheme, and thus *our prior assumption that the solution is sparse*. Other regularization types or models may find very distributed relationships across the brain. Never use a model as evidence for a phenomena when that phenomena is encoded in the assumptions of the model. \n",
730+
"**WARNING**: be careful when interpreting $\\theta$. Never interpret the nonzero coefficients as *evidence* that only those voxels/neurons/features carry information about the outcome. This is a product of our regularization scheme, and thus *our prior assumption that the solution is sparse*. Other regularization types or models may find very distributed relationships across the brain. Never use a model as evidence for a phenomena when that phenomena is encoded in the assumptions of the model.\n",
706731
"\n",
707732
"\n"
708733
],

0 commit comments

Comments
 (0)