-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathmanual.html
More file actions
415 lines (300 loc) · 34.6 KB
/
manual.html
File metadata and controls
415 lines (300 loc) · 34.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Manual — Snap Machine Learning documentation</title>
<link rel="shortcut icon" href="_static/favicon.ico"/>
<script type="text/javascript" src="_static/js/modernizr.min.js"></script>
<script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/language_data.js"></script>
<script type="text/javascript" src="_static/js/theme.js"></script>
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Tutorials" href="tutorials.html" />
<link rel="prev" title="Introduction" href="index.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="index.html" class="icon icon-home"> Snap Machine Learning
</a>
<div class="version">
1.3.0
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<p class="caption"><span class="caption-text">Overview</span></p>
<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="#">Manual</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#linear-regression">Linear Regression</a></li>
<li class="toctree-l2"><a class="reference internal" href="#support-vector-machine">Support Vector Machine</a></li>
<li class="toctree-l2"><a class="reference internal" href="#logistic-regression">Logistic Regression</a></li>
<li class="toctree-l2"><a class="reference internal" href="#decision-tree">Decision Tree</a></li>
<li class="toctree-l2"><a class="reference internal" href="#random-forest">Random Forest</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#references">References</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="tutorials.html">Tutorials</a></li>
<li class="toctree-l1"><a class="reference internal" href="frequentlyaskedquestions.html">FAQ</a></li>
</ul>
<p class="caption"><span class="caption-text">pai4sk ML APIs</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="ridgedoc.html">linear_model.Ridge</a></li>
<li class="toctree-l1"><a class="reference internal" href="lassodoc.html">linear_model.Lasso</a></li>
<li class="toctree-l1"><a class="reference internal" href="sklogregdoc.html">linear_model.LogisticRegression</a></li>
<li class="toctree-l1"><a class="reference internal" href="svcdoc.html">svm.LinearSVC</a></li>
<li class="toctree-l1"><a class="reference internal" href="kmeansdoc.html">cluster.KMeans</a></li>
<li class="toctree-l1"><a class="reference internal" href="dbscandoc.html">cluster.DBSCAN</a></li>
<li class="toctree-l1"><a class="reference internal" href="pcadoc.html">decomposition.PCA</a></li>
<li class="toctree-l1"><a class="reference internal" href="svddoc.html">decomposition.TruncatedSVD</a></li>
</ul>
<p class="caption"><span class="caption-text">pai4sk Loaders APIs</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="sksvmloaderfiledoc.html">load_svmlight_file</a></li>
</ul>
<p class="caption"><span class="caption-text">pai4sk Metrics APIs</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="sklogdoc.html">log_loss</a></li>
<li class="toctree-l1"><a class="reference internal" href="skaccdoc.html">accuracy_score</a></li>
<li class="toctree-l1"><a class="reference internal" href="skhingedoc.html">hinge_loss</a></li>
<li class="toctree-l1"><a class="reference internal" href="skmsedoc.html">mean_squared_error</a></li>
</ul>
<p class="caption"><span class="caption-text">snapML APIs</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="linregapidoc.html">LinearRegression</a></li>
<li class="toctree-l1"><a class="reference internal" href="logregapidoc.html">LogisticRegression</a></li>
<li class="toctree-l1"><a class="reference internal" href="svmapidoc.html">SVM</a></li>
<li class="toctree-l1"><a class="reference internal" href="dectreeapidoc.html">DecisionTreeClassifier</a></li>
<li class="toctree-l1"><a class="reference internal" href="ranforapidoc.html">RandomForestClassifier</a></li>
<li class="toctree-l1"><a class="reference internal" href="logdoc.html">log_loss</a></li>
<li class="toctree-l1"><a class="reference internal" href="accdoc.html">accuracy_score</a></li>
<li class="toctree-l1"><a class="reference internal" href="hingedoc.html">hinge_loss</a></li>
<li class="toctree-l1"><a class="reference internal" href="msedoc.html">mean_squared_error</a></li>
</ul>
<p class="caption"><span class="caption-text">snapML Loaders APIs</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="svmloaderdoc.html">load_from_svmlight_format</a></li>
<li class="toctree-l1"><a class="reference internal" href="snaploaderdoc.html">load_from_snap_format</a></li>
<li class="toctree-l1"><a class="reference internal" href="snaploaderfiledoc.html">load_snap_file</a></li>
<li class="toctree-l1"><a class="reference internal" href="snapwritedoc.html">write_to_snap_format</a></li>
</ul>
<p class="caption"><span class="caption-text">snapML Spark APIs</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="splinregdoc.html">LinearRegression</a></li>
<li class="toctree-l1"><a class="reference internal" href="splogregdoc.html">LogisticRegression</a></li>
<li class="toctree-l1"><a class="reference internal" href="spsvmdoc.html">SupportVectorMachine</a></li>
<li class="toctree-l1"><a class="reference internal" href="spreaddoc.html">DatasetReader</a></li>
<li class="toctree-l1"><a class="reference internal" href="spmetdoc.html">Metrics</a></li>
<li class="toctree-l1"><a class="reference internal" href="sputildoc.html">Utils</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">Snap Machine Learning</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> »</li>
<li>Manual</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="manual">
<span id="id1"></span><h1>Manual<a class="headerlink" href="#manual" title="Permalink to this headline">¶</a></h1>
<p>The Snap Machine Learning (Snap ML) Library is designed to offer fast training of generalized linear models and tree based models. The library is under development and currently supports the following machine learning models:</p>
<ul class="simple">
<li><a class="reference internal" href="#linear-regression"><span class="std std-ref">Linear Regression</span></a></li>
<li><a class="reference internal" href="#svm"><span class="std std-ref">Support Vector Machine</span></a></li>
<li><a class="reference internal" href="#logistic-regression"><span class="std std-ref">Logistic Regression</span></a></li>
<li><a class="reference internal" href="#decision-tree"><span class="std std-ref">Decision Tree</span></a></li>
<li><a class="reference internal" href="#random-forest"><span class="std std-ref">Random Forest</span></a></li>
</ul>
<div class="section" id="linear-regression">
<span id="id2"></span><h2>Linear Regression<a class="headerlink" href="#linear-regression" title="Permalink to this headline">¶</a></h2>
<p><code class="xref py py-class docutils literal notranslate"><span class="pre">LinearRegression</span></code> fits a linear model with coefficients <img class="math" src="_images/math/3a9389c304659d99ca69cbc2d2e3b81d82dd2509.png" alt="{\bf w} = (w_1, ..., w_d)"/> to minimize the residual sum
of squares between the predicted responses <img class="math" src="_images/math/9a594ed1e3d9f589e0f4d626f83686f11b408899.png" alt="\bf \hat{y}"/> and the true labels <img class="math" src="_images/math/1c4a42645c5957e44407faa18fcbc95ec3aad3e7.png" alt="\bf y"/> of the training data.</p>
<p>In order to prevent the model from overfitting you have the option to impose an
<img class="math" src="_images/math/86233e6ab8aa6565d22ae73dc8a75da12dde7476.png" alt="L_1"/> or an <img class="math" src="_images/math/cce73c20b14f5d57454e0ad66f02dd004d949e0c.png" alt="L_2"/> norm penalty on the size of the coefficients.</p>
<ul>
<li><p class="first"><strong>Ridge Regression</strong> adds an <img class="math" src="_images/math/cce73c20b14f5d57454e0ad66f02dd004d949e0c.png" alt="L_2"/>-regularization term to the least-squares loss.
Mathematically it solves the following optimization problem:</p>
<div class="math">
<p><img src="_images/math/c93020fcd5249a1ef9708107f3f380a22bc41b29.png" alt="\underset{{\bf w}}{\min\,} \frac 1 {2}|| X^\top {\bf w} - {\bf y}||_2^2 + \frac{\lambda} 2 \|\bf w\|_2^2"/></p>
</div></li>
<li><p class="first"><strong>Lasso</strong> adds an <img class="math" src="_images/math/86233e6ab8aa6565d22ae73dc8a75da12dde7476.png" alt="L_1"/>-regularization term to the least-squares loss.
Mathematically it solves the following optimization problem:</p>
<div class="math">
<p><img src="_images/math/a3588f4b291e3d8b03f1291958c840fc252f163c.png" alt="\underset{{\bf w}}{\min\,} \frac 1 {2}|| X^\top {\bf w} - {\bf y}||_2^2 + {\lambda} \|\bf w\|_1"/></p>
</div></li>
</ul>
<p>In both cases <img class="math" src="_images/math/2e2f8e3140e270db5d03a8519a19f8455bd665c0.png" alt="X=[{\bf x}_1,...,{\bf x}_n]"/> denotes the training data matrix with samples <img class="math" src="_images/math/8ac96ca9da02ddf3f547e5dc1e1cdadef1bd8618.png" alt="\{{\bf x}_i\}_{i\in [n]}"/> in its columns and <img class="math" src="_images/math/07f6018e00c747406442bb3912e0209766fc9090.png" alt="y_i"/> are the corresponding labels.
The regularization strength is controlled by the regularization parameter <img class="math" src="_images/math/78bc838c22c8244e929d7d8967de830757d28252.png" alt="\lambda\geq 0"/>; the larger <img class="math" src="_images/math/76f1d8ace30435987c01a00ca53a71cba1f40e6c.png" alt="\lambda"/> the more robust the model becomes to overfitting.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last"><em>(Regularization Parameter)</em>
In order to find an appropriate regularization parameter we recommend to perform cross validation.</p>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last"><em>(Feature Selection)</em>
As Lasso regression yields sparse models it can be used to perform feature selection. The sparsity can be controlled by <img class="math" src="_images/math/76f1d8ace30435987c01a00ca53a71cba1f40e6c.png" alt="\lambda"/> - a larger regularization parameter encourages more sparsity.</p>
</div>
<p><em>Snap ML</em> implements different variants of stochastic coordinate descent <a class="reference internal" href="#references">[SCD]</a> and stochastic dual coordinate ascent <a class="reference internal" href="#references">[SDCA]</a> as an algorithm to fit the model parameters <img class="math" src="_images/math/8760a10242fd9b6c3fd1dc5a0712e7de51953c79.png" alt="\bf w"/>.
In order to optimally support GPUs for training <em>Snap ML</em> implements a parallel asynchronous version of these solvers especially designed to leverage the massive parallelism of modern GPUs <a class="reference internal" href="#references">[TPASCD]</a>.</p>
<p>To train the <code class="xref py py-class docutils literal notranslate"><span class="pre">LinearRegression</span></code> model the <code class="docutils literal notranslate"><span class="pre">fit</span></code> method is used; it takes the training data and the labels <img class="math" src="_images/math/40e5819f6e7c5bd244e0e0058bb5fbabec1c7a7d.png" alt="X,\bf y"/> as input and stores the learnt coefficients of the model <img class="math" src="_images/math/8760a10242fd9b6c3fd1dc5a0712e7de51953c79.png" alt="\bf w"/> in its <code class="docutils literal notranslate"><span class="pre">coef_</span></code> member function.
The regularization type can be specified during initialization using the <code class="docutils literal notranslate"><span class="pre">penalty</span></code> argument.
The trained model can then be used to make predictions by calling the <code class="docutils literal notranslate"><span class="pre">predict</span></code> method on unlabelled data.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pai4sk</span> <span class="kn">import</span> <span class="n">LinearRegression</span>
<span class="gp">>>> </span><span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
<span class="gp">>>> </span><span class="n">reg</span> <span class="o">=</span> <span class="n">LinearRegression</span><span class="p">(</span><span class="n">max_iter</span> <span class="o">=</span> <span class="mi">100</span><span class="p">,</span> <span class="n">regularizer</span> <span class="o">=</span> <span class="mf">0.1</span><span class="p">,</span> <span class="n">penalty</span> <span class="o">=</span> <span class="s1">'l2'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">X_train</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">]])</span>
<span class="gp">>>> </span><span class="n">y_train</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">reg</span><span class="o">.</span><span class="n">fit</span> <span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">reg</span><span class="o">.</span><span class="n">coef_</span>
<span class="go">[ 0.495, 0.495]</span>
<span class="gp">>>> </span><span class="n">X_test</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">]])</span>
<span class="gp">>>> </span><span class="n">reg</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="go">[2.97, 0.495]</span>
</pre></div>
</div>
<p>For more details about the API we refer to the <a class="reference internal" href="pythonapidocumentation.html#python-api-documentation"><span class="std std-ref">snap-ml API</span></a>.</p>
</div>
<div class="section" id="support-vector-machine">
<span id="svm"></span><h2>Support Vector Machine<a class="headerlink" href="#support-vector-machine" title="Permalink to this headline">¶</a></h2>
<p>Support Vector Machine (SVM) is a supervised learning method which can be applied for regression as well as classification.
Currently <em>Snap ML</em> implements <code class="xref py py-class docutils literal notranslate"><span class="pre">SupportVectorMachine</span></code> (SVMs) with a linear kernel function and offers
<img class="math" src="_images/math/cce73c20b14f5d57454e0ad66f02dd004d949e0c.png" alt="L_2"/> regularization to prevent the model from overfitting.</p>
<p>Mathematically it solves the following optimization problem:</p>
<div class="math">
<p><img src="_images/math/0eddea35f4c53c5f10f013bc00f7f2a9c3baf780.png" alt="\underset{{\bf w}}{\min\,} \sum_i [{\bf x}_i^\top {\bf w} - y_i]_+ +\frac \lambda 2 \|{\bf w}\|_2^2"/></p>
</div><p>where <img class="math" src="_images/math/07a7ad7bf78ce80c574857ce9a949a157eac470c.png" alt="[u]_+=\max(u,0)"/> denotes the hinge loss with <img class="math" src="_images/math/7f5ec56efd11f6bb4c5619c7576bc3cb7e8635c2.png" alt="\{x_i\}_{i\in [n]}"/> being the training samples and <img class="math" src="_images/math/eaea97645a8bb3b7b8f716dcdfa8e9798266c9d2.png" alt="y_i \in \{\pm 1\}"/> the corresponding labels. The regularization strength <img class="math" src="_images/math/5831214b5e489a86ddc202d723eaf7939dddef5e.png" alt="\lambda>0"/> can be controlled by the user through the <code class="docutils literal notranslate"><span class="pre">regularizer</span></code> parameter.
The larger <img class="math" src="_images/math/76f1d8ace30435987c01a00ca53a71cba1f40e6c.png" alt="\lambda"/> the more robust the model becomes to overfitting.</p>
<p><em>Snap ML</em> implements stochastic dual coordinate <a class="reference internal" href="#references">[SDCA]</a> and the GPU optimized <a class="reference internal" href="#references">[TPASCD]</a> as an algorithm to train the SVM classifier.
SDCA runs on the equivalent SVM dual problem formulation:</p>
<div class="math">
<p><img src="_images/math/1c8973e1300a0c4789ff30ee95c792e5c05d3e64.png" alt="\underset{\boldsymbol \alpha}{\min\,} \sum_i -\alpha_i y_i +\frac 1 {2\lambda} \|X \boldsymbol \alpha\|_2^2"/></p>
</div><p>with the constraint <img class="math" src="_images/math/aaa5eeca78a2f0cc50f1abb79b78effab16d13d7.png" alt="\alpha_i y_i\in[0,1]"/>.</p>
<p>To train the model the <code class="docutils literal notranslate"><span class="pre">fit</span></code> method is used; it takes the training data and the labels <img class="math" src="_images/math/00626504d21b91c1346f484e605444a773467121.png" alt="X,{\bf y}"/> as input and stores the learnt coefficients of the model <img class="math" src="_images/math/d0025313425983b19fea4533e1400d1ac30f9ae9.png" alt="{\bf w}"/> in its <code class="docutils literal notranslate"><span class="pre">coef_</span></code> member function. The trained model can then be used to make predictions by calling the <code class="docutils literal notranslate"><span class="pre">predict</span></code> method on unlabelled data.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pai4sk</span> <span class="kn">import</span> <span class="n">SupportVectorMachine</span>
<span class="gp">>>> </span><span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
<span class="gp">>>> </span><span class="n">reg</span> <span class="o">=</span> <span class="n">SupportVectorMachine</span><span class="p">(</span><span class="n">max_iter</span> <span class="o">=</span> <span class="mi">100</span><span class="p">,</span> <span class="n">regularizer</span> <span class="o">=</span> <span class="mf">1.0</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">X_train</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">]])</span>
<span class="gp">>>> </span><span class="n">y_train</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">reg</span><span class="o">.</span><span class="n">fit</span> <span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">reg</span><span class="o">.</span><span class="n">coef_</span>
<span class="go">[ 0.25, 0.25]</span>
<span class="gp">>>> </span><span class="n">X_test</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">]])</span>
<span class="gp">>>> </span><span class="n">reg</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="go">[1,-1]</span>
</pre></div>
</div>
<p>A full example of training a <code class="xref py py-class docutils literal notranslate"><span class="pre">SupportVectorMachine</span></code> model in a real application can be found in the PowerAI distribution under <em>/opt/DL/snap-ml-local/examples</em>. For more details about the API we refer to the <a class="reference internal" href="pythonapidocumentation.html#python-api-documentation"><span class="std std-ref">snap-ml API</span></a>.</p>
</div>
<div class="section" id="logistic-regression">
<span id="id3"></span><h2>Logistic Regression<a class="headerlink" href="#logistic-regression" title="Permalink to this headline">¶</a></h2>
<p><code class="xref py py-class docutils literal notranslate"><span class="pre">LogisticRegression</span></code> is a linear model for classification. A logistic model is used to estimate the probability of an outcome based on the features of the input data. In order to prevent the model from overfitting <img class="math" src="_images/math/cce73c20b14f5d57454e0ad66f02dd004d949e0c.png" alt="L_2"/> or <img class="math" src="_images/math/86233e6ab8aa6565d22ae73dc8a75da12dde7476.png" alt="L_1"/> regularization can be used.</p>
<p>Mathematically <img class="math" src="_images/math/cce73c20b14f5d57454e0ad66f02dd004d949e0c.png" alt="L_2"/>-regularized Logistic Regression solves the following optimization problem composing of the logistic loss and an <img class="math" src="_images/math/cce73c20b14f5d57454e0ad66f02dd004d949e0c.png" alt="L_2"/> regularization term:</p>
<div class="math">
<p><img src="_images/math/d351555f098316dadc6adcf99ddaab9d753236ca.png" alt="\underset{{\bf w}}{\min\,} \sum_i \log(1+\exp(-y_i {\bf x}_i^\top {\bf w})) + \frac \lambda 2 \|{\bf w}\|_2^2"/></p>
</div><p>Similarly, <img class="math" src="_images/math/86233e6ab8aa6565d22ae73dc8a75da12dde7476.png" alt="L_1"/>-regularized Logistic Regression solves the following optimization problem:</p>
<div class="math">
<p><img src="_images/math/f0aac74892db25aa15b8cf1e2a0a8593380cd9c7.png" alt="\underset{{\bf w}}{\min\,} \sum_i \log(1+\exp(-y_i {\bf x}_i^\top {\bf w})) + \lambda \|{\bf w}\|_1"/></p>
</div><p>where <img class="math" src="_images/math/2e2f8e3140e270db5d03a8519a19f8455bd665c0.png" alt="X=[{\bf x}_1,...,{\bf x}_n]"/> is the training data matrix with samples <img class="math" src="_images/math/8ac96ca9da02ddf3f547e5dc1e1cdadef1bd8618.png" alt="\{{\bf x}_i\}_{i\in [n]}"/> in its columns and <img class="math" src="_images/math/eaea97645a8bb3b7b8f716dcdfa8e9798266c9d2.png" alt="y_i \in \{\pm 1\}"/> denote the corresponding labels.</p>
<p>The regularization strength is controlled by the regularization parameter <img class="math" src="_images/math/78bc838c22c8244e929d7d8967de830757d28252.png" alt="\lambda\geq 0"/>; the larger <img class="math" src="_images/math/76f1d8ace30435987c01a00ca53a71cba1f40e6c.png" alt="\lambda"/> the more robust the model becomes to overfitting. <img class="math" src="_images/math/76f1d8ace30435987c01a00ca53a71cba1f40e6c.png" alt="\lambda"/> can be specified by the user through the <code class="docutils literal notranslate"><span class="pre">regularizer</span></code> input parameter.</p>
<p><em>Snap ML</em> implements stochastic coordinate descent <a class="reference internal" href="#references">[SCD]</a> and stochastic dual coordinate ascent <a class="reference internal" href="#references">[SDCA]</a> as an algorithm to fit the model parameters <img class="math" src="_images/math/d0025313425983b19fea4533e1400d1ac30f9ae9.png" alt="{\bf w}"/>. In order to support GPU acceleration <em>Snap ML</em> implements a parallel asynchronous version of these solvers especially designed to leverage the massive parallelism of moderne GPUs <a class="reference internal" href="#references">[TPASCD]</a>.</p>
<p>The model can be trained using the <code class="docutils literal notranslate"><span class="pre">fit</span></code> method which takes the training data and the labels <img class="math" src="_images/math/40e5819f6e7c5bd244e0e0058bb5fbabec1c7a7d.png" alt="X,\bf y"/> as input and stores the coefficients of the learnt model <img class="math" src="_images/math/d0025313425983b19fea4533e1400d1ac30f9ae9.png" alt="{\bf w}"/> in its <code class="docutils literal notranslate"><span class="pre">coef_</span></code> attribute. This model can then be used to make predictions by calling the <code class="docutils literal notranslate"><span class="pre">predict</span></code> method on unlabelled data. The regularization type can be specified at initialization using the <code class="docutils literal notranslate"><span class="pre">penalty</span></code> argument.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pai4sk</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
<span class="gp">>>> </span><span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
<span class="gp">>>> </span><span class="n">lr</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="n">max_iter</span> <span class="o">=</span> <span class="mi">100</span><span class="p">,</span> <span class="n">regularizer</span> <span class="o">=</span> <span class="mf">0.01</span><span class="p">,</span> <span class="n">penalty</span> <span class="o">=</span> <span class="s1">'l2'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">X_train</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">]])</span>
<span class="gp">>>> </span><span class="n">y_train</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">lr</span><span class="o">.</span><span class="n">fit</span> <span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">lr</span><span class="o">.</span><span class="n">coef_</span>
<span class="go">[0.145, 0.145]</span>
<span class="gp">>>> </span><span class="n">X_test</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">],[</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">]])</span>
<span class="gp">>>> </span><span class="n">lr</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="go">[1,-1]</span>
<span class="gp">>>> </span><span class="n">lr</span><span class="o">.</span><span class="n">predict_proba</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="go">[[0.295, 0.705]</span>
<span class="go"> [0.536, 0.464]]</span>
</pre></div>
</div>
<p>A full example of training a <code class="xref py py-class docutils literal notranslate"><span class="pre">LogisticRegression</span></code> model in Snap ML can be found in the PowerAI distribution under <em>/opt/DL/snap-ml-local/examples</em>. For more details about the API we refer to the <a class="reference internal" href="pythonapidocumentation.html#python-api-documentation"><span class="std std-ref">snap-ml API</span></a>.</p>
</div>
<div class="section" id="decision-tree">
<span id="id4"></span><h2>Decision Tree<a class="headerlink" href="#decision-tree" title="Permalink to this headline">¶</a></h2>
<p>Snap ML offers a <code class="xref py py-class docutils literal notranslate"><span class="pre">DecisionTreeClassifier</span></code> for training a binary decision tree. The training routine is single-threaded and executed on the CPU. Check the <a class="reference internal" href="pythonapidocumentation.html#python-api-documentation"><span class="std std-ref">snap-ml API</span></a> for details about the available options or check out the <a class="reference internal" href="tutorials.html#tutorials"><span class="std std-ref">Tutorials</span></a> for an application example.</p>
</div>
<div class="section" id="random-forest">
<span id="id5"></span><h2>Random Forest<a class="headerlink" href="#random-forest" title="Permalink to this headline">¶</a></h2>
<p>With the <code class="xref py py-class docutils literal notranslate"><span class="pre">RandomForestClassifier</span></code> you can train an ensemble of trees. This is usually preferred over a single decision tree as it improves the generalization performance of the model. Snap ML implements a multi-threaded training routine that is executed on the CPU. Check the <a class="reference internal" href="pythonapidocumentation.html#python-api-documentation"><span class="std std-ref">snap-ml API</span></a> for details about the available options or check out the <a class="reference internal" href="tutorials.html#tutorials"><span class="std std-ref">Tutorials</span></a> for an application example.</p>
<div class="section" id="references">
<h3>References<a class="headerlink" href="#references" title="Permalink to this headline">¶</a></h3>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">[SCD]:</th><td class="field-body"><em>Y. Nesterov</em>. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 2012.</td>
</tr>
<tr class="field-even field"><th class="field-name">[SDCA]:</th><td class="field-body"><em>Shai Shalev-Shwartz and Tong Zhang</em>. Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization. Journal of Machine Learning Research, 2013.</td>
</tr>
<tr class="field-odd field"><th class="field-name">[TPASCD]:</th><td class="field-body"><em>Thomas Parnell, Celestine Dünner, Kubilay Atasu, Manolis Sifalakis and Haris Pozidis</em>. Tera-Scale Coordinate Descent on GPUs. Journal on Future Generation Computer Systems, 2018.</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
<a href="tutorials.html" class="btn btn-neutral float-right" title="Tutorials" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
<a href="index.html" class="btn btn-neutral float-left" title="Introduction" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<p>
© Copyright IBM Corporation 2018, 2019
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>