Skip to content

Commit 724817d

Browse files
kennethmhckennethmhc
authored
[FSTORE-1509] Feature logging (#276)
* fraud batch logging * fix online fraud * async feature logging * online logging * fix batch * log online model * remove redundant import * use hw model
1 parent 5eeb9e3 commit 724817d

File tree

5 files changed

+194
-76
lines changed

5 files changed

+194
-76
lines changed

‎fraud_batch/1_fraud_batch_feature_pipeline.ipynb‎

Lines changed: 39 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"cells": [
33
{
44
"cell_type": "markdown",
5-
"id": "f60c52ce",
5+
"id": "5d5afc66",
66
"metadata": {
77
"tags": []
88
},
@@ -27,7 +27,7 @@
2727
},
2828
{
2929
"cell_type": "markdown",
30-
"id": "7ec7724a",
30+
"id": "d20eee58",
3131
"metadata": {},
3232
"source": [
3333
"## <span style='color:#ff5f27'> 📝 Imports"
@@ -36,7 +36,7 @@
3636
{
3737
"cell_type": "code",
3838
"execution_count": null,
39-
"id": "42efa933",
39+
"id": "802d6425",
4040
"metadata": {},
4141
"outputs": [],
4242
"source": [
@@ -46,7 +46,7 @@
4646
{
4747
"cell_type": "code",
4848
"execution_count": null,
49-
"id": "a4120a95",
49+
"id": "bd55a691",
5050
"metadata": {},
5151
"outputs": [],
5252
"source": [
@@ -64,7 +64,7 @@
6464
},
6565
{
6666
"cell_type": "markdown",
67-
"id": "010286f1",
67+
"id": "879a539d",
6868
"metadata": {},
6969
"source": [
7070
"## <span style=\"color:#ff5f27;\"> 💽 Loading the Data </span>\n",
@@ -84,7 +84,7 @@
8484
{
8585
"cell_type": "code",
8686
"execution_count": null,
87-
"id": "491a37b4",
87+
"id": "586d4111",
8888
"metadata": {},
8989
"outputs": [],
9090
"source": [
@@ -100,7 +100,7 @@
100100
{
101101
"cell_type": "code",
102102
"execution_count": null,
103-
"id": "9ed8c4f0",
103+
"id": "0cdb9c3d",
104104
"metadata": {},
105105
"outputs": [],
106106
"source": [
@@ -118,7 +118,7 @@
118118
{
119119
"cell_type": "code",
120120
"execution_count": null,
121-
"id": "2543c511",
121+
"id": "5a6fb4f4",
122122
"metadata": {},
123123
"outputs": [],
124124
"source": [
@@ -135,15 +135,15 @@
135135
},
136136
{
137137
"cell_type": "markdown",
138-
"id": "717f9102",
138+
"id": "fbb301ce",
139139
"metadata": {},
140140
"source": [
141141
"---"
142142
]
143143
},
144144
{
145145
"cell_type": "markdown",
146-
"id": "16357302",
146+
"id": "bf6f9299",
147147
"metadata": {},
148148
"source": [
149149
"## <span style=\"color:#ff5f27;\"> 🛠️ Feature Engineering </span>\n",
@@ -158,7 +158,7 @@
158158
{
159159
"cell_type": "code",
160160
"execution_count": null,
161-
"id": "0b919bf8",
161+
"id": "d1af85e4",
162162
"metadata": {},
163163
"outputs": [],
164164
"source": [
@@ -181,7 +181,7 @@
181181
{
182182
"cell_type": "code",
183183
"execution_count": null,
184-
"id": "5d6fd9ae",
184+
"id": "4956cb52",
185185
"metadata": {},
186186
"outputs": [],
187187
"source": [
@@ -191,7 +191,7 @@
191191
},
192192
{
193193
"cell_type": "markdown",
194-
"id": "92820fab",
194+
"id": "bc4e505f",
195195
"metadata": {},
196196
"source": [
197197
"Next, you will create features that for each credit card aggregate data from multiple time steps.\n",
@@ -203,7 +203,7 @@
203203
{
204204
"cell_type": "code",
205205
"execution_count": null,
206-
"id": "47116f13",
206+
"id": "ee689954",
207207
"metadata": {},
208208
"outputs": [],
209209
"source": [
@@ -223,7 +223,7 @@
223223
},
224224
{
225225
"cell_type": "markdown",
226-
"id": "0b3abcfc",
226+
"id": "ed7039bb",
227227
"metadata": {},
228228
"source": [
229229
"Next lets compute windowed aggregates. Here you will use 4-hour windows, but feel free to experiment with different window lengths by setting `window_len` below to a value of your choice."
@@ -232,7 +232,7 @@
232232
{
233233
"cell_type": "code",
234234
"execution_count": null,
235-
"id": "f9bf9dd7",
235+
"id": "791708c4",
236236
"metadata": {},
237237
"outputs": [],
238238
"source": [
@@ -248,7 +248,7 @@
248248
},
249249
{
250250
"cell_type": "markdown",
251-
"id": "b3021fa4",
251+
"id": "391e478f",
252252
"metadata": {},
253253
"source": [
254254
"### <span style=\"color:#ff5f27;\">⚙️ Convert date time object to unix epoch in milliseconds </span>"
@@ -257,7 +257,7 @@
257257
{
258258
"cell_type": "code",
259259
"execution_count": null,
260-
"id": "837bafd5",
260+
"id": "bc6bcae3",
261261
"metadata": {},
262262
"outputs": [],
263263
"source": [
@@ -270,7 +270,7 @@
270270
},
271271
{
272272
"cell_type": "markdown",
273-
"id": "a3df26a9",
273+
"id": "bc3fec23",
274274
"metadata": {},
275275
"source": [
276276
"## <span style=\"color:#ff5f27;\">👮🏻‍♂️ Great Expectations </span>"
@@ -279,7 +279,7 @@
279279
{
280280
"cell_type": "code",
281281
"execution_count": null,
282-
"id": "c9391579",
282+
"id": "cbc3ca3c",
283283
"metadata": {},
284284
"outputs": [],
285285
"source": [
@@ -299,7 +299,7 @@
299299
{
300300
"cell_type": "code",
301301
"execution_count": null,
302-
"id": "8969ff27",
302+
"id": "b3891fe5",
303303
"metadata": {},
304304
"outputs": [],
305305
"source": [
@@ -340,23 +340,23 @@
340340
},
341341
{
342342
"cell_type": "markdown",
343-
"id": "adf03efd",
343+
"id": "da2f174f",
344344
"metadata": {},
345345
"source": [
346346
"---"
347347
]
348348
},
349349
{
350350
"cell_type": "markdown",
351-
"id": "7c069f5a",
351+
"id": "c48bbff9",
352352
"metadata": {},
353353
"source": [
354354
"## <span style=\"color:#ff5f27;\"> 📡 Connecting to Hopsworks Feature Store </span>"
355355
]
356356
},
357357
{
358358
"cell_type": "markdown",
359-
"id": "ac32437d",
359+
"id": "4d68f207",
360360
"metadata": {
361361
"tags": []
362362
},
@@ -373,7 +373,7 @@
373373
{
374374
"cell_type": "code",
375375
"execution_count": null,
376-
"id": "287491fa",
376+
"id": "4c259c35",
377377
"metadata": {},
378378
"outputs": [],
379379
"source": [
@@ -386,7 +386,7 @@
386386
},
387387
{
388388
"cell_type": "markdown",
389-
"id": "23b61089",
389+
"id": "0f80ccc5",
390390
"metadata": {},
391391
"source": [
392392
"To create a feature group you need to give it a name and specify a primary key. It is also good to provide a description of the contents of the feature group and a version number, if it is not defined it will automatically be incremented to `1`."
@@ -395,7 +395,7 @@
395395
{
396396
"cell_type": "code",
397397
"execution_count": null,
398-
"id": "58353322",
398+
"id": "c78c614a",
399399
"metadata": {},
400400
"outputs": [],
401401
"source": [
@@ -412,7 +412,7 @@
412412
},
413413
{
414414
"cell_type": "markdown",
415-
"id": "832333fa",
415+
"id": "45fcf5c3",
416416
"metadata": {},
417417
"source": [
418418
"A full list of arguments can be found in the [documentation](https://docs.hopsworks.ai/feature-store-api/latest/generated/api/feature_store_api/#create_feature_group).\n",
@@ -423,7 +423,7 @@
423423
{
424424
"cell_type": "code",
425425
"execution_count": null,
426-
"id": "7f82e4a2",
426+
"id": "e97b5aab",
427427
"metadata": {},
428428
"outputs": [],
429429
"source": [
@@ -435,7 +435,7 @@
435435
{
436436
"cell_type": "code",
437437
"execution_count": null,
438-
"id": "0c6832a9",
438+
"id": "2f6cc0c0",
439439
"metadata": {},
440440
"outputs": [],
441441
"source": [
@@ -462,7 +462,7 @@
462462
},
463463
{
464464
"cell_type": "markdown",
465-
"id": "36f7e30a",
465+
"id": "a57846b4",
466466
"metadata": {},
467467
"source": [
468468
"At the creation of the feature group, you will be prompted with an URL that will directly link to it; there you will be able to explore some of the aspects of your newly created feature group.\n",
@@ -472,7 +472,7 @@
472472
},
473473
{
474474
"cell_type": "markdown",
475-
"id": "1250c3ce",
475+
"id": "777e6d4e",
476476
"metadata": {},
477477
"source": [
478478
"You can move on and do the same thing for the feature group with our windows aggregation."
@@ -481,7 +481,7 @@
481481
{
482482
"cell_type": "code",
483483
"execution_count": null,
484-
"id": "6b8ef9f6",
484+
"id": "35ca5bbb",
485485
"metadata": {},
486486
"outputs": [],
487487
"source": [
@@ -498,7 +498,7 @@
498498
{
499499
"cell_type": "code",
500500
"execution_count": null,
501-
"id": "1970657b",
501+
"id": "e16aa93e",
502502
"metadata": {},
503503
"outputs": [],
504504
"source": [
@@ -510,7 +510,7 @@
510510
{
511511
"cell_type": "code",
512512
"execution_count": null,
513-
"id": "68b0c84f",
513+
"id": "2d80f5e2",
514514
"metadata": {},
515515
"outputs": [],
516516
"source": [
@@ -530,7 +530,7 @@
530530
},
531531
{
532532
"cell_type": "markdown",
533-
"id": "e22dce87",
533+
"id": "eb2254b9",
534534
"metadata": {},
535535
"source": [
536536
"Both feature groups are now accessible and searchable in the UI\n",
@@ -540,7 +540,7 @@
540540
},
541541
{
542542
"cell_type": "markdown",
543-
"id": "4f8b5d8a",
543+
"id": "c6e783a2",
544544
"metadata": {},
545545
"source": [
546546
"## <span style=\"color:#ff5f27;\">⏭️ **Next:** Part 02: Training Pipeline\n",
@@ -555,7 +555,7 @@
555555
"hash": "e1ddeae6eefc765c17da80d38ea59b893ab18c0c0904077a035ef84cfe367f83"
556556
},
557557
"kernelspec": {
558-
"display_name": "Python 3",
558+
"display_name": "Python 3 (ipykernel)",
559559
"language": "python",
560560
"name": "python3"
561561
},
@@ -569,7 +569,7 @@
569569
"name": "python",
570570
"nbconvert_exporter": "python",
571571
"pygments_lexer": "ipython3",
572-
"version": "3.10.11"
572+
"version": "3.10.13"
573573
}
574574
},
575575
"nbformat": 4,

‎fraud_batch/2_fraud_batch_training_pipeline.ipynb‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@
186186
" version=1,\n",
187187
" query=selected_features,\n",
188188
" labels=[\"fraud_label\"],\n",
189-
" transformation_functions=transformation_functions,\n",
189+
" transformation_functions=[label_encoder(\"category\")],\n",
190190
")"
191191
]
192192
},
@@ -482,7 +482,7 @@
482482
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
483483
},
484484
"kernelspec": {
485-
"display_name": "Python 3",
485+
"display_name": "Python 3 (ipykernel)",
486486
"language": "python",
487487
"name": "python3"
488488
},
@@ -496,7 +496,7 @@
496496
"name": "python",
497497
"nbconvert_exporter": "python",
498498
"pygments_lexer": "ipython3",
499-
"version": "3.10.11"
499+
"version": "3.10.13"
500500
}
501501
},
502502
"nbformat": 4,

0 commit comments

Comments
 (0)