You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: md-docs/user_guide/modules/monitoring.md
+36-32Lines changed: 36 additions & 32 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,31 +2,35 @@
2
2
3
3
Data and model monitoring is fundamental to guarantee performance of your ML models.
4
4
With ML cube Platform you can log and monitor different aspects of your ML Task by uploading data batches.
5
+
Before entering in details about what ML cube Platform monitors and analyses, it is worth to mention how data are represented.
6
+
Data are shared as _batches_
5
7
6
8
## Data Taxonomy
7
-
A Batch of data is composed of four types of data:
8
9
9
-
-**input:** set of input features the AI model uses to predict the output.
10
-
ML cube Platform uses the input data that comes at the end of the processing data pipeline and not the raw data.
11
-
This is due to the fact that ML cube Platform detects drifts in what the AI model uses and not in the general data the customer has.
10
+
A Batch of data is composed of four types categories:
11
+
12
+
-**input:** set of input features the AI model uses to predict the output.
13
+
ML cube Platform uses the input data that comes at the end of the processing data pipeline and not the raw data.
14
+
This is due to the fact that ML cube Platform detects drifts in what the AI model uses and not in the general data the customer has.
12
15
-**target:** target quantity predicted by the AI models.
13
-
It is present in the training data but can be not available for production data.
16
+
It is present in the training data but can be not available for production data.
14
17
-**models' predictions:** predicted target for each AI model in the AI Task.
15
18
-**metadata:** additional information that AI models do not use as input but that is important to define the data or the samples.
16
-
Mandatory for this category are the `sample-id`, a unique identifier for each sample used to avoid confusion and misinterpretation; and the
17
-
`sample-timestamp`, a timestamp associated with each sample used for ordering.
18
-
Moreover, the User can provide additional data used to segment the data space.
19
-
For instance, sensitive information like zip code or country are not used by AI models to prevent bias, however, ML cube Platform can use them to
20
-
check and prevent bias in the suggested retraining dataset or to perform segmented drift detection.
19
+
Mandatory for this category are the `sample-id`, a unique identifier for each sample used to avoid confusion and misinterpretation; and the
20
+
`sample-timestamp`, a timestamp associated with each sample used for ordering.
21
+
Moreover, the User can provide additional data used to segment the data space.
22
+
For instance, sensitive information like zip code or country are not used by AI models to prevent bias, however, ML cube Platform can use them to
23
+
check and prevent bias in the suggested retraining dataset or to perform segmented drift detection.
21
24
22
25
## Data Categories
26
+
23
27
ML cube Platform are present three categories of data:
24
28
25
29
-**Reference:** represents the dataset used to train the model.
26
-
Each model version has a reference dataset.
27
-
Detection algorithms use reference data during their initialization.
30
+
Each model version has a reference dataset.
31
+
Detection algorithms use reference data during their initialization.
28
32
-**Production:** represents data that comes from the production environment in which the AI model is operating.
29
-
Detection algorithms analyze production data to detect the presence of drifts.
33
+
Detection algorithms analyze production data to detect the presence of drifts.
30
34
-**Historical:** represents additional data that ML cube Platform can use to define the retraining dataset after a drift.
31
35
32
36
Each data category is uploaded to the application with its specific API call, however, they share the same structure.
@@ -44,35 +48,35 @@ That's why during production data arrive at different times, usually input and p
44
48
45
49
Delta Energy company trained its models using the data in the year 2022 and used the algorithms starting from the 2023. This means that the data in the 2022 are the reference data and every data from the january first 2023 are considered as production data. Data previous 2022 are historical data instead.
46
50
47
-
48
51
## Drift Detection
52
+
49
53
ML cube Platform provides a set of Detectors for each AI Task.
50
54
These detectors are used to monitor the task at different levels.
51
-
The choice over the types of detectors to be instantiated depends on the type of task (classification or regression) and on the type of data available for that task (input, output, model predictions).
55
+
The choice over the types of detectors to be instantiated depends on the type of task (classification or regression) and on the type of data available for that task (input, output, model predictions).
52
56
53
-
There are mainly two classes of detectors:
57
+
There are mainly two classes of detectors:
54
58
55
-
-**Data Detectors:** they take into account data associated with the task. They may be *input only*
56
-
data or *input and ground truth* data. These detectors are independent from the models trained on the
57
-
task as they do not either consider their predictions or performances. These detectors are responsible for the identification of
58
-
input and concept drifts. According to the type of the used detector, changes in data are either monitored at feature
59
-
level or using a multivariate monitoring strategy.
59
+
-**Data Detectors:** they take into account data associated with the task. They may be _input only_
60
+
data or _input and ground truth_ data. These detectors are independent from the models trained on the
61
+
task as they do not either consider their predictions or performances. These detectors are responsible for the identification of
62
+
input and concept drifts. According to the type of the used detector, changes in data are either monitored at feature
63
+
level or using a multivariate monitoring strategy.
60
64
-**Model Detectors:** they monitor the performances associated with the models related to the task.
61
-
In cases where the user has multiple models trained for a single task, a single detector is created for each model.
65
+
In cases where the user has multiple models trained for a single task, a single detector is created for each model.
62
66
63
-
Each detector is initially created using **Reference data** provided by the user. Every time a new batch of data
67
+
Each detector is initially created using **Reference data** provided by the user. Every time a new batch of data
64
68
is uploaded, the detectors observe the batch and update their statistics.
65
-
Each detector updates its statistics independently from the others and each of them presents a double-level alarm scheme in
69
+
Each detector updates its statistics independently from the others and each of them presents a double-level alarm scheme in
66
70
order to either signal a **Warning** or a **Drift** for the monitored task.
67
71
68
-
The detectors may be in three different states:
72
+
The detectors may be in three different states:
69
73
70
-
-**Regular:** the detector is monitoring data that are similar to the reference data,
71
-
-**Warning:** the detector has fired a Warning alarm since the data has started to change. From this zone, it is possible
72
-
to either go into the Drift status or to go back to the Regular one, depending on the monitored data.
73
-
-**Drift:** the detector has fired a Drift alarm and a change has been established between the reference data and the last
74
-
ones. After a drift, the detector is usually reset by defining a new set of reference data. The reset process is different
75
-
according to what has been monitored by the detector.
74
+
-**Regular:** the detector is monitoring data that are similar to the reference data,
75
+
-**Warning:** the detector has fired a Warning alarm since the data has started to change. From this zone, it is possible
76
+
to either go into the Drift status or to go back to the Regular one, depending on the monitored data.
77
+
-**Drift:** the detector has fired a Drift alarm and a change has been established between the reference data and the last
78
+
ones. After a drift, the detector is usually reset by defining a new set of reference data. The reset process is different
79
+
according to what has been monitored by the detector.
76
80
77
81
All the alarms generated during this process are shown in the application like **Detection Events** available in the Task homepage or in the Detection page.
78
-
You can create automation rules based on those events to be notified on specific channels or start retraining, see [Detection automation rules](../detection_event_rules.md) for more details.
82
+
You can create automation rules based on those events to be notified on specific channels or start retraining, see [Detection automation rules](../detection_event_rules.md) for more details.
0 commit comments