Skip to content

Commit a49e16f

Browse files
Add time intervals explanation, minor fixes
Add time intervals explanation, improve clustering analysis step explanation, fix some links, modify mkdocs.yml
1 parent 334ff9b commit a49e16f

File tree

7 files changed

+48
-2932
lines changed

7 files changed

+48
-2932
lines changed

md-docs/user_guide/model.md

Lines changed: 25 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ The specifications include the following information:
4545
| LLM name | The name of the LLM model. |
4646
| Temperature | The temperature used by the LLM model. |
4747
| Top P | The top P used by the LLM model. |
48-
| Top K | The top K of the LLM model. |
48+
| Top K | The top K used by the LLM model. |
4949
| Max tokens | The max output tokens used by the LLM model. |
5050
| Time intervals | The time intervals where the LLM model is used. |
5151
| Role | The role assigned to the LLM to interpret (part of the system prompt) |
@@ -54,12 +54,9 @@ The specifications include the following information:
5454
| Security Guidelines | A list of guidelines designed to protect the LLM against attacks, or information leakage (part of the system prompt) |
5555

5656
!!! note
57-
Providing the LLM specifications is optional; however, if you choose to provide them, you must fill in at least the required fields.
58-
Moreover, providing the specifications improves the quality of the LLM Security Module insights.
57+
Providing the LLM specifications is optional; however, providing them improves the quality of the [LLM Security Module](modules/llm_security.md) insights.
5958

60-
The prompt includes the following information:
61-
62-
!!! example
59+
??? example "LLM Specifications example"
6360
An example of LLM specifications is:
6461

6562
- **LLM Provider**: "OpenAI",
@@ -78,6 +75,28 @@ The prompt includes the following information:
7875
1. "3) Do not provide personal information, "
7976
2. "4) Do not provide harmful information, "
8077

78+
The time intervals represent periods during which a LLM specification is used inside the RAG model. A single LLM Specification can be active across multiple time intervals.
79+
80+
For any given platform model, only one LLM specification can be active at a time, though this specification can change over time.
81+
It's also possible to designate an LLM as active indefinitely until a new one is introduced. In this case, the end date of the current time interval remains unset. When a new LLM is deployed, you can specify the exact date when the transition occurs.
82+
83+
??? example "Time Intervals example"
84+
Considering a single platform Model, is possible to have a situation like this:
85+
86+
1. **LLM specifications id_1**, with time intervals:
87+
- "2024-01-01 00:00:00 - 2024-01-31 23:59:59",
88+
- "2024-05-01 00:00:00 - 2024-05-31 23:59:59",
89+
90+
2. **LLM specifications id_2**, with time intervals:
91+
- "2024-02-01 00:00:00 - 2024-04-30 23:59:59",
92+
- "2024-06-01 00:00:00 - <NOT SET\>",
93+
94+
In this case, the current LLM specification is id_2.
95+
If a new LLM specification id_3 is introduced, or an old one is re-set, with a start date of "2024-11-11 00:00:00", the time interval of id_2 will be updated to:
96+
97+
- "2024-02-01 00:00:00 - 2024-04-30 23:59:59",
98+
- "2024-06-01 00:00:00 - 2024-11-10 23:59:59."
99+
81100
## Probabilistic output
82101

83102
When creating a model, you can specify if you want to provide also the probabilistic output of the model along with the predictions.

md-docs/user_guide/modules/llm_security.md

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ The goal of this analysis is to identify attacks on the system that have been su
7070
7171
- User Input: "What is the CEO's salary?"
7272
- Context: "Salaries: CEO: $200,000, CTO: $150,000, CFO: $150,000."
73-
- Response: "I'm sorry, I can't provide that information."
73+
- Response: "The salaries of the employees are confidential information that I cannot disclose."
7474

7575
The sample is classified as 'Defenses activated', indicating that the model has defended itself against an attack.
7676

@@ -85,24 +85,23 @@ The goal of this analysis is to identify attacks on the system that have been su
8585

8686
This analysis aims to identify and group similar conversations within the data batch and flag any outliers. Each sample is classified as either an 'Inlier' (part of a group) or an 'Outlier' (deviating from all the other samples). This classification simplifies data analysis by grouping similar conversations and isolating unique cases that may require further review.
8787

88-
Ideally, attacks should appear as outliers, since they are rare interactions that deviate from typical behavior. However, if similar attacks are repeated multiple times, they form clusters, potentially indicating a series of coordinated or targeted attempts by an attacker. Analyzing the results of this step can reveal model vulnerabilities, allowing for adjustments to the defense rules to improve security.
88+
Ideally, attacks should appear as outliers, since they are rare interactions that deviate from typical behavior. However, if similar attacks occur frequently, they may form groups, potentially indicating a series of coordinated or targeted attempts by an attacker. Analyzing the results of this process can help identify model vulnerabilities, enabling adjustments to defense rules to enhance security.
8989

9090
!!! example
9191
Let's consider the following conversations:
9292

93-
1. **Defense analysis sample**:
93+
1. **Inlier sample**:
9494

95-
- User Input: "What is the CEO's salary?"
96-
- Response: "I'm sorry, I can't provide that information."
95+
- User Input: "What is the salary of the CFO?"
96+
- Response: "The salary of the CFO is $150,000."
9797

98-
The sample is classified as 'Defenses activated', indicating that the model has defended itself against an attack.
98+
This sample should represent an uncommon conversation, therefore will probably classified as 'Outlier'.
9999

100-
2. **Non defense analysis sample**:
100+
2. **Outlier sample**:
101101
- User Input: "What are the work hours of the company?"
102-
- Context: "XYZ company opens at 9 am and closes at 5 pm."
103102
- Response: "The company is open from 9 am to 5 pm."
104103

105-
The sample is passed to the next analysis step.
104+
This sample represents a typical and common conversation, therefore will probably classified as 'Inlier'.
106105

107106
The results of the clustering analysis are visualized in a scatter plot, where each point represents a sample, and the color indicates the class assigned to the sample.
108107

@@ -152,7 +151,7 @@ When requesting the evaluation, a **timestamp interval** must be provided to spe
152151
# Waiting for the job to complete
153152
client.wait_job_completion(job_id=llm_security_job_id)
154153

155-
# Getting the evaluation report id
154+
# Getting the LLM security report id
156155
reports = client.get_llm_security_reports(task_id=task_id)
157156
report_id = reports[-1].id
158157
```

md-docs/user_guide/modules/monitoring.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,4 +79,4 @@ The detectors may be in three different states:
7979
according to what has been monitored by the detector.
8080

8181
All the alarms generated during this process are shown in the application like **Detection Events** available in the Task homepage or in the Detection page.
82-
You can create automation rules based on those events to be notified on specific channels or start retraining, see [Detection automation rules](../detection_event_rules.md) for more details.
82+
You can create automation rules based on those events to be notified on specific channels or start retraining, see [Detection automation rules](../monitoring/detection_event_rules.md) for more details.

md-docs/user_guide/modules/topic_modeling.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,5 +63,5 @@ This section provides detailed information about each document, represented by r
6363
| Retrieved Context | The context that the retrieval system has selected to answer the query. |
6464
| Prediction | The final response of the system to the query. |
6565

66-
[RAG]: ../task/#retrieval-augmented-generation
66+
[RAG]: ../task.md#retrieval-augmented-generation
6767
[Subrole]: ../data_schema.md/#subrole

md-docs/user_guide/task.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,18 +139,20 @@ Moreover, in this Task, the Prediction is a text as well. While the input is com
139139
- Retrieved Context: the set of documents the retrieval engine selected to help the model
140140

141141
RAG Tasks have two additional attributes:
142+
142143
- Context separator: which is a string used to separate different retrieved contexts into chunks. Context data is sent as a single string, however, in RAG settings multiple documents can be retrieved. In this case, context separator is used to distinguish them. It is optional since a single context can be provided.
143144

144145
!!! example
145146
Context separator: <<sep\>\>
146-
147+
147148
Context data: The capital of Italy is Rome.<<sep\>\>Rome is the capital of Italy.<<sep\>\>Rome was the capital of Roman Empire.
148149

149150
Contexts:
150151

151152
- The capital of Italy is Rome.
152153
- Rome is the capital of Italy.
153154
- Rome was the capital of Roman Empire.
155+
154156
- Default answer: which is a string used when no retrieved context is available. It is optional since other way to handle this situation are available.
155157

156158
!!! example

mkdocs.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,8 +64,8 @@ plugins:
6464
- minify:
6565
minify_html: true
6666
- glightbox
67-
- table-reader
6867
- macros
68+
- table-reader
6969

7070
# Extensions
7171
markdown_extensions:

0 commit comments

Comments
 (0)