You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Pull Request Template
<!-- Please include a summary of the changes and the related issue.
Please also include relevant motivation and context. List any
dependencies that are required for this change. -->
Closes #<issue_number>
**Type of change**
<!-- Please delete options that are not relevant. Remember to title the
PR according to the type of change -->
- Documentation update
**How Has This Been Tested**
<!-- Please add some reference about how your feature has been tested.
-->
**Checklist**
<!-- Please go over the list and make sure you've taken everything into
account -->
- I added relevant documentation
- follows the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)
---------
Co-authored-by: Daniel Vila Suero <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Copy file name to clipboardExpand all lines: argilla/docs/how_to_guides/annotate.md
+47-58Lines changed: 47 additions & 58 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,48 +3,47 @@ description: In this section, we will provide a step-by-step guide to show how t
3
3
---
4
4
# Annotate your dataset
5
5
6
-
!!! note
7
-
To experience the UI features firsthand, you can take a look to the [Demo ↗](https://demo.argilla.io/sign-in?auth=ZGVtbzoxMjM0NTY3OA==).
6
+
!!! note ""
7
+
To experience the UI features firsthand, you can take a look at the [Demo ↗](https://demo.argilla.io/sign-in?auth=ZGVtbzoxMjM0NTY3OA==).
8
8
9
9
Argilla UI offers many functions to help you manage your annotation workflow, aiming to provide the most flexible approach to fit the wide variety of use cases handled by the community.
At the right side of the navigation breadcrumb, you can customize the dataset settings and edit your profile.
17
+
The UI is responsive with two columns for larger devices and one column for smaller devices. This enables you to annotate data using your mobile phone for simple datasets (i.e., not very long text and 1-2 questions) or resize your screen to get a more compact UI.
18
18
19
-
=== "The left pane"
19
+
=== "Header"
20
20
21
-
This area is displayed on top of **the control panel** for performing searches, applying filters and sorting results. The record card(s) are displayed one by one or in a vertical list depending on the active view: **Focus view** or **Bulk view.** A card includes one or many fields and an ellipsis menu to mainly access the record extra info like the metadata.
21
+
At the right side of the navigation breadcrumb, you can customize the dataset settings and edit your profile.
22
22
23
-
=== "The right pane"
23
+
=== "Left pane"
24
+
This area displays the **control panel** on the top. The control panel is used for performing keyword-based search, applying filters, and sorting the results.
24
25
25
-
This is where you annotate your dataset. Simply fill it out as a form, then choose to Submit, Save as Draft, or Discard your response to send the records to their corresponding queues.
26
+
Below the control panel, the record card(s) are displayed one by one (Focus view) or in a vertical list (Bulk view).
26
27
27
-
=== "The left bottom panel"
28
+
=== "Right pane"
28
29
29
-
This expandable area displays the annotation guidelines.
30
+
This is where you annotate your dataset. Simply fill it out as a form, then choose to `Submit`, `Save as Draft`, or `Discard`.
30
31
31
-
=== "The right bottom panel"
32
+
=== "Left bottom panel"
32
33
33
-
This expandable area displays your annotation progress.
34
+
This expandable area displays the annotation guidelines. The annotation guidelines can be edited by owner and admin roles in the dataset settings.
The app is responsive which enable you to adapt your workspace from two to one column. You can even use your mobile to provides simple feedback on your datasets.
38
+
This expandable area displays your annotation progress.
39
39
40
40
### Shortcuts
41
41
42
-
Argilla UI includes a range of shortcuts. For the main actions submit, discard, save as draft and the labels the keys are showed in the button.
42
+
The Argilla UI includes a range of shortcuts. For the main actions (submit, discard, save as draft and selecting labels) the keys are showed in the corresponding button.
43
43
44
-
To move from one question to another or between records using the keyboard take a look at the table below.
44
+
To learn how to move from one question to another or between records using the keyboard, take a look at the table below.
45
45
46
-
!!! tip
47
-
Shortcuts provide a smoother experience, especially with a long list of labels or single-question forms.
46
+
Shortcuts provide a smoother annotation experience, especially with datasets using a single question (Label, MultiLabel, Rating, or Ranking).
48
47
49
48
??? "Available shortcuts"
50
49
@@ -77,23 +76,20 @@ If you are starting an annotation effort, all the records are initially kept in
77
76
78
77
### Suggestions
79
78
80
-
If your dataset includes model predictions, you will see them represented by a sparkle icon `✨` in the label or value button. We call them “Suggestions” and they appear in the form as pre-filled responses. If you agree with the suggestion, you just need to click on the Submit button, and they will be considered as your response.
79
+
If your dataset includes model predictions, you will see them represented by a sparkle icon `✨` in the label or value button. We call them “Suggestions” and they appear in the form as pre-filled responses. If confidence scores have been included by the dataset admin, they will be shown alongside with the label. Additionally, admins can choose to always show suggested labels at the beginning of the list. This can be configured from the dataset settings.
81
80
82
-
If the suggestion is incorrect, you can modify it and submit your response.
83
-
84
-
The score per suggested label/value is displayed for MultiLabelQuestion and RankingQuestion.
85
-
For label questions, admins can also choose the ordering of labels in the datasets settings. Additionally, admins can choose to always show suggested labels at the beginning of the list.
81
+
If you agree with the suggestions, you just need to click on the `Submit` button, and they will be considered as your response. If the suggestion is incorrect, you can modify it and submit your final response.
This is the default view to annotate your dataset linearly, displaying one record after another.
92
88
93
89
!!! tip
94
-
You should use this when getting acquainted with a dataset or when the annotation team is very diverse, the topic is generic.
90
+
You should use this view if you have a large number of required questions or need a strong focus on the record content to be labelled. This is also the recommended view for annotating a dataset sample to avoid potential biases introduced by using filters, search, sorting and bulk labelling.
95
91
96
-
Once you submit your first response, the next record will appear automatically. To see again your submitted record, just click on prev.
92
+
Once you submit your first response, the next record will appear automatically. To see again your submitted response, just click on `Prev`.
97
93
98
94
**Navigating through the records**
99
95
@@ -103,50 +99,45 @@ Each time the page is fully refreshed, the records with modified statuses (Pendi
The bulk view is designed to speed up the annotation and get a quick overview of the whole dataset.
109
105
110
-
It displays the records in a vertical list. Once the view is active, some functions from the **control panel** will be available to optimize the records reading. You define the number of records to display by page between `10`,`25`,`50` , `100` and the option to fix the card height by selecting `Expand records` or`Collapse records`.
106
+
The bulk view displays the records in a vertical list. Once this view is active, some functions from the control panel will activate to optimize the view. You can define the number of records to display by page between `10`,`25`,`50`, `100` and whether records are shown with a fixed (`Collapse records`) or their natural height (`Expand records`).
111
107
112
108
!!! tip
113
-
You should use this when you have a good understanding of your data and want to apply your knowledge based on things like similarity search, filter patterns, and suggestion score thresholds.
109
+
You should use this to quickly explore a dataset. This view is also recommended if you have a good understanding of the domain and want to apply your knowledge based on things like similarity and keyword search, filters, and suggestion score thresholds. For a datasets with a large number of required questions or very long fields, the focus view would be more suitable.
114
110
115
-
!!! note
116
-
You should consider that bulk view does not show suggestions in bulk view (except for Spans) and will always convert questions to Draft queue when working with multiple question types.
111
+
With multiple questions, think about using the bulk view to annotate massively one question. Then, you can complete the annotation per record from the draft queue.
117
112
118
-
!!! tip
119
-
With multiple questions, think about using the bulk view to annotate massively one question. Then, you can complete the annotation per records from the draft queue.
113
+
!!! note
114
+
Please note that suggestions are not shown in bulk view (except for Spans) and that you will need to save as a draft when you are not providing responses to all required questions.
120
115
121
116
### Annotation progress
122
117
123
-
=== "General progress view"
124
-
125
-
On the dataset list, the global progress of the annotation task from all users is displayed. This is indicated in the `Global progress` column, which shows the number of left records to be annotated, along with a progress bar. The progress bar displays the percentage and number of records submitted, conflicting (i.e., those with overlap), discarded, and pending by hovering your mouse over it.
118
+
The global progress of the annotation task from all users is displayed in the dataset list. This is indicated in the `Global progress` column, which shows the number of records still to be annotated, along with a progress bar. The progress bar displays the percentage and number of records submitted, conflicting (i.e., those with both submitted and discarded responses), discarded and pending by hovering your mouse over it.
126
119
127
-
=== "Your own progress view"
120
+
You can track your annotation progress in real time from the righ-bottom panel inside the dataset page. This means that, while you are annotating, the progress bar updates as you submit or discard a record. Expanding the panel, the distribution of `Pending`, `Draft`, `Submitted` and `Discarded` responses is displayed in a donut chart.
128
121
129
-
You can track your annotation progress in real-time. That means once you are annotating, the progress bar is incrementing in real time each time you submit or discard a record. Expanding the panel, the distribution of `Pending`, `Draft`, `Submitted` and `Discarded` responses is displayed in a donut chart.
122
+
## Use search, filters, and sort
130
123
131
-
## Discover patterns and speed up your annotation.
132
-
133
-
The UI offers various features designed to enhance your understanding of data patterns and streamline annotation tasks. Combining filters with bulk annotations can save you and your team hours of time.
124
+
The UI offers various features designed for data exploration and understanding. Combining these features with bulk labelling can save you and your team hours of time.
134
125
135
126
!!! tip
136
127
You should use this when you are familiar with your data and have large volumes to annotate based on verified beliefs and experience.
137
128
138
-
### Search and highlight
129
+
### Search
139
130
140
-
From the **control panel** at the top of the left pane, you can search across the entire dataset or by fields (if you have more than one in your record) and visualize matched results highlighted in color.
131
+
From the **control panel** at the top of the left pane, you can search by keyword across the entire dataset. If you have more than one field in your records, you may specify if the search is to be performed “All” fields or on a specific one. Matched results are highlighted in color.
141
132
142
133
### Order by record semantic similarity
143
134
144
135
You can retrieve records based on their similarity to another record if vectors have been added to the dataset.
145
136
146
137
!!! note
147
-
Consult these guides to know how to add vectors to your [dataset](dataset.md) and [records](record.md).
138
+
Check these guides to know how to add vectors to your [dataset](dataset.md) and [records](record.md).
148
139
149
-
To initiate a semantic search, click on `Find similar` within the record you wish to use as a reference. If multiple vectors are available, select the desired vector or choose whether to retrieve the most or least similar records.
140
+
To use the search by semantic similarity function, click on `Find similar` within the record you wish to use as a reference. If multiple vectors are available, select the desired vector. You can also choose whether to retrieve the most or least similar records.
150
141
151
142
The retrieved records are then ordered by similarity, with the similarity score displayed on each record card.
152
143
@@ -156,29 +147,27 @@ To cancel the search, click on the cross icon next to the reference record.
156
147
157
148
### Filter and sort by metadata, responses, and suggestions
158
149
159
-
<h4>Filter</h4>
150
+
#### Filter
160
151
161
152
If the dataset contains metadata, responses and suggestions, click on **Filter** in the **control panel** to display the available filters. You can select multiple filters and combine them.
162
153
163
154
!!! note
164
155
Record info including metadata is visible from the ellipsis menu in the record card.
165
156
166
-
=== "By metadata properties"
167
-
168
-
From the `Metadata` dropdown, type and select the property. You can set a range for integer and float properties, and select specific values for term metadata.
157
+
From the `Metadata` dropdown, type and select the property. You can set a range for integer and float properties, and select specific values for term metadata.
169
158
170
-
=== "By responses from the current user"
159
+
!!! note
160
+
Note that if a metadata property was set to `visible_for_annotators=False` this metadata property will only appear in the metadata filter for users with the `admin` or `owner` role.
171
161
172
-
From the `Responses` dropdown, type and select the question. You can set a range for rating questions and select specific values for label, multi-label, and span questions.
162
+
From the `Responses` dropdown, type and select the question. You can set a range for rating questions and select specific values for label, multi-label, and span questions.
173
163
174
-
!!! note
175
-
The text and ranking questions are not available for filtering.
164
+
!!! note
165
+
The text and ranking questions are not available for filtering.
176
166
177
-
=== "By suggestions"
178
167
179
-
From the Suggestions dropdown, Filter the suggestions by `Suggestion values`, `Score` , or `Agent`
168
+
From the Suggestions dropdown, filter the suggestions by `Suggestion values`, `Score` , or `Agent`.
180
169
181
-
<h4>Sort</h4>
170
+
#### Sort
182
171
183
172
You can sort your records according to one or several attributes.
184
173
@@ -196,4 +185,4 @@ The suggestion scores, response, and suggestion values for rating questions and
196
185
As an `owner` or `admin`, you can edit the guidelines as much as you need from the icon settings on the header. Markdown format is enabled.
197
186
198
187
!!! tip
199
-
If you want further guidance on good practices for guidelines during the project development, check our [blog post](https://argilla.io/blog/annotation-guidelines-practices/).
188
+
If you want further guidance on good practices for guidelines during the project development, check this [blog post](https://argilla.io/blog/annotation-guidelines-practices/).
0 commit comments