Skip to content

Commit 8505e11

Browse files
renzhifeishijinPJ
renzhifei
authored and
shijinPJ
committed
添加 Mutli_Lan Dataset.md
1 parent db10370 commit 8505e11

File tree

1 file changed

+129
-0
lines changed

1 file changed

+129
-0
lines changed

‎docs/eval/dataset_multi_lan.md‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Multi_Lan Dataset
2+
3+
## Dataset Introduction
4+
Multi_Lan Dataset aims to evaluate the ability of Dingo's built-in prompt to mine low-quality data in multi-language pre-training datasets. We extracted a portion of data from the Common Crawl (CC) dataset, which was then annotated by experts in these languages based on seven quality dimensions（[quality_metrics](../metrics.md)）. If any dimension has problems, the data will be marked as low-quality data.
5+
6+
| Field Name | Description |
7+
|--------------|------------------------------|
8+
| data_id | A unique identifier for each data entry, without special significance; users can modify it according to their needs. |
9+
| content | The text content awaiting quality inspection. |
10+
| language | The language of the content. |
11+
| error_status | Data status: True indicates low-quality data, False indicates high-quality data.|
12+
| type_list | Types of problems found in low-quality data; this field is an empty list for normal data. |
13+
| name_list | Names of issues found in low-quality data; this field is an empty list for normal data. |
14+
| reason_list | Descriptions of problems found in low-quality data; this field is an empty list for normal data. |
15+
16+
### Dataset Link
17+
The dataset is available for different languages through the following links:
18+
19+
| Language | Dataset Link |
20+
|----------|----------------------------------------------|
21+
| Russian | https://huggingface.co/datasets/chupei/cc_ru |
22+
| Thai | https://huggingface.co/datasets/chupei/cc_th |
23+
| Vietnamese | https://huggingface.co/datasets/chupei/cc_vi |
24+
| Hungarian | https://huggingface.co/datasets/chupei/cc_hu |
25+
| Serbian | https://huggingface.co/datasets/chupei/cc_sr |
26+
27+
28+
### Dataset Composition
29+
The dataset includes five languages: Russian, Thai, Vietnamese, Hungarian, and Serbian. Below is a summary of each language's data:
30+
31+
| Language | Number of dataset | Number of High-Quality Data | Number of Low-Quality Data |
32+
|------|-------------------|-----------------------------|----------------------------|
33+
| Russian | 154 | 71 | 83 |
34+
| Thai | 267 | 128 | 139 |
35+
| Vietnamese | 214 | 101 | 113 |
36+
| Hungarian | 225 | 99 | 126 |
37+
| Serbian | 144 | 38 | 76 |
38+
39+
40+
41+
## Prompt Validation Metrics
42+
### Indicator Description
43+
For prompt validation, we focus on the precision of identifying low-quality data. The metrics are defined as follows:
44+
45+
| Indicator | Description |
46+
|-------------------------------|----------------------------------------|
47+
| TP(True Positive) |The number of high-quality data points correctly identified as high-quality. |
48+
| FP(False Positive) |The number of low-quality data points incorrectly identified as high-quality. |
49+
| TN(True Negative) |The number of low-quality data points correctly identified as low-quality. |
50+
| FN(False Negative) |The number of high-quality data points incorrectly identified as low-quality. |
51+
| Precision of Low-Quality Data | TN / (TN + FN) , the ratio of low-quality data correctly identified as such among all data marked as low-quality. |
52+
53+
54+
## prompt_text_quality_multilan 设计
55+
When evaluating different languages, the Role should be set to correspond with the language being evaluated. For instance, when evaluating Serbian, the prompt would be as follows:
56+
<pre>
57+
### Role
58+
You are an expert in Serbian language model.
59+
### Background
60+
The dataset has been compiled from a variety of sources, including social media platforms, news outlets, academic journals, and online forums.
61+
### Goals
62+
Your primary objective is to assess the suitability of this dataset for training a large language model.
63+
### Criteria
64+
ineffectiveness: Verify the effectiveness of the data. Data is considered ineffective if it is primarily composed of carriage returns or spaces. Additionally, data that includes a substantial amount of garbled text, either in Chinese or English, or contains nonsensical content, is also deemed ineffective. A text is labeled invalid if it is empty, consists only of a URL, contains only line breaks, or lacks sufficient length to provide meaningful information.
65+
irrelevance: Determine whether the data contains irrelevant information. Irrelevant information includes citation details, header and footer content, entity markers, non-visible characters, HTML tags, and special symbols. If the text contains a large amount of aggregated data, then this data must be relevant to the topic and separated using high-quality separators, otherwise this aggregated data is irrelevant content.
66+
incompleteness: Check the completeness of the text. Incomplete text may abruptly end with a colon or an ellipsis, or have mismatched parentheses, leading to incomplete meaning.
67+
disunderstandability: Assess the comprehensibility of the text. Ensure that LaTeX formulas and Markdown data are correctly formatted. In addition, the text should ensure correct segmentation and line breaks, and there should be no situations where sentences are unreasonably separated. If there is a list number in the text, the list number must be formatted consistently, correctly, and continuously readable. The text should not contain any tag links that cannot be parsed, nor should it contain a large number of spaces and line breaks that affect reading.
68+
dissimilarity: Examine the text for the presence of duplicate information, including consecutive repeated text and multiple occurrences of special symbols and characters.
69+
disfluency: Examine the text for fluency. The text should not have excessively long English words, large fragments lacking punctuation marks, anti crawling text, or content that is chaotic and does not conform to coherent reading order.
70+
insecurity: Ensure the data does not contain insecure content. Texts should be free from sensitive personal information, and should not include content related to gambling, pornography, political issues, or prohibited information.
71+
### Workflow
72+
1. Thoroughly read and comprehend the text provided by the user.
73+
2. Assign a score to the text. If the text does not meet any negative criteria mentioned above, the score is 1; otherwise, the score is 0.
74+
3. Assign a type to the text. If score is 1, type is none. If score is 0, type is one of the list: ["ineffectiveness", "incompleteness", "disunderstandability", "dissimilarity", "disfluency", "irrelevance", "insecurity"].
75+
4. State the reason for your evaluation.
76+
5. Return the results in JSON format: {"score": x, "type":"xxx", "reason": "xxx"}.
77+
### Warning
78+
Please remember to output only a JSON format data, without any additional content.
79+
80+
</pre>
81+
82+
83+
### Experiment Results
84+
Below are the experimental results showcasing the performance of the prompt across different languages:
85+
86+
| Language | number | TP | FP | TN | FN | Precision of Low-Quality Data (%) |
87+
|-------|--------|-----|----|----|----|-----------|
88+
| Russian | 154 | 79 | 16 | 55 | 4 | 93.22 |
89+
| Thai | 267 | 130 | 30 | 98 | 9 | 91.59 |
90+
| Vietnamese | 214 | 107 | 32 | 69 | 6 | 92.0 |
91+
| Hungarian | 225 | 124 | 30 | 69 | 2 | 97.18 |
92+
| Serbian | 114 | 75 | 6 | 32 | 1 | 96.97 |
93+
94+
95+
## Evaluation
96+
97+
```python
98+
from dingo.io import InputArgs
99+
from dingo.exec import Executor
100+
101+
102+
input_data = {
103+
"eval_group": "detect_text_quality_th",
104+
"input_path": "/your/dataset/path",
105+
"data_format": "jsonl",
106+
"column_content": "content",
107+
"save_data": True,
108+
"save_correct": True,
109+
"save_raw": True,
110+
"max_workers": 10,
111+
"batch_size": 10,
112+
"custom_config": {
113+
"prompt_list": ["PromptTextQualityTh"],
114+
"llm_config":
115+
{
116+
"detect_text_quality_detail":
117+
{
118+
"key": "EMPTY",
119+
"api_url": "your_model_api",
120+
}
121+
}
122+
}
123+
}
124+
input_args = InputArgs(**input_data)
125+
executor = Executor.exec_map["local"](input_args)
126+
result = executor.execute()
127+
print(result)
128+
129+
```

0 commit comments

Comments
 (0)