Skip to content

Commit ae4197c

Browse files
shijinpjlabpekopoke
shijinpjlab
and
pekopoke
authored
feat: support sql datasouce multi-column eval (#259)
* feat: temp * feat: merge_result_info 移动位置 * feat: rule已支持 * feat: fix bug * feat: prompt临时code * feat: field由list改成dict，跑通rule * feat: 改名map_data * feat: prompt与llm合并 * feat: 批量合并 * feat: delete evaldata * feat: 改名evalpipline * feat: 调整map_data * feat: 合并evaluate_rule和evaluate_prompt * feat: 并发v3 * feat: 合并evaluate_single_data与evaluate_by_type * feat: 合并execute与evaluate * feat: 修复bug并发导致的配置覆盖 * feat: 调整位置 * feat: 修改local文件，适配新版result_info和modelres的error_type（summary模块待更新） * feat: summary模块 * feat: error_type的value由reason列表改为dict，包含2个key：metric、reason * feat: update * feat: 添加ResTypeInfo类 * feat: rule_common.py更新返回 * feat: 4个rule文件更新返回 * feat: llm更新（除了type是列表） * feat: 移动位置 * feat: 移动位置引发的import修改 * feat: error_type删除一层 * feat: result_save.good判断逻辑 * feat: update * feat: rule_common.py更新res，添加label * feat: 更新res，添加label * feat: 更新res，添加label * feat: fix lint * feat: 4中base convertor * feat: plaintext情况 * feat: plaintext save * feat: json修复 * feat: jsonl修复 * feat: listjson修复 * feat: hf_plaintext.json 修复 * feat: hf_json 修复 * feat: hf_jsonl 修复 * feat: hf_listjson 修复 * feat: 修复bug 多规则结果异常 * feat: 修复bug 多规则结果异常 * feat: custom config rule 修复 * feat: fix test_local.py * feat: fix test_local.py * feat: fix test_continue.py * feat: fix test_write.py 修复复杂rule * feat: fix test_rule_common.py * feat: ImageConverter * feat: fix lint * feat: label是数组的情况 * feat: 文件夹名 * feat: example更新 * feat: 删除特殊prompt * feat: 删除prompt类 * feat: fix lint * feat: ModelRes优化赋值，Model删除prompt相关 * feat: fix lint * feat: ignore * feat: ModelRes固定字段 * update res b_box overlap and visual rule * update res b_box overlap and visual rule * feat: spark的evaluate完成 * feat: spark的summarize更新 * feat: fix bug * feat: 更新model * feat: fix lint * feat: TestModelRes * feat: chupei 特殊场景 * feat: fix bug * update res b_box overlap and visual rule * feat: delete old convertor * feat: 优化local，改prompt为llm * feat: 添加sql来源 * feat: fix lint * feat: LLMHtmlExtractCompareEn * feat: fix lint * feat: change name * feat: 删除DatasetArgs中fields功能 * feat: fix bug plaintext * feat: fix lint * feat: test ignore rag --------- Co-authored-by: pekopoke <[email protected]>
1 parent 1cd4e97 commit ae4197c

File tree

215 files changed

+7782
-6313
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

215 files changed

+7782
-6313
lines changed

‎.github/env/custom_config_rule.json‎

Lines changed: 9 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,16 @@
11
{
22
"input_path": "test/data/test_local_json.json",
3-
"log_level": "DEBUG",
43
"dataset": {
54
"source": "local",
6-
"format": "json",
7-
"field": {
8-
"content": "prediction"
9-
}
10-
},
11-
"executor": {
12-
"rule_list": ["RuleSpecialCharacter", "RuleWatermark"]
5+
"format": "json"
136
},
14-
"evaluator": {
15-
"rule_config": {
16-
"RuleSpecialCharacter": {
17-
"pattern": "[�^□]|\\{\\/U\\}"
18-
},
19-
"RuleWatermark": {
20-
"key_list": ["谢邀", "Architecture of dingo"]
21-
}
7+
"evaluator": [
8+
{
9+
"fields": {"content": "prediction"},
10+
"evals": [
11+
{"name": "RuleSpecialCharacter", "config": {"pattern": "[�^□]|\\{\\/U\\}"}},
12+
{"name": "RuleWatermark", "config": {"key_list": ["谢邀", "Architecture of dingo"]}}
13+
]
2214
}
23-
}
15+
]
2416
}

‎.github/env/hf_json.json‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,14 @@
22
"input_path": "chupei/format-json",
33
"dataset": {
44
"source": "hugging_face",
5-
"format": "json",
6-
"field": {
7-
"prompt": "origin_prompt",
8-
"content": "prediction"
9-
}
5+
"format": "json"
106
},
11-
"executor": {
12-
"eval_group": "default"
13-
}
7+
"evaluator": [
8+
{
9+
"fields": {"prompt": "origin_prompt", "content": "prediction"},
10+
"evals": [
11+
{"name": "RuleColonEnd"}
12+
]
13+
}
14+
]
1415
}

‎.github/env/hf_jsonl.json‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,14 @@
22
"input_path": "chupei/format-jsonl",
33
"dataset": {
44
"source": "hugging_face",
5-
"format": "jsonl",
6-
"field": {
7-
"content": "content"
8-
}
5+
"format": "jsonl"
96
},
10-
"executor": {
11-
"eval_group": "default"
12-
}
7+
"evaluator": [
8+
{
9+
"fields": {"content": "content"},
10+
"evals": [
11+
{"name": "RuleColonEnd"}
12+
]
13+
}
14+
]
1315
}

‎.github/env/hf_listjson.json‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,14 @@
22
"input_path": "chupei/format-listjson",
33
"dataset": {
44
"source": "hugging_face",
5-
"format": "listjson",
6-
"field": {
7-
"prompt": "instruction",
8-
"content": "output"
9-
}
5+
"format": "listjson"
106
},
11-
"executor": {
12-
"eval_group": "default"
13-
}
7+
"evaluator": [
8+
{
9+
"fields": {"prompt": "instruction", "content": "output"},
10+
"evals": [
11+
{"name": "RuleColonEnd"}
12+
]
13+
}
14+
]
1415
}

‎.github/env/hf_plaintext.json‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,13 @@
22
"input_path": "chupei/format-text",
33
"dataset": {
44
"source": "hugging_face",
5-
"format": "plaintext",
6-
"field": {
7-
"content": "text"
8-
}
5+
"format": "plaintext"
96
},
10-
"executor": {
11-
"eval_group": "default"
12-
}
7+
"evaluator": [
8+
{
9+
"evals": [
10+
{"name": "RuleColonEnd"}
11+
]
12+
}
13+
]
1314
}

‎.github/env/local_json.json‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,14 @@
22
"input_path": "test/data/test_local_json.json",
33
"dataset": {
44
"source": "local",
5-
"format": "json",
6-
"field": {
7-
"content": "prediction"
8-
}
5+
"format": "json"
96
},
10-
"executor": {
11-
"eval_group": "default"
12-
}
7+
"evaluator": [
8+
{
9+
"fields": {"content": "prediction"},
10+
"evals": [
11+
{"name": "RuleColonEnd"}
12+
]
13+
}
14+
]
1315
}

‎.github/env/local_jsonl.json‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,14 @@
22
"input_path": "test/data/test_local_jsonl.jsonl",
33
"dataset": {
44
"source": "local",
5-
"format": "jsonl",
6-
"field": {
7-
"content": "content"
8-
}
5+
"format": "jsonl"
96
},
10-
"executor": {
11-
"eval_group": "default"
12-
}
7+
"evaluator": [
8+
{
9+
"fields": {"content": "content"},
10+
"evals": [
11+
{"name": "RuleColonEnd"}
12+
]
13+
}
14+
]
1315
}

‎.github/env/local_listjson.json‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,14 @@
22
"input_path": "test/data/test_local_listjson.json",
33
"dataset": {
44
"source": "local",
5-
"format": "listjson",
6-
"field": {
7-
"content": "output"
8-
}
5+
"format": "listjson"
96
},
10-
"executor": {
11-
"eval_group": "default"
12-
}
7+
"evaluator": [
8+
{
9+
"fields": {"content": "output"},
10+
"evals": [
11+
{"name": "RuleColonEnd"}
12+
]
13+
}
14+
]
1315
}

‎.github/env/local_plaintext.json‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,11 @@
44
"source": "local",
55
"format": "plaintext"
66
},
7-
"executor": {
8-
"eval_group": "default"
9-
}
7+
"evaluator": [
8+
{
9+
"evals": [
10+
{"name": "RuleColonEnd"}
11+
]
12+
}
13+
]
1014
}

‎.github/env/local_plaintext_save.json‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,15 @@
55
"format": "plaintext"
66
},
77
"executor": {
8-
"eval_group": "default",
98
"result_save": {
109
"bad": true
1110
}
12-
}
11+
},
12+
"evaluator": [
13+
{
14+
"evals": [
15+
{"name": "RuleColonEnd"}
16+
]
17+
}
18+
]
1319
}

0 commit comments

Comments
 (0)