Skip to content

Commit 1a18247

Browse files
Update readme.md
1 parent daafdce commit 1a18247

File tree

1 file changed

+2
-36
lines changed

1 file changed

+2
-36
lines changed

readme.md

Lines changed: 2 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -23,73 +23,39 @@ You need the following to run the QueryCraft pipeline:
2323

2424
| Base Model | Eval Dataset | Accuracy |
2525
| -------------------------------- | ------------ | -------- |
26-
| DB-chat-sql model/Codellama-13b | Spider Dev | 80.00% |
2726
| DB-chat-sql model/Codellama-13b | Spider Dev | 63.60% |
2827
| Granite 20B code Instruct | Spider Dev | 63% |
2928
| CodeLlama 34B instruct | Spider Dev | 62.11% |
3029
| CodeLlama 13B Instruct | Spider Dev | 57.98% |
3130
| CodeLlama 7B Instruct | Spider Dev | 56% |
3231
| CodeLlama 7B Instruct -finetune | Spider Dev | 55.62% |
33-
| codeLlama 34B instruct | Spider Dev | 55.37% |
34-
| CodeLlama 7B Instruct -finetune | Spider Dev | 55.23% |
35-
| codeLlama 34B instruct | Spider Dev | 55.00% |
3632
| CodeLlama 13B Instruct -finetune | Spider Dev | 53.87% |
3733
| Lllama-2-70B | Spider Dev | 53.44% |
38-
| CodeLlama 13B Instruct-finetune | Spider Dev | 53.29% |
39-
| CodeLlama 7B Instruct -finetune | Spider Dev | 53.00% |
4034
| Lllama-2-70B-Chat | Spider Dev | 52.00% |
41-
| Codellama 13B Instruct | Spider Dev | 52.00% |
4235
| Defog-sqlcoder-34b-alpha | Spider Dev | 51.93% |
43-
| CodeLlama 7B Instruct | Spider Dev | 52% |
4436
| sqlcoder-34b-alpha | Spider Dev | 51.21% |
45-
| Granite 20B code Instruct | Spider Dev | 49% |
46-
| CodeLlama 7B Instruct -finetune | Spider Dev | 48.06% |
4737
| Llama-2-7B-Chat -finetune | Spider Dev | 27.00% |
48-
| CodeLlama 7B Instruct -finetune | Spider Dev | 11.00% |
38+
4939

5040
### 2. Enhanced outcomes are evident with the query correction service, as demonstrated by its post-processing accuracy.
5141

5242
| Base Model | Eval Dataset | Accuracy | Post Processed Accuracy |
5343
| -------------------------------- | ------------ | -------- | ----------------------- |
54-
| DB-chat-sql model/Codellama-13b | Spider Dev | 80.00% | 80.00% |
5544
| DB-chat-sql model/Codellama-13b | Spider Dev | 63.60% | 70.57% |
5645
| Granite 20B code Instruct | Spider Dev | 63% | 69% |
5746
| codeLlama 34B instruct | Spider Dev | 62.11% | 62.11% |
5847
| CodeLlama 13B Instruct | Spider Dev | 57.98% | 64.04% |
5948
| CodeLlama 7B Instruct | Spider Dev | 56% | 57.00% |
6049
| CodeLlama 7B Instruct -finetune | Spider Dev | 55.62% | 61.24% |
61-
| codeLlama 34B instruct | Spider Dev | 55.37% | 55.37% |
62-
| CodeLlama 7B Instruct -finetune | Spider Dev | 55.23% | 60.30% |
63-
| codeLlama 34B instruct | Spider Dev | 55.00% | 55.00% |
6450
| CodeLlama 13B Instruct -finetune | Spider Dev | 53.87% | 59.96% |
6551
| Lllama-2-70B | Spider Dev | 53.44% | 53.44% |
66-
| CodeLlama 13B Instruct-finetune | Spider Dev | 53.29% | 58.32% |
67-
| CodeLlama 7B Instruct -finetune | Spider Dev | 53.00% | 58.27% |
68-
| Codellama 13B Instruct | Spider Dev | 52.00% | 52.00% |
6952
| Lllama-2-70B-Chat | Spider Dev | 52.00% | 53.04% |
70-
| CodeLlama 7B Instruct | Spider Dev | 52% | 57.11% |
7153
| Defog-sqlcoder-34b-alpha | Spider Dev | 51.93% | 62.92% |
7254
| sqlcoder-34b-alpha | Spider Dev | 51.21% | 56.05% |
73-
| Granite 20B code Instruct | Spider Dev | 49% | 57% |
74-
| CodeLlama 7B Instruct -finetune | Spider Dev | 48.06% | 52.91% |
7555
| Llama-2-7B-Chat -finetune | Spider Dev | 27.00% | 32% |
76-
| CodeLlama 7B Instruct -finetune | Spider Dev | 11.00% | 25.96% |
7756

78-
### 3. Smaller fine-tuned models outperforming larger ones
57+
### 3. Smaller fine-tuned models outperform larger ones
7958

80-
| Base Model | Eval Dataset | Accuracy | Post Processed Accuracy |
81-
| -------------------------------- | ------------ | -------- | ----------------------- |
82-
| CodeLlama 7B Instruct -finetune | Spider Dev | 55.62% | 61.24% |
83-
| codeLlama 34B instruct | Spider Dev | 55.37% | 55.37% |
84-
| CodeLlama 7B Instruct -finetune | Spider Dev | 55.23% | 60.30% |
85-
| CodeLlama 13B Instruct -finetune | Spider Dev | 53.87% | 59.96% |
86-
| Lllama-2-70B | Spider Dev | 53.44% | 53.44% |
87-
| CodeLlama 13B Instruct-finetune | Spider Dev | 53.29% | 58.32% |
88-
| CodeLlama 7B Instruct -finetune | Spider Dev | 53.00% | 58.27% |
89-
| Lllama-2-70B-Chat | Spider Dev | 52.00% | 53.04% |
90-
| CodeLlama 7B Instruct | Spider Dev | 52% | 57.11% |
91-
| sqlcoder-34b-alpha | Spider Dev | 51.21% | 56.05% |
92-
| Granite 20B code Instruct | Spider Dev | 49% | 57% |
9359

9460

9561

0 commit comments

Comments
 (0)