@@ -23,73 +23,39 @@ You need the following to run the QueryCraft pipeline:
2323
2424| Base Model | Eval Dataset | Accuracy |
2525| -------------------------------- | ------------ | -------- |
26- | DB-chat-sql model/Codellama-13b | Spider Dev | 80.00% |
2726| DB-chat-sql model/Codellama-13b | Spider Dev | 63.60% |
2827| Granite 20B code Instruct | Spider Dev | 63% |
2928| CodeLlama 34B instruct | Spider Dev | 62.11% |
3029| CodeLlama 13B Instruct | Spider Dev | 57.98% |
3130| CodeLlama 7B Instruct | Spider Dev | 56% |
3231| CodeLlama 7B Instruct -finetune | Spider Dev | 55.62% |
33- | codeLlama 34B instruct | Spider Dev | 55.37% |
34- | CodeLlama 7B Instruct -finetune | Spider Dev | 55.23% |
35- | codeLlama 34B instruct | Spider Dev | 55.00% |
3632| CodeLlama 13B Instruct -finetune | Spider Dev | 53.87% |
3733| Lllama-2-70B | Spider Dev | 53.44% |
38- | CodeLlama 13B Instruct-finetune | Spider Dev | 53.29% |
39- | CodeLlama 7B Instruct -finetune | Spider Dev | 53.00% |
4034| Lllama-2-70B-Chat | Spider Dev | 52.00% |
41- | Codellama 13B Instruct | Spider Dev | 52.00% |
4235| Defog-sqlcoder-34b-alpha | Spider Dev | 51.93% |
43- | CodeLlama 7B Instruct | Spider Dev | 52% |
4436| sqlcoder-34b-alpha | Spider Dev | 51.21% |
45- | Granite 20B code Instruct | Spider Dev | 49% |
46- | CodeLlama 7B Instruct -finetune | Spider Dev | 48.06% |
4737| Llama-2-7B-Chat -finetune | Spider Dev | 27.00% |
48- | CodeLlama 7B Instruct -finetune | Spider Dev | 11.00% |
38+
4939
5040### 2. Enhanced outcomes are evident with the query correction service, as demonstrated by its post-processing accuracy.
5141
5242| Base Model | Eval Dataset | Accuracy | Post Processed Accuracy |
5343| -------------------------------- | ------------ | -------- | ----------------------- |
54- | DB-chat-sql model/Codellama-13b | Spider Dev | 80.00% | 80.00% |
5544| DB-chat-sql model/Codellama-13b | Spider Dev | 63.60% | 70.57% |
5645| Granite 20B code Instruct | Spider Dev | 63% | 69% |
5746| codeLlama 34B instruct | Spider Dev | 62.11% | 62.11% |
5847| CodeLlama 13B Instruct | Spider Dev | 57.98% | 64.04% |
5948| CodeLlama 7B Instruct | Spider Dev | 56% | 57.00% |
6049| CodeLlama 7B Instruct -finetune | Spider Dev | 55.62% | 61.24% |
61- | codeLlama 34B instruct | Spider Dev | 55.37% | 55.37% |
62- | CodeLlama 7B Instruct -finetune | Spider Dev | 55.23% | 60.30% |
63- | codeLlama 34B instruct | Spider Dev | 55.00% | 55.00% |
6450| CodeLlama 13B Instruct -finetune | Spider Dev | 53.87% | 59.96% |
6551| Lllama-2-70B | Spider Dev | 53.44% | 53.44% |
66- | CodeLlama 13B Instruct-finetune | Spider Dev | 53.29% | 58.32% |
67- | CodeLlama 7B Instruct -finetune | Spider Dev | 53.00% | 58.27% |
68- | Codellama 13B Instruct | Spider Dev | 52.00% | 52.00% |
6952| Lllama-2-70B-Chat | Spider Dev | 52.00% | 53.04% |
70- | CodeLlama 7B Instruct | Spider Dev | 52% | 57.11% |
7153| Defog-sqlcoder-34b-alpha | Spider Dev | 51.93% | 62.92% |
7254| sqlcoder-34b-alpha | Spider Dev | 51.21% | 56.05% |
73- | Granite 20B code Instruct | Spider Dev | 49% | 57% |
74- | CodeLlama 7B Instruct -finetune | Spider Dev | 48.06% | 52.91% |
7555| Llama-2-7B-Chat -finetune | Spider Dev | 27.00% | 32% |
76- | CodeLlama 7B Instruct -finetune | Spider Dev | 11.00% | 25.96% |
7756
78- ### 3. Smaller fine-tuned models outperforming larger ones
57+ ### 3. Smaller fine-tuned models outperform larger ones
7958
80- | Base Model | Eval Dataset | Accuracy | Post Processed Accuracy |
81- | -------------------------------- | ------------ | -------- | ----------------------- |
82- | CodeLlama 7B Instruct -finetune | Spider Dev | 55.62% | 61.24% |
83- | codeLlama 34B instruct | Spider Dev | 55.37% | 55.37% |
84- | CodeLlama 7B Instruct -finetune | Spider Dev | 55.23% | 60.30% |
85- | CodeLlama 13B Instruct -finetune | Spider Dev | 53.87% | 59.96% |
86- | Lllama-2-70B | Spider Dev | 53.44% | 53.44% |
87- | CodeLlama 13B Instruct-finetune | Spider Dev | 53.29% | 58.32% |
88- | CodeLlama 7B Instruct -finetune | Spider Dev | 53.00% | 58.27% |
89- | Lllama-2-70B-Chat | Spider Dev | 52.00% | 53.04% |
90- | CodeLlama 7B Instruct | Spider Dev | 52% | 57.11% |
91- | sqlcoder-34b-alpha | Spider Dev | 51.21% | 56.05% |
92- | Granite 20B code Instruct | Spider Dev | 49% | 57% |
9359
9460
9561
0 commit comments