chatgpt_fn are far worse than alpaca_7b

[lucasjinreal]

Any possible reason for this?

[Spico197]

Hi there, thanks for the question! The reasons may be:

1. The seed data is lack of diversity, so $M_{0}$ is worse than alpaca.
2. The augmented data quality is low. From my personal observation, the curation response (evaluating the augmentation data by the seed model $M_{0}$ predicting scores) does not contain explanations of why the model evaluate such a score. So the quality may be worse.

Besides the above issues, there are some differences with the original paper:

1. The template: whether or not to use vicuna v1.1 as the instruction-response as a template may affect the performance.
2. The data source: since clueweb is not a free open-source dataset, it's hard to know the difference between falcon and clueweb.
3. The training steps: I found the number of steps affects the evaluation results significantly. Maybe current models are not trained sufficiently.

I'm also very curious about the low performance. Further discussions are warmly welcomed.

What do you think about it?