Skill Knowhow: Refining skills

[obuzek]

We need better documentation on ensuring that the skills you contribute will actually improve the model.

People have reported their generate_.jsonl and train_.jsonl files (as a result of running lab generate) contain relatively inaccurate alternate instruction suggestions. We'll likely get a better skill tree if people learn to look at these outputs, and refine the verbiage of their skills until they're getting strong answers from the teacher model (the one you load up with lab serve before you run lab generate).

@xukai92's commentary:

• how easy/hard to get good synthetic data depends on the q/a themselves. perhaps rewriting or rephrasing could help
• there are some improvements on the method itself we will deploy in the next few days
• potentially switch the model used by lab generate could be helpful, but it requires a better machine. all it needs is an endpoint but instructions to be added (instructions to use a stronger teacher model instructlab#398)
• if you can load mixtral-8x7b-v0.1.Q4_K_M.gguf from https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF in lab serve and document how to do this it would be very helpful for others

Tasks

• Add Kai's commentary to the skill info docs
• Suggest in both skill development and triage info docs that changing your wording and rerunning the skill until your .jsonl output has improved will be more beneficial to the model

[obuzek]

See also instructlab/instructlab#398

[code2automate]

We need better documentation on ensuring that the skills you contribute will actually improve the model.

People have reported their generate_.jsonl and train_.jsonl files (as a result of running lab generate) contain relatively inaccurate alternate instruction suggestions. We'll likely get a better skill tree if people learn to look at these outputs, and refine the verbiage of their skills until they're getting strong answers from the teacher model (the one you load up with lab serve before you run lab generate).

@xukai92's commentary:

• how easy/hard to get good synthetic data depends on the q/a themselves. perhaps rewriting or rephrasing could help
• there are some improvements on the method itself we will deploy in the next few days
• potentially switch the model used by lab generate could be helpful, but it requires a better machine. all it needs is an endpoint but instructions to be added (instructions to use a stronger teacher model cli#398)
• if you can load mixtral-8x7b-v0.1.Q4_K_M.gguf from https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF in lab serve and document how to do this it would be very helpful for others

Tasks

• Add Kai's commentary to the skill info docs
• Suggest in both skill development and triage info docs that changing your wording and rerunning the skill until your .jsonl output has improved will be more beneficial to the model

Hi @obuzek

It would be helpful for the first time contributers if you could please provide references to the docs you've mentioned above (skill info, skill development & triage info docs)

I've taken this opportunity to provide the references, kindly update if they are incorrect

skill info doc : SKILLS_GUIDE.md
skill development doc : SKILLS_GUIDE.md
triage info doc : skills-triage.md

[obuzek]

The Community Learning Guide covers some relevant topics, but not yet this one.

Automatic generation in response to PRs is getting added soon, which may help people with this.

Leaving this open since this remains relevant information.

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 31 days.