Input and Output Schema

[ryanmon1]

I have created 3900 Q&A pairs in the alpaca instruct format using the input and output schema. The dataset looks great and i want it exactly the way it is now. However, when i go to export the dataset, I can only export using a hugging face or open ai chat template. Why does the input and output schema allow for customizing only to be ignored on export? How do i export in the fromat i made the entire dataset in?

[scosman]

As in the output you see in Kiln has Alpaca formatting?

You're using it in a non typical way. Usual output is just the output data, not including any chat formatter data. That way we can create any chat format we need at runtime. By baking it in you're going to get double the chat format:

<export chat format>
   <alpaca chat format>
      data
    </alpaca chat format>
</export chat format>

The typical way is to just have synthetic data produce the output data, ignoring chat format, and add the chat format later. You system will eventually hit issues (tool call messages, etc).

Options you have with where you are:

1. start over, use the more typical format - only output data in the output, exclude chat format data
2. write a custom script using our python library to build alpaca format JSONL with what you have (works onetime, but you still have issue): https://kiln-ai.github.io/Kiln/kiln_core_docs/kiln_ai.html
3. Write a custom script using our python library to convert current data into a CSV with more standard input/output (parse then alpaca to get the output), then import than import that csv into a new task and continue with normal format.