Scope out question generation

[kcarnold]

• Collect a bunch of example inputs and outputs from last summer's exploratory work
  • We have one or two examples already.
• Wrangle these examples into something we can use to train (fine-tune) a LM
  • we could start with the approach of the Interviewer even though it's not perfect...
• Pick an LM that we can feasibly run inference on
  • OpenAI API?
  • One of the existing open-source ones we've used (maybe an encoder-decoder one like flan-ul2)
  • one of the new batch of open-source models (LLaMa etc), fine-tuned perhaps
• Fine-tune the LM to generate questions like our examples
  • We have training scripts from 1.5 yr ago for Interview, and last summer for Inquisitive. example
  • Alternative: https://arxiv.org/abs/2305.14314, https://github.com/huggingface/peft
• Collect ranking data on LM generations
  • build Streamlit app for this? Maybe there's already an app that people are using, e.g., any open-source ChatGPT replication project will have something like this. Vicunia, Alpaca... see llama.cpp repo. or https://arxiv.org/abs/2204.05862
• Use ranking data to optimize the LM
  • see https://github.com/eric-mitchell/direct-preference-optimization for one impl
• Deploy the optimized LM as an API that the frontend can access.
  • We've already got an API for the interviewer model.

Design a simple format for input and output. e.g., input is document_text, cursor_position, and optional question_type, output is question, start_position, end_position (where positions are character offsets from the beginning of the texts)

[kcarnold]

Example of using GPT-3.5 for this:

You an helping a writer You ask the writer questions to help them think of what to discuss. Write 5 questions for the following paragraph.

ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.

[
{
    "question": "How can we get the model to learn from the demonstrator's knowledge without using supervised training?",
   "phrase_from_paragraph":"training the model to be more cautious causes it to decline questions that it can answer correctly"
},
{
    "question":"How can we make sure that the model's answers are accurate without a source of truth?",
    "phrase_from_paragraph": "during RL training, there’s currently no source of truth"
},
{
    "question":"What challenges do we face when fixing ChatGPT's incorrect answers?",
    "phrase_from_paragraph":"Fixing this issue is challenging"
},
{
    "question":"What are the consequences of supervised training on the model?",
    "phrase_from_paragraph": "supervised training misleads the model"
},
{
    "question": "What changes when the model knows the ideal answer?",
    "phrase_from_paragraph": "the ideal answer depends on what the model knows, rather than what the human demonstrator knows"
}
]

A tool like jsonformer or guidance could help ensure it generates valid JSON.

[kcarnold]

You an helping a writer You ask the writer questions to help them think of what to discuss. Write 5 questions for the following paragraph. For each question, include the phrase from the paragraph that it applies to.

paragraph = """
Whatever we make of love between humans, we likely will agree that it involves a great deal of intimacy, fulfillment, and passion. With cocnern to romantic love especially, sexual relations are often interwoven with the emotional attachment and relational exclusivitycommonly expected in monogamous contexts. As fleeting as each human life is, love with a partner can make life feel like a worthwhile eternity. Yet that worthwhile eternity implies in it that love has a binding power, one that can influence if not determine our life trajectory. The hookup culture has arisen in recent decades as a response to previous framings of love which deeply tie it to one’s life trajectory and decouple the previously accepted link between sexual relations from emotional attachment. How can this culture, running seemingly counter to decades of common wisdom, affect a generation of people looking for lifelong stability and fulfillment?
"""

questions = 
[
{
    "question":  "What is the definition of love between humans?",
    "phrase": "Whatever we make of love between humans"
},
{
    "question": "What does romantic love involve?",
    "phrase": "With cocnern to romantic love especially"
},
{
    "question": "What does a worthy eternity imply?",
    "phrase": "Yet that worthwhile eternity implies in it"
},
{
    "question": "What ties love to one's life trajectory?",
    "phrase": "one that can influence if not determine our life trajectory"
},
{
    "question": "How can hook up culture affect the search for stability and fulfillment?",
    "phrase": "How can this culture, running seemingly counter to decades of common wisdom"

}
]