API Design: Prompt templating

[jjtang1985]

We want to have the alignment of SDK API design for the prompt templating module of the orchestration service.

1. Background

Orchestration service changed their spec for templating_module_config.template

1.1 Before

The template is a string w/ or w/o variables

            "templating_module_config": {
                "template": "my prompt: {myVariable}"
            }
...
            "messages_history": [
...

1.2 Now

The template is an array of object

            "templating_module_config": {
                "template": [
                  { "role": "system", "content": "my prompt: {myVariable}" },
                  { "role": "user", "content": "my prompt: {myVariable}" },
                ]
            }
...
            "messages_history": [
...

2. Reason of the change

There are two major scenarios, where the scenario 1 needs an array:

2.1 Scenario 1: Start a chat

• Users provide a prompt w/ or w/o system prompt
  • w/ system prompt, where the prompt can also be a template, so both of them (${{\color{red}array}}\ $) need to be part under the template key
  • w/o system prompt, works for both old (1.1) and new version (1.2)
• The history is then empty

2.2 Scenario 2: Continue a chat with previous context

• Users provide a prompt, and logically the system prompt should not be there
• The history is not empty

3. API Alternatives

  const systemPrompt = {role: "system", content: "system prompt: {systemParam}"}
  const userPrompt = {role: "user", content: "user prompt: {userParam}"}

3.1 Option A: same structure as orchestration service

• SDK prompt will be passed as orchestration templating_module_config.template
• SDK message_history will be passed as orchestration templating_module_config.messages_history

Scenario 1

.chatCompletion({
  model: 'gpt-4', 
  // without system prompt 
  // prompt:[userPrompt],
  prompt:[systemPrompt, userPrompt], 
  template_params: {
    systemParam: "You are a helpful assistant.", 
    userParam: "What's the weather like today in Potsdam?"
  }
})

Scenario 2

.chatCompletion({
  model: 'gpt-4', 
  prompt:[userPrompt], 
  template_params: {
    userParam: "What's the weather like today in Potsdam?"
  }, 
  // if there is a system message, it's not a template
  // message_history: [sysPrompt, oldUserMessage, oldAIMessage]
  message_history: [oldUserMessage, oldAIMessage]
})

• Pros

1. easy SDK development
2. understandable API, where last prompt is defined in the prompt
3. extendable to prompt registry, where prompt can be replaced by "prompt id"

3.2 Option B: Put everything (last prompt + history) into one array

• pick the last one or two items from the array, and pass to orchestration templating_module_config.template
• pass the rest of the items in the array to orchestration templating_module_config.messages_history

Scenario 1

.chatCompletion({
  model: 'gpt-4', 
  template_params: {
    systemParam: "You are a helpful assistant.", 
    userParam: "What's the weather like today in Potsdam?"
  },
  // without system prompt
  // message_history: [userPrompt]
  message_history: [systemPrompt, userPrompt] // both of these will be passed to orchestration templating_module_config.template
})

Scenario 2

.chatCompletion({
  model: 'gpt-4', 
  template_params: {
    userParam: "What's the weather like today in Potsdam?"
  }, 
  // if there is a system message, it's not a template
  // message_history: [sysPrompt, oldUserMessage, oldAIMessage, userPrompt]
  message_history: [oldUserMessage, oldAIMessage, userPrompt] // only the userPrompt will be passed to orchestration templating_module_config.template
})

• Pros

1. similar UX to Azure OpenAI client and LangChain

• Cons

1. make our life hard

Appendix

Orchestration merges the info in the following order:

• start building messages with history
  messages = [].append(message_history)
• handle templates
  prompt = template.replace(placeholder, value)
• add prompts/templates to the message array
  messages = messages.append(prompt)

source code