How do you do the OpenAI LoCoMo evaluation?

[Salmon-f42]

I have read the code about LoCoMo evaluation in this repo, but I don't find how you generate the response with OpenAI memory, per my understanding, OpenAI doesn't support memory in their API, so how do you evaluate its performance on LoCoMo dataset? could you kindly explain your method to evaluate it?

[Van0SS]

+100
Very interested as well

[hush-cd]

I have read the code about LoCoMo evaluation in this repo, but I don't find how you generate the response with OpenAI memory, per my understanding, OpenAI doesn't support memory in their API, so how do you evaluate its performance on LoCoMo dataset? could you kindly explain your method to evaluate it?

Thank you for your thoughtful question. You're absolutely right that OpenAI's memory feature is not accessible via the API. Currently, memory is only supported through the ChatGPT web interface, and users can view and manage it under Settings > Personalization > Memory. Here’s how we handled memory extraction and evaluation for the LoCoMo dataset:

1. Memory Extraction via ChatGPT Web Interface

• For each conversation in the LoCoMo dataset, we manually input its sessions (multi-turn dialogues) one at a time into the ChatGPT web interface, with memory enabled.
• After all sessions for a given conversation had been submitted, we went to Settings > Personalization > Manage memories to view the Saved memories generated by ChatGPT.
• We manually copied and saved these memories to a local file. All sessions from the same conversation shared the same memory file.
• Before moving on to the next conversation, we deleted all previously saved memories from ChatGPT to avoid memory contamination across conversations.
  

2. Structuring Conversation Memory Files

• Each memory file stored the accumulated memories extracted across all sessions within a single conversation.
• This ensured that memory extraction was scoped cleanly at the conversation level, and no unintended context carried over between different conversations.

3. Prompting gpt-4o‑mini with Simulated Memory

• At test time, for each LoCoMo evaluation question, we loaded the corresponding conversation’s memory file.
• We constructed a single API prompt by embedding:
  1. A brief instruction to GPT‑4o‑mini.
  2. All of the pre‑extracted memory entries for that conversation.
  3. The test question itself.
• We then sent this prompt to the GPT‑4o‑mini API and recorded the model’s response.

4. Rationale

• Because the API does not support automatic memory retrieval, we supply GPT‑4o‑mini with exactly the same “knowledge” that ChatGPT’s memory feature would have provided.
• This manual memory‑injection ensures that our evaluation of GPT‑4o‑mini on LoCoMo is not handicapped by the lack of an official memory API and is at least as strong as, if not stronger than, the built‑in ChatGPT memory retrieval mechanism.

5. Example Memory‑Extraction Prompt

   Can you please extract relevant information from this conversation and create memory
   entries for each user mentioned? Please store these memories in your knowledge base in
   addition to the timestamp provided for future reference and personalized interactions.
   
   (1:56 pm on 8 May, 2023) Caroline: Hey Mel! Good to see you! How have you been?
   (1:56 pm on 8 May, 2023) Melanie: Hey Caroline! Good to see you! I’m swamped with the
   kids & work. What’s up with you? Anything new?
   (1:56 pm on 8 May, 2023) Caroline: I went to a LGBTQ support group yesterday and it was so
   powerful.
   ...
   

I hope this clarifies our methodology. Please let me know if you have any further questions!