VertexAI - Decoding the Chat history fails with "No text, inline data or function call was found." when a functionResponse exists #13593
Description
Description
In order to save the Chat history, I encode it and save it to a file. When the app starts, I load the Chat history from the file and decode it. This works fine unless the Chat history contains a functionResponse. Decoding the history when it contains a functionResponse fails.
Reproducing the issue
Use the code below to test. Call the await test() function.
The first time, loading the chat history will fail because no file called"testFile" exists. However, saving is successful so the file will exist the next time you call test().
The second time, the file exists but decoding fails.
let getExchangeRate = FunctionDeclaration(
name: "getExchangeRate",
description: "Get the exchange rate for currencies between countries",
parameters: [
"currencyFrom": Schema(
type: .string,
description: "The currency to convert from."
),
"currencyTo": Schema(
type: .string,
description: "The currency to convert to."
),
],
requiredParameters: ["currencyFrom", "currencyTo"]
)
func makeAPIRequest(currencyFrom: String,
currencyTo: String) -> JSONObject {
// This hypothetical API returns a JSON such as:
// {"base":"USD","rates":{"SEK": 10.99}}
return [
"base": .string(currencyFrom),
"rates": .object([currencyTo: .number(10.99)]),
]
}
func test() async {
let testFile : String = "testFile"
// Initialize the Vertex AI service
let vertex = VertexAI.vertexAI()
// Initialize the generative model
// Use a model that supports function calling, like a Gemini 1.5 model
let model = vertex.generativeModel(
modelName: "gemini-1.5-flash",
// Specify the function declaration.
tools: [Tool(functionDeclarations: [getExchangeRate])]
)
let history = loadChatHistory(from: testFile)
let chat = model.startChat(history : history)
let prompt = "How much is 50 US dollars worth in Swedish krona?"
// Send the message to the generative model
Task {
let response1 = try await chat.sendMessage(prompt)
// Check if the model responded with a function call
guard let functionCall = response1.functionCalls.first else {
fatalError("Model did not respond with a function call.")
}
// Print an error if the returned function was not declared
guard functionCall.name == "getExchangeRate" else {
fatalError("Unexpected function called: \(functionCall.name)")
}
// Verify that the names and types of the parameters match the declaration
guard case let .string(currencyFrom) = functionCall.args["currencyFrom"] else {
fatalError("Missing argument: currencyFrom")
}
guard case let .string(currencyTo) = functionCall.args["currencyTo"] else {
fatalError("Missing argument: currencyTo")
}
// Call the hypothetical API
let apiResponse = makeAPIRequest(currencyFrom: currencyFrom, currencyTo: currencyTo)
// Send the API response back to the model so it can generate a text response that can be
// displayed to the user.
let response = try await chat.sendMessage([ModelContent(
role: "function",
parts: [.functionResponse(FunctionResponse(
name: functionCall.name,
response: apiResponse
))]
)])
// Log the text response.
guard let modelResponse = response.text else {
fatalError("Model did not respond with text.")
}
saveChatHistory(chat.history, to: testFile)
print(modelResponse)
}
}
func saveChatHistory(_ history: [ModelContent], to filename: String) {
let fileManager = FileManager.default
let urls = fileManager.urls(for: .documentDirectory, in: .userDomainMask)
if let documentDirectory = urls.first {
let fileURL = documentDirectory.appendingPathComponent(filename)
let encoder = JSONEncoder()
encoder.outputFormatting = .withoutEscapingSlashes
do {
let data = try encoder.encode(history)
try data.write(to: fileURL)
print("Chat history saved successfully.")
} catch {
print("Failed to save chat history: \(error)")
}
}
}
func loadChatHistory(from filename: String) -> [ModelContent] {
let fileManager = FileManager.default
let urls = fileManager.urls(for: .documentDirectory, in: .userDomainMask)
if let documentDirectory = urls.first {
let fileURL = documentDirectory.appendingPathComponent(filename)
let decoder = JSONDecoder()
do {
let data = try Data(contentsOf: fileURL)
if let jsonString = String(data: data, encoding: .utf8) {
print("Loaded JSON string: \(jsonString)")
}
let history = try decoder.decode([ModelContent].self, from: data)
print("Chat history loaded successfully.")
return history
} catch {
print("Failed to load chat history: \(error)")
}
}
return []
}
Firebase SDK Version
11.1
Xcode Version
16.0
Installation Method
Swift Package Manager
Firebase Product(s)
VertexAI
Targeted Platforms
iOS
Relevant Log Output
Failed to load chat history: Error Domain=NSCocoaErrorDomain Code=260 "The file “testFile” couldn’t be opened because there is no such file." UserInfo={NSUnderlyingError=0x600000cd36c0 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}, NSURL=file:///Users/Shane/Library/Developer/CoreSimulator/Devices/C98366BC-B53E-466D-AAEC-2D950385105D/data/Containers/Data/Application/D490FDC4-6CB9-4F46-90B4-AC9D76562B42/Documents/testFile, NSFilePath=/Users/Shane/Library/Developer/CoreSimulator/Devices/C98366BC-B53E-466D-AAEC-2D950385105D/data/Containers/Data/Application/D490FDC4-6CB9-4F46-90B4-AC9D76562B42/Documents/testFile}
nw_connection_copy_connected_local_endpoint_block_invoke [C1] Connection has no local endpoint
nw_connection_copy_connected_local_endpoint_block_invoke [C1] Connection has no local endpoint
Chat history saved successfully.
50 US dollars is worth 549.5 Swedish krona.
Submit function called
[FirebaseVertexAI] To enable additional logging, add `-FIRDebugEnabled` as a launch argument in Xcode.
[FirebaseVertexAI] Model projects/expense-tracking-59dac/locations/us-central1/publishers/google/models/gemini-1.5-flash initialized.
Loaded JSON string: [{"role":"user","parts":[{"text":"How much is 50 US dollars worth in Swedish krona?"}]},{"parts":[{"functionCall":{"name":"getExchangeRate","args":{"currencyTo":"SEK","currencyFrom":"USD"}}}],"role":"model"},{"role":"function","parts":[{"functionResponse":{"response":{"base":"USD","rates":{"SEK":10.99}},"name":"getExchangeRate"}}]},{"parts":[{"text":"50 US dollars is worth 549.5 Swedish krona. \n"}],"role":"model"}]
Failed to load chat history: dataCorrupted(Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "text", intValue: nil), CodingKeys(stringValue: "inlineData", intValue: nil)], debugDescription: "No text, inline data or function call was found.", underlyingError: nil))
Chat history saved successfully.
50 US dollars is worth 549.5 Swedish krona.If using Swift Package Manager, the project's Package.resolved
Expand Package.resolved snippet
Replace this line with the contents of your Package.resolved.
If using CocoaPods, the project's Podfile.lock
Expand Podfile.lock snippet
Replace this line with the contents of your Podfile.lock!