fix: ensure valid UTF-8 byte sequence in HTTP response

[mattjohnsonpint]

Sanitizes HTTP response from models or API calls to strip away any invalid UTF-8 sequences or null bytes, if the expected content is a string, or an object serialized as a JSON string.

Fixes #903

[linear]

MOD-5 ERROR: invalid byte sequence for encoding \"UTF8\": 0xf8 (SQLSTATE 22021)

[Copilot]

Pull Request Overview

This PR fixes an issue with ensuring that HTTP responses contain valid UTF-8 byte sequences by sanitizing the content. Key changes include:

• Introducing Test_SanitizeUTF8 in strings_test.go with various edge cases.
• Adding the SanitizeUTF8 function in strings.go to remove invalid UTF-8 and null bytes.
• Applying the sanitation of HTTP response content in http.go before processing it.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
runtime/utils/strings_test.go	Added tests to verify UTF-8 sanitation logic.
runtime/utils/strings.go	Implemented SanitizeUTF8 to filter out invalid UTF-8 and null bytes.
runtime/utils/http.go	Integrated UTF-8 sanitation in HTTP response handling.
CHANGELOG.md	Updated changelog to reflect the fix for invalid UTF-8 responses.

Comments suppressed due to low confidence (1)

runtime/utils/strings.go:49

• It may be helpful to update the documentation comment to clarify that invalid multi-byte sequences and null bytes are skipped rather than replaced. This can help avoid any potential confusion about the function's behavior.

// SanitizeUTF8 removes invalid UTF-8 sequences from a byte slice.