Crashing with Diacritics #36
Description
It took me forever to track this down, but submitting a prompt with a diacritic (é, ç, etc.), it throws a corrupted memory exception. I was able to fix this by filtering my prompts when they come in. Just wanted to let you know:
private static string ReplaceDiacritics(string text)
{
if (string.IsNullOrWhiteSpace(text))
{
return text;
}
// Normalize the text to decompose accents into separate characters
var normalizedString = text.Normalize(NormalizationForm.FormD);
int length = normalizedString.Length;
// Use string.Create for optimized memory allocation
return string.Create(length, normalizedString, (span, value) =>
{
int index = 0;
foreach (var c in value)
{
// Directly check for non-spacing marks and only copy valid characters
if (CharUnicodeInfo.GetUnicodeCategory(c) != UnicodeCategory.NonSpacingMark)
{
span[index++] = c;
}
}
// Truncate the span to the valid length and normalize back to Form C
var cleanedSpan = span.Slice(0, index);
var cleanedString = new string(cleanedSpan);
cleanedString.Normalize(NormalizationForm.FormC).CopyTo(span);
});
}