ID3v2: COMM/USLT frames with a single BOM can't be parsed

[loxoron218]

Reproducer

1. Create the Directory Structure

In your terminal, create a directory path that includes the special U+2010 hyphen. Note: This character looks identical to a normal hyphen but is a different Unicode code point. You can copy-paste the command below.

# Create a directory named "Low‐Life" where the hyphen is the Unicode character U+2010
mkdir -p "/tmp/music-test/Low‐Life"

2. Add a Sample File

Place any valid FLAC file into this directory. The contents and tags of the file do not matter.

Copy a known-good FLAC file: cp "/path/to/any/good.flac" "/tmp/music-test/Low‐Life/test.flac"

# Or for example, create a dummy FLAC file for testing
# To create a dummy file if you don't have one:
dd if=/dev/zero bs=1k count=44 | flac - -o "/tmp/music-test/Low‐Life/test.flac"

3. Use the following Rust code

• Cargo.toml:

  [dependencies]
  lofty = "*"

• src/main.rs:

     use std::{borrow::Cow, error::Error, future::Future, path::Path, pin::Pin, rc::Rc, sync::Arc};
     use std::cell::{Cell, RefCell};
  
     use glib::MainContext;
     use gtk4::Label;
     use libadwaita::ViewStack;
     use libadwaita::prelude::WidgetExt;
     use lofty::probe::Probe;
     use lofty::prelude::{Accessor, AudioFile, TaggedFileExt, ItemKey};
     use lofty::tag::items::Timestamp;
     use regex::Regex;
     use sqlx::{query, Row, SqlitePool};
     use tokio::fs::{File, read_dir};
     use tokio::io::{AsyncBufReadExt, BufReader};
     use tokio::sync::mpsc::{UnboundedReceiver, UnboundedSender};
  
     ...
     ...
     
     /// Process a single audio file: extract tags, handle missing metadata, and insert into the database.
     async fn process_file(
        pool: &SqlitePool,
        path: &Path,
        folder_id: i64,
        dr_value: Option<u8>,
      ) -> Result<(), Box<dyn Error>> {
        // Probe the file to read metadata. If it fails, we can't process it.
        let tagged_file = match Probe::open(path) {
            Ok(probe) => match probe.read() {
                Ok(tf) => tf,
                Err(e) => {
                    eprintln!("Error reading metadata for {:?}: {}. Skipping.", path, e);
                    return Ok(());
                }
            },
            Err(e) => {
                eprintln!("Error opening file {:?}: {}. Skipping.", path, e);
                return Ok(());
            }
         };
  
         let tag = tagged_file.primary_tag(); // This is an Option<&Tag>
         let properties = tagged_file.properties();

Then I see:
Error reading metadata for "/home/arch/Musik/New Order/Low-Life/01 Love Vigilantes.flac": Text decoding: UTF-16 string has an invalid byte order mark. Skipping.
Error reading metadata for "/home/arch/Musik/New Order/Low-Life/02 The Perfect Kiss.flac": Text decoding: UTF-16 string has an invalid byte order mark. Skipping.
Error reading metadata for "/home/arch/Musik/New Order/Low-Life/03 This Time of Night.flac": Text decoding: UTF-16 string has an invalid byte order mark. Skipping.
Error reading metadata for "/home/arch/Musik/New Order/Low-Life/04 Sunrise.flac": Text decoding: UTF-16 string has an invalid byte order mark. Skipping.
Error reading metadata for "/home/arch/Musik/New Order/Low-Life/05 Elegia.flac": Text decoding: UTF-16 string has an invalid byte order mark. Skipping.
Error reading metadata for "/home/arch/Musik/New Order/Low-Life/06 Sooner Than You Think.flac": Text decoding: UTF-16 string has an invalid byte order mark. Skipping.
Error reading metadata for "/home/arch/Musik/New Order/Low-Life/07 Sub-Culture.flac": Text decoding: UTF-16 string has an invalid byte order mark. Skipping.
Error reading metadata for "/home/arch/Musik/New Order/Low-Life/08 Face Up.flac": Text decoding: UTF-16 string has an invalid byte order mark. Skipping.

Summary

lofty fails to parse valid FLAC files when their directory path contains a Unicode Hyphen (U+2010), producing a misleading error, Text decoding: UTF-16 string has an invalid byte order mark, which incorrectly points to file metadata corruption instead of the path issue.

Expected behavior

I expect lofty to successfully open and parse the metadata of any valid FLAC file, regardless of the Unicode characters present in its file path. The program should run and not print any error.

Assets

No specific sample file is required, as the bug is path-dependent, not file-dependent. Any valid FLAC file will trigger the bug when placed in the described directory.

The critical asset is the Unicode character itself:

• Problematic Character: ‐ (U+2010, HYPHEN)
• Normal Character: - (U+002D, HYPHEN-MINUS)

[Serial-ATA]

Text decoding errors are coming from ID3v2 tag parsing. Lofty doesn't do any parsing of the file path, it's immediately consumed in Probe::open() to open the file.

Since the errors are all coming from the same album in your example I imagine the whole thing was just encoded in a weird way. Can you email one of the files to serial@[DOMAIN ON MY PROFILE]?

[Serial-ATA]

The issue is that the file has a COMM frame where only the description has a BOM. I've only ever seen that issue in SYLT frames before:

lofty-rs/lofty/src/id3/v2/items/sync_text.rs

Lines 160 to 211 in 9b68924

	// It's possible for the description to be the only string with a BOM
	// To be safe, we change the encoding to the concrete variant determined from the description
	if encoding == TextEncoding::UTF16 {
	endianness = match cursor.get_ref()[..=1] {
	[0xFF, 0xFE] => u16::from_le_bytes,
	[0xFE, 0xFF] => u16::from_be_bytes,
	// Since the description was already read, we can assume the BOM was valid
	_ => unreachable!(),
	};
	}
	let mut pos = 0;
	let total = (data.len() - 6) as u64 - cursor.stream_position()?;
	let mut content = Vec::new();
	while pos < total {
	let text = (|| -> Result<String> {
	if encoding == TextEncoding::UTF16 {
	// Check for a BOM
	let mut bom = [0; 2];
	cursor
	.read_exact(&mut bom)
	.map_err(|_| Id3v2Error::new(Id3v2ErrorKind::BadSyncText))?;
	cursor.seek(SeekFrom::Current(-2))?;
	// Encountered text that doesn't include a BOM
	if bom != [0xFF, 0xFE] && bom != [0xFE, 0xFF] {
	let (raw_text, _) = read_to_terminator(&mut cursor, TextEncoding::UTF16);
	return utf16_decode_bytes(&raw_text, endianness)
	.map_err(|_| Id3v2Error::new(Id3v2ErrorKind::BadSyncText).into());
	}
	}
	let decoded_text = decode_text(
	&mut cursor,
	TextDecodeOptions::new().encoding(encoding).terminated(true),
	)
	.map_err(|_| Id3v2Error::new(Id3v2ErrorKind::BadSyncText))?;
	pos += decoded_text.bytes_read as u64;
	Ok(decoded_text.content)
	})()?;
	let time = cursor
	.read_u32::<BigEndian>()
	.map_err(|_| Id3v2Error::new(Id3v2ErrorKind::BadSyncText))?;
	pos += 4;
	content.push((time, text));
	}

Just need to handle it in the same way.

No idea why some software chooses to encode strings like this... really annoying.

[loxoron218]

Thanks for your work! :D