Unexpected parsing errors on unquoted strings

[andjsrk]

Description

toon-rust/src/decode/parser.rs

Lines 1125 to 1175 in c22f0ae

	Token::Integer(i) => {
	let val = *i;
	self.advance()?;
	// If followed by string tokens, treat the whole value as a string
	if let Token::String(..) = &self.current_token {
	let mut accumulated = val.to_string();
	while let Token::String(next, _) = &self.current_token {
	accumulated.push(' ');
	accumulated.push_str(next);
	self.advance()?;
	}
	Ok(Value::String(accumulated))
	} else {
	Ok(Number::from(val).into())
	}
	}
	Token::Number(n) => {
	let val = *n;
	self.advance()?;
	// If followed by string tokens, treat the whole value as a string
	if let Token::String(..) = &self.current_token {
	let mut accumulated = val.to_string();
	while let Token::String(next, _) = &self.current_token {
	accumulated.push(' ');
	accumulated.push_str(next);
	self.advance()?;
	}
	Ok(Value::String(accumulated))
	} else if val.is_finite() && val.fract() == 0.0 && val.abs() <= i64::MAX as f64 {
	Ok(Number::from(val as i64).into())
	} else {
	Ok(Number::from_f64(val)
	.ok_or_else(|| ToonError::InvalidInput(format!("Invalid number: {val}")))?
	.into())
	}
	}
	Token::String(s, _) => {
	// Tabular fields can have multiple string tokens joined with spaces
	let mut accumulated = s.clone();
	self.advance()?;
	while let Token::String(next, _) = &self.current_token {
	if !accumulated.is_empty() {
	accumulated.push(' ');
	}
	accumulated.push_str(next);
	self.advance()?;
	}
	Ok(Value::String(accumulated))
	}

Currently, the parser confuses "tokens that can be part of unquoted strings" and "string tokens", leading to weird parsing error.

Reproduction Steps

1. Call the decoder with:
   Case 1:

   arr[1]{a}:
     1 null
   

   Case 2:

   arr[1]{a}:
     a 1
   

2. See error.

Expected Behavior

Case 1: Successfully parsed to { "arr": [{ "a": "1 null" }] } (JSON for convenience)
Case 2: Successfully parsed to { "arr": [{ "a": "a 1" }] } (JSON for convenience)

Actual Behavior

Case 1: Parse error at line 2, column 8: Expected newline after tabular row 1
Case 2: Parse error at line 2, column 5: Expected newline after tabular row 1

Environment

• Package version: 0.4.4
• Runtime: Rust 1.96.0 (nightly)
• OS: Windows 11

This bug is probably environment independent. (I have checked the source code)

Additional Context

No response

[andjsrk]

I'm digging into the source code and this is not only for tabular array field values:

Root Level

Call the decoder with:

a 1

Unexpectedly errors: Parse error at line 1, column 4: Multiple values at root level are not allowed in strict mode

Note that a 1 is generated from the encoder in this crate, and the spec does not explicitly ban unquoted strings on root level.

[shreyasbhat0]

Fixed in #62. The parser now handles all value token types (not just strings) when concatenating multi-token values. 1 null and a 1 now correctly parse as strings in tabular rows, field values, and at root level — per spec §B.4 which requires type inference on the complete text.