Make the lexer more robust to errors and invalid input

The previous lexer implementation was brittle. It would halt at the
first sign of an error, preventing the parser from identifying other
issues in the file. It was also too permissive with invalid number
formats and inefficient when parsing certain string patterns.

This patch hardens the lexer by introducing an optional recovery mode
to gracefully handle errors and continue tokenizing, which allows for
more complete diagnostics in a single pass.

Validation for numeric literals is now stricter, and string parsing
has been made more performant.

Author: mkpro118