Refactor scanner for a more context-rich token stream

The lexer and token definitions were updated to produce a more
context-rich token stream. These changes were necessary to support
the advanced error recovery and contextual parsing capabilities
of the new recursive descent parser.

This enhancement lays the foundational groundwork for the subsequent
parser implementation by providing it with a more robust and
informative stream of tokens to consume.

Author: mkpro118