Tokenizer (cpp)

[hoosierEE]

• read input text line by line
• keep track of nesting brackets and strings
• allow strings and () [] {} to span multiple lines
• correctly tokenize all lexemes in the language
• error if input contains non-language characters
• allow unicode, at least in strings
• exit early on error if reading a file or string from the command line
• otherwise if there's an error, display the error message but don't crash