WIP tokenizer

[erlend-aasland]

No description provided.

[erlend-aasland]

cc. @serhiy-storchaka: Instead of polluting the CPython PR list, I chose to experiment a little bit more over here first. I gave the tokenizer approach a new chance, but this time using the example from the re docs as a template.

Pros:

• using pattern matching on the generated tokens is very readable (IMO)
• using a tokenizer makes it easy to further improve error messages

Cons:

• it may be a little bit heavyweight to use a tokenizer for AC syntax; OTOH, the tokenizer is dead simple and just a handful of lines
• using regexes directly is a handful of lines shorter (but IMO also less readable)

[erlend-aasland]

Also, it could be nice to be able to reuse the tokenizer to refactor the parameter parsing.

[erlend-aasland]

FTR, I also think the regex approach is nice; it is a step forward from what we have to day, anyway.