Export an array of all tokens from ct_token_map

[taminomara]

This helps with writing structured input adapters for fuzzing. When fuzzing a parser specifically (as opposed to fuzzing lexer and parser at the same time), we'd like to supply it with an array of valid lexemes. This export helps us build such an array as we don't have to manually list all tokens in a fuzzing entry point.

Note that I didn't implement this functionality for generated lexers because there's already a way to get all tokens via mod_l::lexerdef().iter_rules().

Example of a fuzzing implementation after this PR:

#[derive(Debug)]
struct Token(u32, String);

impl<'a> Arbitrary<'a> for Token {
    fn arbitrary(u: &mut Unstructured<'a>) -> libfuzzer_sys::arbitrary::Result<Self> {
        Ok(Token(*u.choose(token_map::TOKENS)?, u.arbitrary()?))
    }
}

fuzz_target!(|data: Vec<Token>| {
    let mut text = String::new();
    let lexemes = data.into_iter().map(|tok| {
        let lexeme = DefaultLexeme::new(
            tok.0,
            text.len(),
            tok.1.len(),
        );
        text.push_str(&tok.1);
        lexeme
    }).collect();

    // Run parser...
}

[ltratt]

This is a part of the system I haven't thought about for a while. Is it possible to do the same thing with mod_l::lexerdef().iter_rules().map(|x| x.name()).collect() or similar? [Warning: untried!]

[taminomara]

Is it possible to do the same thing with mod_l::lexerdef().iter_rules().map(|x| x.name()).collect() or similar?

Yes, this seems to work.

[ltratt]

OK, then I think we don't need to generate the array?

[taminomara]

OK, then I think we don't need to generate the array?

That will only work when user has a generated lexer. If there's a custom lexer with ct_token_map, then there's no way to get a full array of tokens.

[ltratt]

If there's a custom lexer with ct_token_map, then there's no way to get a full array of tokens.

I take your point.

[ratmice]

Took a bit of head scratching until I grokked it (building a token stream directly rather than an intermediate vector!),
But once it clicked, it all seemed fine to me.

Seems fine to me now, unless Laurence has any further comments.

[ltratt]

@ratmice Thanks for the review!

@taminomara Thanks for the PR!