Enhancement Request: Rules affected by semantic actions

[divyekapoor]

Hi,
This a feature request. Feel free to close / move around as you see fit. If you can point me towards how to implement this cleanly, I'm happy to try and improve PEGTL (it's been a great help for me).

Context

I'm trying to build a parser for extracting URLs in context (eg. given free / structured text, extract all the URLs within it). Repository: https://github.com/divyekapoor/grepurls. The code in there is an enhancement of the contrib/uri.hh implementation.

While specifying the grammar for this, one of the issues to be resolved is the handling of contextual delimiters:

eg.

print('mailto:[email protected]')

should be parsed as:

mailto:[email protected]

instead of:

mailto:[email protected]')

because single quote and closing parens are allowed within the URI.

To enable this, I have to change a rule called sub_delims in response to a context:

struct sub_delims : one< '!', '$', '&', '\'', '(', ')', '*', '+', ',', ';', '=' > {};

When within a single quoted context, sub_delims should be changed to sub_delims_without_single_quote, within double quotes it should be sub_delims_without_double_quotes or sub_delims_with_escaped_double_quote. There are similar extensions to parenthesis etc.

Issue

In a simple recursive descent parser, I would put the allowed delimiters in a vector and when switching to a single-quoted, double-quoted or parenthesized context, I would simply remove the appropriate delimiters from sub_delims and when returning from that context, I would add the delimiters back. In PEGTL however, this doesn't seem to be possible. To implement this, it seems that I would have to create separate rules for each of the contextual delimiters and then fork all the dependent rules to reference the appropriate rules in context. This is an N x M blowup in the number of rules required in the grammar (N = number of delims; M = number of dependent rules). I've looked through the code and wiki and it seems that the closest analogs are change_action and change_state but I haven't found change_rule. Is there a way to inform the parse by altering parse rules from the semantic actions? Have I missed something in the documentation instead? Is there a better way? If not, would you consider adding this to PEGTL?

Thanks & I appreciate all the hard work that has gone into PEGTL.

Cheers!
Divye

[ColinH]

Hi Divye,

thanks for writing in such a clear way that I don't have to guess what you are trying to achieve and what the issues are.

Now regarding the issue, the short answer is: It's possible, but requires the implementation of custom parsing rules:

• Add the vector of delimiters to the states, the additional arguments to a parsing run that are passed around to all parsing rules and actions; works either as an additional argument or as an additional data member of an existing one (or for initial testing a global variable).
• Write a custom parsing rule that uses the vector of delimiters from the states to determine which characters it matches; grep for static bool match in the examples directory for some examples of how to implement custom rules, also documentation page.
• You can then use either more custom rules or some actions to change the vector of delimiters whenever necessary.

This is just the short run-through, let us know if you need some more information... You can also write us at pegtl(at)colin-hirsch.net .

Best regards, Colin

[divyekapoor]

Thank you for an excellent response. I got the hang of it.