Regex Enhancements

[wrmthorne]

On a side note: I'll probably also add fancy_regex at some point to support e.g. look-around. It should perform similar for the current regular expressions but I want to test that first.
Originally posted by @uniQIndividual in #2 (comment)

I thought I would open an issue I had some thoughts to contribute if you're open to discussion.

As a quick feature proposal that I have been debating implementing is another config option to toggle the regex filter between include and exclude (one). Alternatively, the pattern could be separated out into an include and exclude pattern which would be flexible and intuitive (one or both). I'm unsure of the performance implications vs a single more complex regex.

To provide my own example of a need for look-arounds:
As it stands, I can't find a reliable way to filter out specific subreddits from the push shift archive with just zstd-jsonl-filter. The way they store crosspost information is by nesting the whole submission object under crosspost_parent_list as part of a list. Without lookarounds, I can't reliably tell whether I am fetching e.g. a submission from r/rust or whether the source of a crosspost was from r/rust. Thankfully this only leads to false-positives that are easily excluded in polars. There are, however, lots of them.

I have some other unrelated feature ideas/improvements - I can open a new issue if you'd prefer

[uniQIndividual]

Do I understand your first proposal correctly that you seek to add a flag invert-match (similar to ugrep) which only returns lines which do no match the regex term? That might be a great idea given how likely that use case is and I suspect it would be substantially faster than any regex term using look-arounds.

If we're going the route of more than a single regex term, perhaps it would make sense to allow a list of regex terms which are then all matched? (The idea being that writing them as a single expression would quickly become very large)
My thought would be something like the following:

pattern = ["term1", "?negativ_term2" , "term3"]

In this case the second term would (mis)use the ? to indicate a pattern which should be applied inverted (and it wouldn't be a valid regular expression). From what I can tell this would be similar to ugrep's "boolean query patterns" but so far I'm not entirely sure yet how they separate the query patterns e.g. | from the ones contained in regular expressions. But either could also be a potential direction.

For now I have created a new branch fancy_regex which supports look-arounds and more so you can also test how well it works for your use case. When the fancy regex support isn't needed it should fall back to the regex package I used previously, i.e. it should reach similar speeds.

btw. happily keep the suggestions coming but it's probably cleaner to open a new issue for those

[wrmthorne]

Yes, your understanding is correct.

I like the idea of decomposing long "or-chains". I've had these occur frequently when filtering for 10s of subreddits or users.

I'll challenge the choice of ? as the inverting character as it would overload ? even more and there is no precedent for its use as negation. While ! would be my typical association it is easily missed at a glance. - is semantically to remove something and is used elsewhere (e.g. Google search). Alternatively, I like ~ as it has no operator in rust or regex (to my knowledge) and typically represents bitwise not.

I do like the idea of the list syntax style. To resolve the operator issue entirely, we could take inspiration from the JSON-Schema syntax:

[pattern]
any_of = ["term1", "term2"]
all_of = ["term3", "term4"]
none_of = ["term5", "term6"]
# ... and the rest

It is a bit clearer semantically and potentially breaks up the single long list that we may be at risk of introducing. I do see some issues however; first, the config.toml is no longer a simple key-value map but more significant is the limited ability to express more complex boolean statements e.g.

term1 & term2 & ~term3 | (alt1 & ~alt2)

I have some ideas for how to reclaim this expressiveness but it comes at the expense of a more complex application of toml syntax which may not be desirable.