Compiled regex exceeds size limit

[thomas-zahner]

The latest CI error is too much for me to fix. It appears to come from lychee, and it appears to happen whenever you have a very big lycheeignore.

$  wc -l .lycheeignore    
16816 .lycheeignore
$ cargo run -- ads.com   
Error: Compiled regex exceeds size limit of 10485760 bytes.

Originally posted by @katrinafyi in lycheeverse/lycheeverse.github.io#111 (comment)

[katrinafyi]

I should add that the regex size limit isn't a problem per se. Rather, there just needs to be some way of ignoring a very large list of URLs. At the moment, the lychee ignore list is compiled into a big regex which hits the size limit, but a hashset would probably be just fine.

Thanks for filling the bug!

[thomas-zahner]

No problem! See df44f57 for my experiment. This limit is only hit with quite big files. In my test it was around 5500-6000 exclusions. I have tested a fast and hacky workaround by increasing the size limit.

I currently see these options:

• Close this issue as won't do. Instead change the approach we use in the docs in CI.
• Use the workaround from my commit.
• Don't use regex for excludes as you suggested.

Removing the regex functionality from the excludes IMO is not viable. This would make the feature less powerful and be a breaking change. Also I the current automatic limit by regex sounds sensible to me. So personally I'm in favour of closing this issue as won't do and change the approach used in CI.

[katrinafyi]

I imagined that the set-based exclude would be in addition to the regex based exclude, with the set only being used for literal file paths.

I think this is a useful thing to think about, even independently of this bug, because lychee uses a lot of regex but when you don't need regex's power, this necessitates a lot of escaping. The user will either remember to escape, then they have to type github\.com, or (worse) they will forget to escape and it will accidentally work because . matches anything, but it will match more than their intention.

In fact, the CI pipeline's use of lycheeignore falls into the second case. It dumps URLs which are not escaped then get treated as regex.

Anyway, I agree with changing the CI pipeline, but I think there is something to think about here with regex vs literal paths (maybe in a new issue tho).

[thomas-zahner]

True, it's something we could think about. But I'm not quite sure what a good approach would be.

I've worked around the error in my PR and simplified the pipeline, see this comment.

I'm closing this issue now. But feel free to open a new one and reference this issue.

[mre]

I like the idea to add support for file paths, but how would that work in practice?
If the .lycheeignore contains an entry like

file:///some/file.txt

That could theoretically be meant as a regular expression, or? And on top of that, there are many edge cases.

We could think of some "raw" marker before an entry, but that might not be very intuitive either.

Sorry, am I missing something here?

[katrinafyi]

Yeah, it is fundamentally ambiguous. It will need either a marker or a separate ignore file entirely.