Use a more sane tokenizer for source code search

[bsofiato]

Feature Description

As of today, the elastic search search uses the default analizer when indexing the source code contents. This implementation uses whitespaces to break the tokens.

I feel this approach is not particularly suitable for source code search. To illustrate the issue, let us consider the code snippet below:

public baz(Foo foo) {
   return foo.bar();
}

It is fair to think that searching for bar returns the code above. As of today, however, this is not the case: ES will assume that foo.bar() is a single token. As such, ES will not match the criterion bar.

I suggest we use the pattern tokenizer instead. It uses regular expressions to separate tokens. By default, it uses any (non-word character as a token separator). In such a case, the snippet foo.bar() would yield two tokens -- foo and bar (the second token will match the given criterion).

What do you guys think?

Screenshots

No response

[bsofiato]

Hey guys, an update on this issue.

At my workplace, we run a Gitea instance with about 3K repositories. Our L2 and L3 support teams (about 200 people) rely heavily on the code search feature on Gitea. They asked me if I could make the code search case insensitive.

I was thinking of allowing this kind of search. In such a case, the results would have less relevance than matches on with the cases match.

What do you guys think? Do you think it is worthwhile to have this in Gitea's main line? If you guys are cool with it, I'll change the PR #32261 to handle it as well.

[lunny]

I think currently it's already Case insensitive? At least for bleve engine. Maybe you mean Case Sensitive? If that, maybe we can have an option or filter for that. Looks like Github code search doesn't support case sensitive search.

[bsofiato]

Yeah, as a matter of fact, in bleve's case it is indeed already case insensitive (as the screenshot below shows).

However, the ES backend the content field is not normalized. So for ES (which is our case) it is indeed case sensitive (as shown below).

If it is ok with you guys, I'll update the #32261 to make the ES content field case insensitive like bleve's. What do you think ?

P.S. Another option would be to create another PR just for this fix. But I think that rebuilding the index might be too much for a patch version.

[bsofiato]

@lunny I've pushed some changes to #32261 to make ES' search case insensitive :)