Allow '.' in symbol tokens

[ahornace]

Suggester suggests OpenGrok's package names as single tokens for search; however, the search will split the text by . into multiple tokens and the final result is empty.

Example:

http://demo.opengrok.org/search?project=OpenGrok&full=&defs=org.opengrok.indexer.history&refs=&path=&hist=&type=&xrd=&nn=1

This change fixes the problem as now the search is successful:

However, the xref is still wrong, e.g.: http://demo.opengrok.org/xref/OpenGrok/opengrok-indexer/src/main/java/org/opengrok/indexer/history/PerforceHistoryParser.java?r=51e20d51#26

I'm not sure if this can have an adverse effect on other things and if we should rather try to split the defs terms by . -> but then they do not represent the whole definition (in this instance the package).

@vladak, what do you think?

[vladak]

Makes me wonder what will happen with e.g. C structure members. I.e. if you have a structure:

struct mystruct {
    int foo;
    char bar;
};

and it gets used in the code like this:

struct mystruct my;
my.foo += my.bar;

will it be still possible to find references of foo and bar no matter what the variable name is ?

[tarzanek]

workaround is of course distance search
so look for "org opengrok indexer" not further than 3 away

xref has its own analyzer, hence it might need changes there too

[tarzanek]

if I'd have to vote, I'd vote against it ...
what would happen when you would have a sentence ending with "." and without space after it? (typo, I know)
would it still be able to search for parts?
I expect not (which answers Vlads Q above), since you'd have to use org.* to find the word, which would now be a full token
suggester however might show it as a partial hit, but full search will depend on "*"

[tarzanek]

OR the other option is to allow it only for certain languages that use such notation ... such as java ...
but I am not sure if there is any other language ... likely not

[ahornace]

workaround is of course distance search
so look for "org opengrok indexer" not further than 3 away

Unfortunately, no. As the whole term is stored, then the proposed workaround won't work. We can try to run the definitions found by ctags through the tokenizer again but that brings more problems and it's not completely correct.

[ahornace]

The idea here was that the C analyzer will take care of things as my.foo += my.bar; and split it accordingly into multiple terms separated by . (the . would stored as a token as well).

The purpose of this PR was that . would be treated fairly when doing a search as now it's completely discarded.