search does not display results, despite seemingly finding them

[code-consensus]

What happened?

v3.0.1 introduced FTS5 search syntax and v3.0.18 made search diacritic "agnostic" -- but the issue is that something must have happened since then and now, because on v3.3.8 these don't seem to be working any longer (I am a new user, and the only version I've ever used is this latest version).

Steps to reproduce the problem

Diacritics

from v3.0.18 release notes: "Previously, if you searched for cafe, it wouldn't return notes that contain café. Now, it will return notes that contain either cafe or café" -- the search doesn't behave this way any longer:

1. I go to a note where there is such content, e.g. the word también or España.
2. I do Ctrl+F to open the note-only search (not the global database search)
3. I type tambi and it finds it and highlights it.
4. I continue typing tambie and the highlight disappears and it says 0/0 found.
5. I erase the e and do tambié, the term is found and highlighted.

So something must have broken between v3.0.18 and v3.3.8 to make the "diacritic free" search no longer function. Note that another user reported this related to imported notes (issue #9227), except I don't think this has anything to do with only imported notes, because I am seeing this in also completely brand new notes.

Global search finding, but not displaying, results

There are certain search letters, terms, phrases, that seem are being found properly, but not displayed to the user.

In global search, if nothing is found the program informs you of this in one way or another.
e.g. if I search for something that is definitively not in my notes, e.g. zzzzzzffff:

• On mobile (v3.3.12 f-droid) you get a specific message on what you searched: "TIP No results found for zzzzzzffff".
• On desktop (v3.3.8), I get "TIP Wrap a query in double quotes to search for an exact match."

Going now to my example of "también" above, except this time, don't search in the note, but search in the "global database search" bar (however you call it):

1. I go to the global search, or type Ctrl+F+F
2. I enter the term I know is in my database, here I say again también. The search will show the note in the list that has that word.
3. I enter now instead tambifn, a slight variation that I know does not exist in the database, and I get the "tip: no results found for tambifn"
4. I enter now tambien (the variant without the diacrtic) -- now the search list is empty showing no notes, but there is NOT the message that no results were found. It seems as if the app is finding the term, but somehow just not displaying the search results.

I have tried this on many different kind of terms that I know are in the database, that when I do the individual-note internal search they do get found inside the note, but when you search the same thing at the "global search" level they are seemingly found and not displayed.

Examples of this are with certain special characters, e.g. I have note with the following term: Path \ here

1. Inside the note, if I search in the note-only search (Ctrl+F) for Path \, the program finds it.
2. I go to the global search (Ctrl+F+F). As per FTS5 search syntax, I wrap the term in double-quotes and here I search for "Path " (note no backslash) -- and the note is found.
3. I now search "Path \\" (note two backslashes, a phrase I know does not exist), and I get the "tip: No results found for "Path \\"".
4. I now search for "Path \", and I get that that empty search list, but without the message that no results were found. For reference, same behavior searching for the full term above "Path \ here"

This happens not just with backslash, but with many special characters (though not all), for example forward slash "/", "[" (but not "]"). Whereas regular parenthesis "(" and ")" work fine, among others. I haven't done an exhaustive search of every special character.

I also have managed to crash the program when searching for "[" (without or with quotes) or something with a word and "[", e.g. "[deleted":

SyntaxError: Invalid regular expression: /<nn-search-result\s+id="(.+?)">[<\/nn-search-result>/i: Unterminated character class
SyntaxError: Invalid regular expression: /<nn-search-result\s+id="(.+?)">[<\/nn-search-result>/i: Unterminated character class
at new RegExp (<anonymous>)
    at https://app.notesnook.com/assets/root-<twelve-char string>.js:238:71674
    at Array.every (<anonymous>)
    at D3 (https://app.notesnook.com/assets/root-<twelve-char string>.js:238:71665)
    at JS.<anonymous> (https://app.notesnook.com/assets/root-<twelve-char string>.js:238:62924)
    at Generator.next (<anonymous>)
    at u (https://app.notesnook.com/assets/root-<twelve-char string>.js:238:58012)

[I don't know if the root-xxxxx.js is personally identifiable information, so I didn't include it, but let me know]

On the topic of FTS search syntax, I also noticed that the escape sequence doesn't seem to translate, for example if I have a phrase like this "phrase and I want to search for it, if I try via the the escape for double quote"this ""phrase", it doesn't result in a hit .

non-breaking space - also is "hidden" in the global search - maybe treat like a diacritic?

This is a two-part observation.

Some of my imported notes had non-breaking spaces. I never intentionally added such spaces, but they were somehow there. Maybe copy-pasting from another document, don't know!

But the end-result is that when searching, I can't find certain things anymore because there was an unknown breaking space. e.g. A phrase in a note like a good dog, with the space between "good" and "dog" being a non-breaking space (that was not ever realized to be one). So then if I search in the note-only search for a good dog with just normal spaces, it will not have a hit. If I search with an actual non-breaking space between "good" and "dog" (I copy-paste it from Windows Character Map), again in the note-only search, then it is found.

On the global database search there is a different behavior, which is interesting:
-searching for "a good dog" (with quotes), with regular spaces, the program will not find it (will bring up the "tip not found" message).
-searching for "a good dog" with the space between "good" and "dog" being an actual non-breaking space, I get the result as described above for special characters where the program seemingly is finding it but not showing it (the "tip not found" message is not shown, but rather just an empty search list).
-If I add a g to the same search with the actual nbsp, i.e. "a good dogg" (a phrase that doesn't exist), I get the "tip not found" message, i.e. actually not finding anything
-but if I use the HTML escape characters, it works! "a good dog" does find it and display it in the list!

I would imagine most users wouldn't care about non-breaking spaces where searching text (non-technical users wouldn't likely even know that a non-breaking space even exists as a thing). At least me, if I am searching for something with a space (a good dog), I am wanting to find both cases of it with a normal or a non-breaking space.
I think there could be some utility to searching specifically for a non-breaking space (e.g. if you want to remove them from your notes!) -- but maybe that can be relegated to a more advanced search (e.g. searching via regex).

---

I hope this information is useful to correct and refine the search function. Please let me know if I can provide any further information.

Version

v3.3.8

Platform/OS

Windows

Relevant log output

SyntaxError: Invalid regular expression: /<nn-search-result\s+id="(.+?)">[<\/nn-search-result>/i: Unterminated character class
SyntaxError: Invalid regular expression: /<nn-search-result\s+id="(.+?)">[<\/nn-search-result>/i: Unterminated character class
at new RegExp (<anonymous>)
    at https://app.notesnook.com/assets/root-<twelve-char string>.js:238:71674
    at Array.every (<anonymous>)
    at D3 (https://app.notesnook.com/assets/root-<twelve-char string>.js:238:71665)
    at JS.<anonymous> (https://app.notesnook.com/assets/root-<twelve-char string>.js:238:62924)
    at Generator.next (<anonymous>)
    at u (https://app.notesnook.com/assets/root-<twelve-char string>.js:238:58012)

[makeplane]

Synced with Plane Workspace 🔄

[NN-4] search does not display results, despite seemingly finding them

[code-consensus]

On the topic of searching, I would also add one small "issue" searching for attachments in the Attachment Manager:

• if a file is called "file search test.jpg" if you type "file search", the program will find it
• if the file is called "file_search_test.jpg" and you search "file search", it will not be found

While technically "file search" literally is not "file_search", I think that type of matching would make sense.

[thecodrr]

@code-consensus kindly open a separate issue for that.