Search should find only matching items

[ildar170975]

Initially posted here:
https://community.home-assistant.io/t/2022-5-streamlining-settings/418477/108?u=ildar_gabdullin
https://community.home-assistant.io/t/2022-5-streamlining-settings/418477/128?u=ildar_gabdullin
https://community.home-assistant.io/t/2022-5-streamlining-settings/418477/136?u=ildar_gabdullin
https://community.home-assistant.io/t/2022-5-streamlining-settings/418477/138?u=ildar_gabdullin

Earlier (at least in 2022.4) the search feature worked as per this method:
a) the user specifies a substring -> the outputted list contains only rows with matching items (i.e. containing the substring);
b) the more precise the substring is (i.e. more letters are specified) - the less number of rows is found.

Now (2022.5) the output results may contain not only exactly matching items but similar items too.
Moreover, sometimes number of matches may INCREASE while the more precise search string is specified.
See examples in the posts above.

Probably (just my guess) these changes were added with a new search feature.

[TheHolyRoger]

@farzher it would be great if you could take a look at this please, i see you're the developer of fuzzysort :)

farzher/fuzzysort#97

@vanackej is the one who implemented the library in this commit 8da73d4

Demo here of the "issue" with exact matches:

[vanackej]

The behavior I originally most wanted is not this one (it is just a side effect of using the same matcher as the quick bar)
The one I reallly wanted is the first one I implemented (then removed after PR review) : "batt foo" => All entities having "batt" and "foo" in it, in any order, with exact matching for each term.

omg. This is really broken without additional usability given

I'm sorry but I feel this is a pretty harsh statement. I can understand you don't see the usability improvement in your situation (and I agree this introduced some breaking changes and even some regressions regarding existing bahvior), but please consider the rationnals explained in the original PR and have some respect for the time and effort spent on trying to bring new features.

[TheHolyRoger]

@vanackej where's your buymeacoffee link please sir?

[vanackej]

Don't need one, I already have very good coffee at home and not looking for retribution, but thanks for asking ;-)
Sometimes trying to make good things, result is not always perfect, just expecting some kindness in comments...

[emufan]

pretty harsh statement.
Sorry, if you get it wrong. Don't expect, that not everyone is a native speaker here. I'm always open and of course everything I write is my personal feelings. If I like something, that I write it and if I think that it is broken (and for me it is), I write it as well. Don't search for harshness, if this was not meant that way. 💙

[vanackej]

@zsarnett Feedbacks on the new entity picker behavior seems globally negative.
I can, if you are ok with it:

• Remove fuzzy search from entity picker/filtering
• Just reintroduce the simple behavior I originally wanted to implement : If search terms are separated with spaces, ensure all terms are included in the id or name of the entity (which I think is a real improvement over the simple "contains" initial approach, with no breaking change, just a slight improvement)
• Optionnaly, add some simple wildcard mecanism like suggested by @emufan : nice to have, easy to implement, but not really mandatory
  Let me know what you think should be done, I'd be happy to propose a PR quickly.

[bschatzow]

What gets lost in the development is that different sections have different target users. People in general using developers tools, understand what they are looking for. Finding more options does not help ( answer is lost in the forest). Finding complete information is what is needed, I. E. Entity id etc. Maybe have two search boxes will answer to both groups?

[vanackej]

I don't know what to think, so many options/opinions here :-)

My simple initial intention, as an end user, is to make the entity search a little more functionnal for this simple (and I think very frequent) use case:

• I want to find one specific entity, let's say a temperature sensor located in roomX
• I have a lot of temperature sensors (so typing "temp" in search return too many results)
• I have a lot of entities in room X (so typing "roomX" in search also return too many results)
• I don't know in what order/the exact naming is, so I'm not able to find easily with the previous "includes" filtering

=> I just want to type "temp" to filter temperature sensors, and then "roomX" to filter on the location

Another example : I want one of my hvac units total consumption entity. Just want to type, in any order, the words "consumption roomName total" to get it.
The currently (pre 2022.5) entity filtering totally unable to achieve this kind of search (which are the ones I always need)

I'm not sure having advanced search mode/switching between search mode with checkboxes would be really that usefull, and it would add a lot of complexity in the user interface and underlying code

[farzher]

er really doesn't want to match to tracker. while tr.life360 would match perfectly.

er.life360 is a substring of device_tracker.life360_xxxxx.

yup. i'll definitely be fixing this. the first change to the algorithm in 5 years. but don't expect that anytime soon, and feel free to replace fuzzysort; and don't ask me how nobody's complained about this over the past 5 years. ¯\(ツ)/¯

[farzher]

don't ask me how nobody's complained about this over the past 5 years

Mmmm, I have was been using this search for last 2 months, it worked perfectly)))

oh sorry, i'm the dev of https://github.com/farzher/fuzzysort. this project just started using my search. i don't work here :)

[vanackej]

@farzher Thanks for having a look and the explaination about how the fuzzysort is intended to works

It confirms what I see in my tests :

Search for er.life360_iphone_6 in binary_sensor.battery_charging_life360_iphone_6,: -51028
Search for er.life360_iphone_6 in sensor.battery_life360_iphone_6,: -21012
Search for er.life360_iphone_6 in device_tracker.life360_iphone_6,: -23012

Search for tr.life360_iphone_6 in binary_sensor.battery_charging_life360_iphone_6,: No score
Search for tr.life360_iphone_6 in sensor.battery_life360_iphone_6,: No score
Search for tr.life360_iphone_6 in device_tracker.life360_iphone_6,: -33

When matching sequence is at the end of the word, thats not really that it is not "supported", but score is a lot lower (then it depends on the minimum accepted score to be considered as a match) => Results appears to be out of the logical order

A simple workaround (until @farzher maybe fix this in fuzzysort library) is to first test if the search term is included in the result before using fuzzysort :

Search for er.life360_iphone_6 in binary_sensor.battery_charging_life360_iphone_6,: -51028
Search for er.life360_iphone_6 in sensor.battery_life360_iphone_6,: -21012
Search for er.life360_iphone_6 in device_tracker.life360_iphone_6,: 0

[zsarnett]

Hi @farzher

Thanks for the insight into the Algorithm from FuzzySort! I do think we can continue to utilize this algorithm with a few tweaks to some of the scorings. This is possible using your currently extendable options.

I will discuss with @vanackej who created the original PRs for this and find a better method. I do think if the filter string is included in one of the strings in the list then we should return that to the top of the list.

So we can do something where we remove those from the fuzzy sort options, before filtering them using our own method or possibly updating the scoring option. I haven't fully dived into this yet but we will make the change for the next release for sure

Thanks to everyone who has been discussing this and giving feedback!

[zsarnett]

In a patch release tomorrow

Edit: Released in the .3 Patch

[vanackej]

@zsarnett do you want me to restore previous behavior (but keeping infrastructure change in order to allow latter improvements ?). Tell me if I can help on this

[zsarnett]

@vanackej if you want to try the fuzzy sort library again then we can see what we can customize about it

Or we can just update the current implementation from vscode and add in functionality

[bschatzow]

I do not see it fixed in 5.3. I just tried the drop down in "people" typed in gps, should have found 4 sensors, found none.

[zsarnett]

It's reverted for me in the .3 release. Options not showing is more of a case for the revert state than the non reverted state.

[MRobi1]

Just jumping in as I had opened another issue on this same topic here home-assistant/core#71435

I hope this can be fixed in an upcoming release because the search function as it is is barely usable.

[sonnstein]

absolutely agree! It is of no worth to have a search result, with 50% result, that has virtually no (some mystery) relation to the search string!