Implement Validation for Non-Queryable Attributes in Filtering

[bountx]

Implement validation to reject filter or search queries that use attributes not defined in the collections list of queryable fields, thus reducing processing time on invalid queries.

Solution proposition

• Helper script should be introduced to aggregate and maintain a local cache of all queryable attributes across all collections, updating this cache automatically on an hourly basis.

• When a filter or search request is received, the system will check the requested attributes against this local, up-to-date list. If an attribute is missing from the local cache, the request will be rejected with an error message explicitly stating which attribute is undefined or non-queryable, ensuring faster validation and preventing errors due to stale or incorrect attribute names.

[jonhealy1]

Hi @bountx. This is a really interesting idea. This would need to be optional, as other people using this project may not see a use for this. I see maybe a config file that would let people know if they are searching on an attribute that's not available. Then on your end you could update this file whenever you want.

As far as the cache goes, do you see the list of available attributes changing frequently over time? Even if new data changed attributes, old data would still be searchable with the old attributes?

[bountx]

Hey @jonhealy1. I totally agree about the feature being optional. Regarding your questions:

The attributes change usually when a new collection is added, but it is not guaranteed - other changes may happen (very rarely). I thought about also putting frequency of pulling available queryables in the config file.

If new data changed attributes and there's still old data with old attributes somewhere they'd probably have to be in different collections - but yes they would still be searchable with the old attributes. Probably mostly because of how list of queryables already works than my changes though. (Please correct me if I'm wrong)

[jonhealy1]

@bountx Totally makes sense I think.