Solr queries not working due to local parameters not being allowed since CKAN 2.10.4/2.9.11

[mjanez]

Environment

• CKAN 2.9.11 (custom ckan-docker)
  • Custom ckan-solr with spatial config for solr-bbox and solr-spatial-field backends.
  • ckanext-spatial

Problem:

• Spatial searches with the Solr backend solr-bbox and solr-bbox using the {!frange} local parameter.
• Local parameters in Solr currently do not work in ckan-2.9.11 due to security restrictions disallowing local parameters.
• The error message encountered are:

solr-bbox log:

2024-04-12 13:26:32,813 INFO  [ckan.lib.search.query] SearchError query: {'facet.field': ['theme', 'groups', 'theme_es', 'dcat_type', 'publisher_name', 'publisher_type', 'spatial_uri', 'owner_org', 'res_format', 'frequency', 'tags', 'tag_uri', 'conforms_to'], 'fq': ['  -dataset_type:harvest', '{!frange incl=false l=0 u=1}div(mul(mul(max(0,sub(min(7.910156250000001,maxx),max(-29.355468750000004,minx))),max(0,sub(min(45.91485503041036,maxy),max(23.193387206172922,miny)))),2),add(846.7296993875983,mul(sub(maxy,miny),sub(maxx,minx))))', '+site_id:"default"', '+state:active'], 'q': '*:*', 'rows': 21, 'sort': 'score desc, metadata_modified desc', 'start': 0, 'df': 'text', 'fq_list': ['{!frange incl=false l=0 u=1}div(mul(mul(max(0,sub(min(7.910156250000001,maxx),max(-29.355468750000004,minx))),max(0,sub(min(45.91485503041036,maxy),max(23.193387206172922,miny)))),2),add(846.7296993875983,mul(sub(maxy,miny),sub(maxx,minx))))'], 'bf': 'div(mul(mul(max(0,sub(min(7.910156250000001,maxx),max(-29.355468750000004,minx))),max(0,sub(min(45.91485503041036,maxy),max(23.193387206172922,miny)))),2),add(846.7296993875983,mul(sub(maxy,miny),sub(maxx,minx))))', 'defType': 'edismax', 'fl': 'id validated_data_dict', 'facet': 'true', 'facet.limit': '50', 'facet.mincount': 1, 'wt': 'json', 'tie': '0.1', 'mm': '2<-1 5<80%', 'qf': 'name^4 title^4 tags^2 groups^2 text', 'q.op': 'AND'}
2024-04-12 13:26:32,813 ERROR [ckan.views.dataset] Dataset search error: ("Local parameters are not supported in param 'fq'. value: '{!frange incl=false l=0 u=1}div(mul(mul(max(0,sub(min(7.910156250000001,maxx),max(-29.355468750000004,minx))),max(0,sub(min(45.91485503041036,maxy),max(23.193387206172922,miny)))),2),add(846.7296993875983,mul(sub(maxy,miny),sub(maxx,minx))))'",)

solr-spatial-field log:

2024-04-12 13:05:20,561 INFO  [ckan.lib.search.query] SearchError query: {'facet.field': ['theme', 'groups', 'theme_es', 'dcat_type', 'publisher_name', 'publisher_type', 'spatial_uri', 'owner_org', 'res_format', 'frequency', 'tags', 'tag_uri', 'conforms_to'], 'fq': ['  -dataset_type:harvest', '{!field f=spatial_geom}Intersects(ENVELOPE(-29.355468750000004, 7.910156250000001, 45.91485503041036, 23.193387206172922))', '+site_id:"default"', '+state:active'], 'q': '*:*', 'rows': 21, 'sort': 'score desc, metadata_modified desc', 'start': 0, 'df': 'text', 'fq_list': ['{!field f=spatial_geom}Intersects(ENVELOPE(-29.355468750000004, 7.910156250000001, 45.91485503041036, 23.193387206172922))'], 'fl': 'id validated_data_dict', 'facet': 'true', 'facet.limit': '50', 'facet.mincount': 1, 'wt': 'json', 'q.op': 'AND'}
2024-04-12 13:05:20,561 ERROR [ckan.views.dataset] Dataset search error: ("Local parameters are not supported in param 'fq'. value: '{!field f=spatial_geom}Intersects(ENVELOPE(-29.355468750000004, 7.910156250000001, 45.91485503041036, 23.193387206172922))'",)

Reference the solution:

CKAN pull request ckan/ckan#8053 introduces a new configuration option ckan.search.solr_allowed_query_parsers which allows to specify whitelisted local parameters. If the search parameters are not included, this will cause some problems in ckanext-spatial like {!frange}:

ckanext-spatial/ckanext/spatial/search/__init__.py

Lines 123 to 124 in 9383084

	search_params["fq_list"] = search_params.get("fq_list", [])
	search_params["fq_list"].append("{!frange incl=false l=0 u=1}%s" % bf)

or {!field}:

ckanext-spatial/ckanext/spatial/search/__init__.py

Lines 202 to 211 in 9383084

	if not search_params.get("fq_list"):
	search_params["fq_list"] = []
	default_spatial_query = "{{!field f=spatial_geom}}Intersects(ENVELOPE({minx}, {maxx}, {maxy}, {miny}))"
	spatial_query = config.get("ckanext.spatial.solr_query", default_spatial_query)
	search_params["fq_list"].append(
	spatial_query.format(spatial_field="spatial_geom", **bbox)
	)

Proposed fix:

Updated the doc to add params to the ckan.search.solr_allowed_query_parsers list in the ckan.ini configuration file when setting up spatial search.

Setup
-----


To enable the spatial search you need to add the ``spatial_query`` plugin to
your ini file. This plugin in turn requires the ``spatial_metadata`` plugin, eg::


  ckan.plugins = ... spatial_metadata spatial_query


To define which backend to use for the spatial search use the following
configuration option (see `Choosing a backend for the spatial search`_)::


  ckanext.spatial.search_backend = solr-bbox


For CKAN >2.10.4 and >2.9.11 to avoid errors in the spatial search, use the following configuration elements. For `solr-bbox` use `frange`, for `solr-spatial-field` use `field`::


  ckan.search.solr_allowed_query_parsers = ... frange | field