reindexObjectSecurity patch causes ALL indexes to be computed recursively

[lukasgraf]

In b06bfc2 we monkey patch reindexObjectSecurity to make sure the relevant security indexes get updated in Solr as well.

This is done by adding REINDEX operations with attributes=self._cmf_security_indexes to the collective.indexing queue for the affected objects (recursively, just like the original method does).

While collective.indexing provides the concept of the indexing queue, the actual processing of the queue happens in collective.solr (by implementing a IIndexQueueProcessor utility).

The index() method of that SolrIndexProcessor takes an attributes kwarg, the only use it makes of it is to check if the object happens to have no fields that are indexed in Solr (in which case it skips the operation).

The issue that Solr can't properly do partial updates has been recognized, according to this comment:

            # unfortunately with current versions of solr we need to provide
            # data for _all_ fields during an <add> -- partial updates aren't
            # supported (see https://issues.apache.org/jira/browse/SOLR-139)
            # however, the reindexing can be skipped if none of the given
            # attributes match existing solr indexes... 

But the way they chose to deal with it is to call all the indexers for all the fields on the object that are part of Solr's schema.

Until Solr properly supports partial updates (for both XML and JSON update messages) we need to supply all the fields in every update message, there's no way around that.

But instead of letting Plone index all the fields again, one could get them from Solr (assuming they are stored).

So this is the fix I would propose:

• Override the SolrIndexProcessor utility in ftw.solr, inheriting from the original
• Override the index() method on the subclass, making sure the attributes get passed through to the self.getData() call
• Override the getData() method and change it do to the following:
  • If attributes is not None AND attributes doesn't contain all the fields in the Solr schema, get the remaining fields by querying Solr.
  • If a Solr query fails (e.g. because the field isn't stored), fall back on calling the indexer in Plone for those fields

[lukasgraf]

/cc @jone @buchi

[buchi]

I would favour using Solr's partial update feature now.

The solution would still be to override the SolrIndexProcessor utility.
If partial updates still do not work with Solr 4.3 or later using the XML-Interface we should push this bug to be fixed.
Is there already an issue about this bug in the Solr Bug Tracker?
If we cannot get the XML-Interface fixed, we have to do the partial updates using the JSON-Interface.

[lukasgraf]

One issue I reported has already been fixed: SOLR-4127 (follow up to SOLR-139.

There's still one issue with partial updates using the XML interface though: Update messages containing multi-valued fields don't get parsed correctly, causing corruption of the contents of the respective fields.

I will therefore reduce the bug to a simple test case and file an issue with Solr.