Indexing nodes with job:work (Doctrine) only applies first changes to a node: remaining are lost

[baschny]

Hi,

we just upgraded a site from Neos 4.x to Flow/Neos 7.x and were indexing nodes using the DoctrineQueue with:

./flow job:work Flowpack.ElasticSearch.ContentRepositoryQueueIndexer.Live --verbose

We see the indexing is being done when we edit nodes in the backend, and also indexed is triggered when we later publish the same node.

The problem: only the first change is stored in ES, all further changes to the node (i.e. the "publish") are not being reflected into the ES index. Just try it yourself, change the title of a node from "node1" to "node2", publish that, and then change it to "node3" etc and see what is really written in the ES.

Could you reproduce that, is it a bug? It worked before with the same setup.

It looks like there is some caching going on, and the problem only happens in the endless PHP runner (job:work without a --limit). Our workaround is thus currently:

while true; do ./flow job:work Flowpack.ElasticSearch.ContentRepositoryQueueIndexer.Live --limit 1 ; done

so that every indexing starts a fresh new PHP process.

[cvette]

Can confirm this is still an issue.

Seems to be caused by this:
https://github.com/neos/neos-development-collection/blob/6af1939bb5dac41e58475f848cb90e4b82493e87/Neos.ContentRepository/Classes/Domain/Factory/NodeFactory.php#L60

This factory has a reset() method. Maybe we could call that for each indexed document.

[gradinarufelix]

@cvette @baschny I also ran into that problem right now. I have an idea of a solution and wanted to ask what you think of it.

So NodeData has a version property. What if we add a few lines in NodeFactory that checks if the cached Node respectively its NodeData has the same version or not. If not, then we disregard the cached Node and create a new one. What do you think of that?

Edit: dang, the version is not exposed as a property :-(