Using ElasticSearchProvider with PreProcessFieldDefinition

[Interritor]

Hello,

I have a little problem, I hope you can help me with that.

I have DocumentDefinition which I want to add a PreProcessFieldDefinition. But it got the error "Pre process field definitions are not allowed in current context (Maybe you are using a pre configured index provider) ".

I found that's because the ElasticSearchProvider is an PreConfiguredIndexProviderInterface.

Is there any way to setup PreProcessFieldDefinition using ElasticSearch as IndexProvider?

I hope you can help. I really need the option to work with PreProcessFieldDefinition.

Or is there any other option? I need way to add definitions based on the return value of an data object method?

I working with following versions (still at Pimcore 10):

dachcom-digital/dynamic-search v2.1.2
dachcom-digital/dynamic-search-data-provider-trinity v2.0.3
dachcom-digital/dynamic-search-index-provider-elasticsearch 2.2.0

Thanks in advance,

Andre

[solverat]

Hi @Interritor,

I'm afraid, but this is not possible within Elastic/OpenSearch.

Those engines force us to prebuild the structure table, before data gets indexed. And that's also the problem, why preprocessed fields can't work here: Data structure within a PreProcessFieldDefinition will be created if index data is available - which is too late!

However, we introduced this feature to allow filter-like indexing with lucene. With ES/OS - both much more powerful - we actually don't need to use PreProcessFieldDefinition.

The closest thing would be the transform_callback within the ObjectGetterExtractor data transformer:

->addSimpleDocumentFieldDefinition([
    'name'              => 'categories',
    'index_transformer' => [
        'type'          => 'explicit',
        'configuration' => [
            'type' => 'keyword',
        ]
    ],
    'data_transformer'  => [
        'type'          => ObjectGetterExtractor::class,
        'configuration' => [
            'method'             => 'getCategories',
            'transform_callback' => fn($categories) => array_map(static fn(
                CategoryInterface $c
            ) => $c->getName($normalizerOptions['locale']), $categories),
        ]
    ]
])

Or even more complex by using a nested field type:

->addSimpleDocumentFieldDefinition([
  'name'              => 'field_attributes',
  'index_transformer' => [
      'type'          => 'explicit',
      'configuration' => [
          'type'       => 'nested',
          'properties' => [
              'attribute_id'   => [
                  'type' => 'integer'
              ],
              'attribute_group_id'   => [
                  'type' => 'integer'
              ],
              'attribute_value' => [
                  'type' => 'keyword'
              ],
          ]
      ]
  ],
  'data_transformer'  => [
      'type'          => AttributesExtractor::class,
      'configuration' => ['locale' => $normalizerOptions['locale'] ?? null, 'range' => false]
  ]
])

This would allow you to create nested attributes without using any pre-processed field definitions.

Hope this helps.

[bajizs]

Hello,

with Pimcore 10.5 I must to write Definition class like:

<?php

namespace App\DynamicSearch\IndexDefinition\Trinity;

use DynamicSearchBundle\Document\Definition\DocumentDefinitionBuilderInterface;
use DynamicSearchBundle\Document\Definition\DocumentDefinitionInterface;
use DynamicSearchBundle\Normalizer\Resource\ResourceMetaInterface;

class Definition implements DocumentDefinitionBuilderInterface
{
    public function isApplicable(string $contextName, ResourceMetaInterface $resourceMeta): bool
    {
        if ($resourceMeta->getResourceCollectionType() !== 'object') {
            return false;
        }

        if ($resourceMeta->getResourceSubType() !== 'Article') {
            return false;
        }

        return true;
    }

    public function buildDefinition(DocumentDefinitionInterface $definition, array $normalizerOptions): DocumentDefinitionInterface
    {
        $definition
            ->addSimpleDocumentFieldDefinition([
                'name'              => 'pimcoreId',
                'index_transformer' => [
                    'type' => 'keyword',
                ],
                'data_transformer'  => [
                    'type'          => 'element_id_extractor'
                ]
            ])
            ->addSimpleDocumentFieldDefinition([
                'name'              => 'title',
                'index_transformer' => [
                    'type' => 'text',
                ],
                'data_transformer'  => [
                    'type'          => 'object_getter_extractor',
                    'configuration' => ['method' => 'getTitle']
                ]
            ])
            ->addSimpleDocumentFieldDefinition([
                'name'              => 'subtitle',
                'index_transformer' => [
                    'type' => 'text',
                ],
                'data_transformer'  => [
                    'type'          => 'object_getter_extractor',
                    'configuration' => ['method' => 'getSubtitle']
                ]
            ])
            ->addSimpleDocumentFieldDefinition([
                'name'              => 'supertitle',
                'index_transformer' => [
                    'type' => 'text',
                ],
                'data_transformer'  => [
                    'type'          => 'object_getter_extractor',
                    'configuration' => ['method' => 'getSuperTitle']
                ]
            ])
            ->addSimpleDocumentFieldDefinition([
                'name'              => 'content',
                'index_transformer' => [
                    'type' => 'text',
                ],
                'data_transformer'  => [
                    'type'          => 'object_getter_extractor',
                    'configuration' => ['method' => 'getDescription']
                ]
            ])
            ->addSimpleDocumentFieldDefinition([
                'name'              => 'fulltext',
                'index_transformer' => [
                    'type' => 'unStored',
                ],
                'data_transformer'  => [
                    'type'          => 'object_getter_extractor',
                    'configuration' => ['method' => 'getArticleTextualContent']
                ]
            ])
            ->addSimpleDocumentFieldDefinition([
                'name'              => 'publishdate',
                'index_transformer' => [
                    'type' => 'text',
                ],
                'data_transformer'  => [
                    'type'          => 'object_getter_extractor',
                    'configuration' => ['method' => 'getDateForSearch']
                ]
            ])
            ->addSimpleDocumentFieldDefinition([
                'name' => 'uri',
                'index_transformer' => [
                    'type' => 'unIndexed',
                ],
                'data_transformer' => [
                    'type' => 'object_path_generator',
                ],
            ]);

        return $definition;
    }
}

With this class I successfully generate index in ES. But query don't work on it. TO be Query work I must create custom search action and rewrite whole query to ES (drop anything what original bundle code generate):

<?php

namespace App\DynamicSearch\OutputChannel\Action\Search;

use DynamicSearchBundle\Event\OutputModifierEvent;
use DynamicSearchBundle\OutputChannel\Allocator\OutputChannelAllocatorInterface;
use DynamicSearchBundle\OutputChannel\Modifier\OutputChannelModifierActionInterface;
use Pimcore\Logger;

use Symfony\Component\HttpFoundation\RequestStack;


class MySearchAction implements OutputChannelModifierActionInterface
{
    public function __construct(
        protected RequestStack $requestStack,
    ) {
    }

    /**
     * {@inheritdoc}
     */
    public function dispatchAction(string $action, OutputChannelAllocatorInterface $outputChannelAllocator, OutputModifierEvent $event): OutputModifierEvent
    {
        /** @var Search $query */
        $query = $event->getParameter('query');
        $term = $event->getParameter('term');
        $outputChannelName = $outputChannelAllocator->getOutputChannelName();
        $request = $this->requestStack->getCurrentRequest();

        if ($outputChannelName == 'search' && $action == 'post_query_build') {
            Logger::debug("Modify Query: " . $query::class. ' '.$query);
            Logger::debug("Modify Term: " . $term);
                // old search over lucene
            return $event;
        }

        if ($outputChannelName !== 'search_all') {
            return $event;
        }
        if ($action !== 'post_query_build') {
            return $event;
        }
        if($request->query->get('author') == 1) {
            $matchQuery = new \ONGR\ElasticsearchDSL\Query\FullText\MatchQuery('author.name', $term);
            $boolQuery = new \ONGR\ElasticsearchDSL\Query\Compound\BoolQuery();
            $boolQuery->add($matchQuery);
            $queryStringQuery = new \ONGR\ElasticsearchDSL\Query\Joining\NestedQuery('author',$boolQuery);
        } else {
            $queryStringQuery = new \ONGR\ElasticsearchDSL\Query\FullText\QueryStringQuery(
                $term,
                [
                    // 'fields' => ['keyword'],
                    'analyzer' => 'keyword_analyzer',
                    // 'default_field' => 'title',
                    'analyze_wildcard' => true,
                    'auto_generate_synonyms_phrase_query' => true
                ]
            );
        }
        $query = new \ONGR\ElasticsearchDSL\Search();
        $sortField = null;
        switch ($request->query->get('sort')) {
            case 'score':
                $sortField = null;
                break;
            case 'date_asc':
                $sortField = new \ONGR\ElasticsearchDSL\Sort\FieldSort('publishdate', \ONGR\ElasticsearchDSL\Sort\FieldSort::ASC);
                break;
            case 'date_desc':
            case 'date':
            default:
                $sortField = new \ONGR\ElasticsearchDSL\Sort\FieldSort('publishdate', \ONGR\ElasticsearchDSL\Sort\FieldSort::DESC);
                break;
        }
        $query->addQuery($queryStringQuery);

        if($request->query->get('date_start') || $request->query->get('date_end')) {
            $start = strtotime($request->query->get('date_start'));
            $end = strtotime($request->query->get('date_end'));
            $filter_array = array();
            if($start) {
                $filter_array['gte'] = date('Ymd000000',$start);
            }
            if($end) {
                $filter_array['lte'] = date('Ymd000000', $end);
            }
            if($start != $end && $filter_array) {
                $datefilter = new \ONGR\ElasticsearchDSL\Query\TermLevel\RangeQuery('publishdate',$filter_array);
                $boolQuery = new \ONGR\ElasticsearchDSL\Query\Compound\BoolQuery();
                $boolQuery->add($datefilter, \ONGR\ElasticsearchDSL\Query\Compound\BoolQuery::FILTER);
                $query->addQuery($boolQuery);    
            }
        }

        if($sortField) {
            $query->addSort($sortField);
        }

        // modify query here, e.g. $query->addQuery(...)
        Logger::debug("Modify Query: " . $query::class. ' '.json_encode($query->toArray()));
        Logger::debug("Modify Term: " . $term);
        $event->setParameter('query', $query);

        return $event;
    }
}

You can find in this code some customisations too for ordering or filtering by date (if user check it on frontend), or filtering to authors name....