Are you working with SEAL? Let me know!

[alexander-schranz]

It is nice to see that some people are developing projects they work on daily basis which are using seal as seen in the packagist statistics:

https://packagist.org/packages/cmsig/seal/stats

I'm highly interesting which developers and companies using SEAL to get there feedback about it. So let me know who you are and what kind of software you created with SEAL if possible. What adapter and in which framework you use SEAL.

[ToshY]

Hey @alexander-schranz 👋

I've been using SEAL with the schranz-search/symfony-bundle and schranz-search/seal-meilisearch-adapter for roughly a year now (see #315) for a hobby project, and up until now I have been only using some functionality of it, in particularly schemas and reindexing (partially). As you might be worndering why I'm only partially using SEAL, I will try to explain certain aspects and the ups-and-downs I personally encountered when using it.

Schemas

I find the schemas are very straighforward to use and it's easy to add new fields. The ability to have searchable, filterable and sortable makes it really flexible and not complicated at all. I have no further remarks about this, it just works nicely 👍

Reindex operations

While I initially used the provided schranz:search:reindex a couple of times, I realised it will reindex indexes sequentially. This is doable if your indexes are small, but having to rebuild multiple larger indexes this can take some time, and rebuilding one index requires waiting for the other one to finish. As this is a bit of a "blocking" action (in the context of contious deployment), I instead created a custom command to dispatch messages (note: Symfony Messenger) for each index, and in the handler it calls the schranz ReindexCommand using a Symfony Process.

// This logic is in the message handler
$index = 'test';
$process = new Process(
    [
        'bin/console',
        ReindexCommand::getDefaultName(),
        ...['--index', $index, '--drop'],
    ],
    cwd: '/app',
    timeout: null,
);

This way, when intially calling the custom command, it is 1) non-blocking (as messages will be handled async through the bus) and 2) having multiple workers you can handle multiple different indexes at the same time (and thus reindex faster).

_{FYI, I also just saw #430, and initially thought this would solve the above, but it seems "bulk" refers to inserts on single index.}

Conditions

I wanted to use the filter and conditions SEAL had to offer, but I quickly find myself that operations like OR/AND (#317), distuingish between null or empty values (#318), or operators like IN / NOT IN (Meilisearch) were not available at the time. And while #317 has been merged recently, I can still not fully use the MeilisearchSearcher and have to rely on Meilisearch Client directly (which mostly has to do with the next point).

"Filters"

The main problem I am still facing with Meilisearch, as well as SEAL, is that both lack the ability to add and remove "filters" dynamically. To further explain what I mean with (for other readers as well), I've described this before here meilisearch/meilisearch-symfony#315 and #315.

This ability to add and remove these in a flexibile manner is something that is essential for my use cases, and I had to resort to create custom logic for this to achieve what I needed (see meilisearch/meilisearch-symfony#315 (comment)). Over time, I've refined this approach and have a working system with a customised Meilisearch Client that allows for this flexibility.

While I've come to peace with the idea that something like this (filters) is not something that will be implementend in either libraries any time soon (or at all), it is the main reason I'm not able to use SEAL to its full extend.

---

Hope you didn't get the wrong idea about the above, because I'm still happily using SEAL as part of my project and I will keep on using it, even if it's just for the easy-to-use schemas and reindexing. 🙂

Keep up the great work 🥇

[alexander-schranz]

Really thank you for your feedback.

Hope you didn't get the wrong idea about the above, because I'm still happily using SEAL as part of my project and I will keep on using it, even if it's just for the easy-to-use schemas and reindexing. 🙂

I really want to start with this one as that is mostly also something I really try to communicate in my presentation and talks about SEAL. There are situations like yours where you will end up where SEAL does not provide all functionality which you are searching for. But the solution you are doing is the one I recommend, use the tools of SEAL (Schema, Reindexing, ...) what you can do, but if you end in search queries for special cases where SEAL does not fulfill your requirements just use the client of your search engine directly. If you need migrate to a other Engine it is still easier as when not using SEAL, as still SEAL will provide you the other functionality, really happy to see that you did go that way and kept SEAL in your codebase.

Filters: The main problem I am still facing with Meilisearch, as well as SEAL, is that both lack the ability to add and remove "filters"

Why I may think that filters or may "Global Filters" will not be part of the core I think I have some idea in mind to make things easier here and add some situation to the docs.

Conditions: distuingish between null or empty values (#318)

Can you add some more detail information about this to #318. That would be great.

Conditions: InCondition

Please create an issue for this one.

Conditions: NotInCondition

Please create an own issue for this one.

Reindex operations

You did interpret #430 correctly. It is bulk insert for a single index. But still it will improve the reindexing a lot see comment in #24. Would be interesting how faster that is in your case. PS: you can define a bulk size with --bulk-size=100. Another feature which might be interesting for you is #212 but I need rethink about that implementation.

Schemas

Thx for the feedback, good to hear it works for you.

---

@ToshY Really thank you, this kind of feedbacks really helps to see where the limits are and where things can be improved in two cases of features but also documentation of extending. Thank you for your time for this detailed feedback.

[empiricompany]

Hey @alexander-schranz!

Just wanted to give you a shoutout for creating SEAL! We've integrated it into our experimental openmage-search extension for the backend indexing, while using instantsearch.js for the frontend UI.

This is the project currently under development:
empiricompany/openmage-search#1

Even though our project is still experimental, your library has already made a big difference in how we approach search. So far we've implemented Typesense fully, and Meilisearch is partially implemented (backend only).

Here's what we're loving about SEAL:

• Adapter Pattern: The way you've designed the adapter interface makes it really straightforward to support different search engines without changing our core code.

• Schema System: The Schema, Index, and Field classes provide a clean way to map OpenMage product attributes to search fields.

• Reindexing System: After starting with the basic saveDocument/deleteDocument APIs, we switched to using the indexer provider system which handles the bulk operations nicely.

We did have one challenge that I wanted to mention - we found it a bit surprising that we need to declare the complete schema every time we connect to the engine. I'm guessing this was an intentional design choice?

Also, we noticed there aren't methods to modify individual fields in the schema (like dropping a single field). Currently, when we need to change the schema, we set the drop schema flag in the indexer and recreate everything. This works fine with our test data, but we're a bit concerned about how it might perform with larger datasets, having to recreate the entire schema instead of just modifying the affected fields.

Here’s how we used to do it before:
https://github.com/empiricompany/openmage-search/blob/d4cf91120f985853f2dbf8777e4147417e4450da/app/code/community/MM/Search/Model/Api.php#L195

Despite these small challenges, SEAL has been a great foundation for our search implementation. The abstraction layer really does make it much easier to work with different search engines.

Thanks for your awesome work!

Another project that seems interesting to us and that we want to integrate is Loupe.

Cheers,
Tony

[alexander-schranz]

Hello @empiricompany,

Really thank you for your feedback.

We did have one challenge that I wanted to mention - we found it a bit surprising that we need to declare the complete schema every time we connect to the engine. I'm guessing this was an intentional design choice?

Yes SEAL is build on top of a strict Schema, in most cases I really recommend to a have schema which is not generated by dynamic data and statically like a database schema inside your code base. Thats also why in the integrations its mostly a config file which is commited to your GIT repo and not generated out of something else. With the strict Schema we are able to support complex data structures even for Search engines which example doesn't support nested JSON objects are supported by SEAL – as via a strict schema SEAL can be able to flatten the data and unflatten on read. So the document structure you give to SEAL does not in all search engines match the raw document inside the index. Thats why for all operations SEAL requires the full Schema. Also I recommend you that a Schema is simplified and only create the way for the queries performaned on it, see also docs about complex schema and best practices: https://php-cmsig.github.io/search/schema/index.html#best-practices.

Also, we noticed there aren't methods to modify individual fields in the schema (like dropping a single field)

Yes as an abstraction that something we are not supporting, and I also think we can't. But this mostly goes into the same topic have a more simpler schema as like duplicating your complex entity structure on indexes, Schema shouldnt change too much in my opinion like your database tables also normally don't do.

we set the drop schema flag in the indexer and recreate everything

You may want todo it async via a job trigger if you have a lot of data. While reindexer via Generators mostly can if you directly fetch your data be very fast. Depending on the Engine you might can call createIndex to update the Index settings without a reindex if only new fields are added which not require any data to be reindexed.

Another project that seems interesting to us and that we want to integrate is Loupe.

Loupe is rich on features and if you already support Meilisearch or Typesense it will be easy for you to support Loupe. Even Facets are coming to Loupe in the next release, we are working on getting facets also to SEAL after it.

---

If you face any issues or if you are missing something which else is not yet abstracted for your case let us know. Its always great to know where the limits are and if we maybe can do something against it. Even it is I want to document such cases to maybe provide for people some workarounds.

[alexander-schranz]

@empiricompany maybe interesting for you Facets are now also supported by SEAL by all adapters. Which is sure a common Ecommerce features you need maybe.

[lochmueller]

Hey @alexander-schranz

I’ve recently become a user of your SEAL framework—thank you for making it available! It’s really well implemented and a pleasure to work with.

As part of a TYPO3 project, I was able to secure funding to support the integration of extensions for TYPO3. Feel free to get in touch with me—you should also get a share of the pie :)

The following TYPO3 extensions came out of this effort (and are still under development):

• https://github.com/lochmueller/index – An indexing extension that prepares content for indexing. The goal was to decouple indexing from search while mapping TYPO3-specific features. It works asynchronously via the Symfony Messenger (built into TYPO3 core) and triggers PSR-14 events containing the content to be indexed.
• https://github.com/lochmueller/seal – The integration of your SEAL framework. The adapter can be configured on a per-site basis in TYPO3 (so a single TYPO3 instance can use different adapters). The Index PSR-14 event feeds into the indexing process, and there is currently a basic search plugin for simple queries. Additionally, it includes a default TYPO3 adapter that uses a FULLTEXT index on the local database, so the extension works “out of the box.”
• https://github.com/lochmueller/seal_ai – A proof-of-concept adapter for integrating Symfony AI (https://github.com/symfony/ai). During indexing, content is written into a MariaDB store (store is not yet configurable). For searching, the query is vectorized with embeddings and then matched accordingly. This PoC essentially integrates an AI-powered RAG search.

Initially, I considered writing a general PDO adapter for you. Since TYPO3 can run on PDO, this would have simplified the TYPO3 adapter quite a bit—but I ultimately decided against it :-)

Challenges I encountered along the way:

• Autocomplete: At the moment, I trigger the search directly with the query term and parse the results myself. Not very elegant, and I haven’t explored it further yet.
• Pagination: TYPO3’s paginator expects to know the number of elements on the current page (before I iterate the items in the output). Since the result is a generator that only provides the total amount, integration is a bit tricky.

Feel free to write me a PM to share the cake...

Thanks again,
Tim

[alexander-schranz]

Wow, really great what you already created. It is sure the master challange you did not only create a integration into Typo3 but also create your own Adapters (Typo3DB and Ai), if you have anything to add about the documentation about creating an adapter here let me know. A PDO adapter was also something I was thinking of but currently Loupe was enough for me as stumbled not yet over any hosting not having the SQLite extension installed. It is also the default adapter in the new Contao CMS Backend search.

Autocomplete: At the moment, I trigger the search directly with the query term and parse the results myself. Not very elegant, and I haven’t explored it further yet.

Mostly must say I did not have any issues in the past with autocomplete / live search but I think it depends on the used JS library. As SEAL does not provide any Controllers the responsibility to get it into a format your JS library accepts need to be handled in your controller, My code in such cases mostly look like this:

public function searchAction(Request $request): Response
{
    $result = $this->engine->createQuery()->...->getResult();
    
    if ($request->getFormat() === 'json') {
        $jsonData = [
            'result' => \array_map(function() {
            
            }, [...$result]),
            'total' => $result->total(),
        ];
    
        return new JsonResponse($jsonData);
    }
    
    return new Response($this->twig->render('search.html.twig', ['result' => $result]));
}

And had a JS implace which trigger the /search.json on input changes and form submits to /search without the JSON. Maybe something I could add as a cookbook how to implement something like that into a project.

Pagination: TYPO3’s paginator expects to know the number of elements on the current page (before I iterate the items in the output). Since the result is a generator that only provides the total amount, integration is a bit tricky.

SEAL does not know what kind of pagination mechanic (limitoffset vs page/pagePerItem) you use. So you need to keep the variables to provide them to your paginator:

    $page = 1; // get from query parameter
    $pageSize = 10;

    $result = $this->engine->createSearchBuilder('blog')
        ->addFilter(/* ... */)
        ->limit($pageSize)
        ->offset(($page - 1) * $pageSize)
        ->getResult();

    $total = $result->total();
    $maxPage = ceil($total / $pageSize) ?: 1;

See also: https://php-cmsig.github.io/search/search-and-filters/index.html#pagination

If your paginator can only work with array and not with iterable / \Generator classes the easiest way to convert a generator to an array is $items = [...$generator]; the $page, $pageSize, $maxPage as seen you need to keep somewhere as variables or constants if that required by your paginiator, as SEAL doesn't know about the pagination mechanic.

I Hope this answers a few questions. I will write you an Email. If you have any more questions let me know.