Alternative solutions for searching that fit existing system requirements

[NateWr]

The search functionality in OJS and OMP has not had much attention over the years. It may be worth exploring what third-party solutions we can integrate which would provide a better search experience, but which wouldn't have steep system requirements [like Lucene/Solr].

Also, the smarty functions for pagination are only used on search pages. If these pages are brought in line with the changes in #1922, those functions could be removed.

[asmecher]

@lilients, just FYI, as I believe you're working on modernizing the Lucene plugin? We have no concrete goals here, just a sense that the built-in search engine could be improved, and that there may be good alternatives that we can integrate rather than maintaining our own.

[asmecher]

(See also #321 -- searching should be extended to issue galleys etc.)

[Glottotopia]

Speaking only for OMP, the amount of metadata in a catalog is very small, and searching/arranging/filtering that data could completely be done client side. That excludes the full texts of course. See here for an example with actual (static) data from an OMP installation: http://langsci.github.io/catalog

[asmecher]

Speaking only for OMP, ...

There's a related thread about this in the forum: https://forum.pkp.sfu.ca/t/omp-filtering-for-book-catalog/39618

[NateWr]

An alternative to our own searching toolset that wouldn't require additional server dependencies: https://github.com/teamtnt/tntsearch

[asmecher]

After some experimentation, I'm discarding TNTSearch as an option. Some strikes against:

• It generally uses SQLite; that would be a new deployment requirement, and I'm not sure it would lead us into trouble or not with our community. (Not a show-stopper.)
• It does have a MySQL storage driver, but not a PostgreSQL one. It doesn't 100% play well with our environment -- I get some complaints about prepared statements not being closed that I haven't investigated fully (General error: 2014 Cannot execute queries while other unbuffered queries are active.). Probably not a show-stopper.
• It flat out does not support hard constraints (e.g. "all results must be in this context ID" or "all results must be in that section/category/etc"). The Laravel Scout driver works around this by first searching for all matches, then discarding any nonconforming matches. This will not scale well in our environment. We can't work around this by modifying the queries, as queries are hard-coded and would need to be overridden in several successive functions with similar constraints added: totalDocumentsInCollection, totalMatchingDocuments, getAllDocumentsForKeyword, at least. In short, this is not a good fit for our needs for multitenancy, faceting, etc.

[asmecher]

^ Interesting! It does require SQLite, which it uses for its own storage, so I'd want to verify that it has very wide uptake in general -- but I suspect it does. It's a shame it can't just use the same PDO toolset it uses for a data source as the indexing storage as well.

[jonasraoni]

If SQLite is still a requirement for this library, it will need an extra note for sysadmins to be cautious when sharing the index folder (e.g. "cloud" environment) with other nodes, given the underlying OS might not be that great with file locks 👀

Another issue related to the locks, is the lack of concurrency, so larger journals will probably face issues if the indexing process is too slow... So we'll also need to change how the indexing is working, and offload it to a single-running task.

[asmecher]

(I've confirmed that TNTSearch is not an option.)

[NateWr]

Ah, that's a shame. I saw a couple of issues (1, 2) talking about alternate db implementations. I wonder if they'd accept a PR going forward...

[jonasraoni]

According to the comments, the issue 1 should be done, as the maintainer said it would be part of the v2 (they've just released the v3).

[asmecher]

(I've confirmed that TNTSearch is not an option.)

[Glottotopia]

For reference, here is a client side implementation of metadata search in OMP
https://langsci-press.org/catalogSearch

Here is the code https://github.com/langsci/catalogSearchPage

[asmecher]

Coming to a concrete proposal for this:

• We use Laravel Scout as a layer separating indexing calls from the underlying implementation. It's simple and covers the functionality we need (searching, filtering, pagination).
• We drop our existing fulltext indexing/searching implementation -- it doesn't perform or scale well, and is too much code.
• For out-of-the-box OJS/OMP/OPS, we use MySQL and PostgreSQL full-text searching via a custom Scout driver. This adds a new table called submissions_fulltext for titles, abstracts, and body texts, and leaves other types of searching/filtering (by section, category, etc.) to the existing relational tables.
• For more advanced deployments, we can implement Scout drivers like OpenSearchEngine.php. These are pretty quick and easy to build. I'm not sure how far I can push towards using the preexisting implementations -- that remains to be seen.

[jonasraoni]

Oh, I was waiting for a request to prioritize this, but I guess you've started working on it, right? If yes, I'll drop it from my radar, and maybe just come up with a search plugin to fix old installations.

I agree with the full-text index, that's the best out-of-the-box option we can offer and it's pretty fast. AFAICR mixing a full-text index with non full-text filters leads to performance issues, but that's not going to be visible on relatively small datasets (our case).

[asmecher]

Closing this discussion in favour of the concrete implementation proposed here: #8920 (comment)