Various issues consider to improve

[donhardman]

While working with library I mean some issues that would be nice to improve

1. Lack of ES support. This is kind of hard sometimes to import modules when your project is written in ES while we have only CommonJS, to make it work I used the following code, suggest to support both ES and CommonJS as many libs do

import { createRequire } from 'module';
const require = createRequire(import.meta.url);
const manticoreSearch = require('manticoresearch-ts');
let {ServerConfiguration, createConfiguration, IndexApi, SearchApi, UtilsApi, SearchQuery, SearchRequest, KnnQuery} = manticoreSearch;

2. A bit hard and feels like we're overkilling it with multiple objects and many calls to be done. While others do not do it and are simpler with a single search method call.

// First search in segments to find matching content
const segmentQuery = new SearchQuery();
segmentQuery.query_string = query;

const segmentRequest = new SearchRequest();
segmentRequest.table = SEGMENT_INDEX;
segmentRequest.query = segmentQuery;
segmentRequest.offset = 0;
segmentRequest.limit = limit * 3; // Get more segments to ensure we don't miss any

const segmentResults = await searchApi.search(segmentRequest);

can be easier like

something.search('index', 'query');

Like here: https://github.com/qdrant/qdrant-js/blob/master/examples/node-js-basic/index.js#L75-L77
Or here: https://github.com/meilisearch/meilisearch-js?tab=readme-ov-file#basic-search-

3. We probably should improve docs, as they are now hard to understand regarding how to apply filters or use Knn search and so on. I suggest creating an examples directory and filling it with some handy examples.

4. I tried to do vector search, but due to lack of docs I assigned knn to Query object and the code let me do so! Surprisingly, it does not work and we should assign it to searchRequest instead. To make things stricter, maybe we can implement some kind of error when something is not set properly? Instead of using defaults when performing a query, if the user doesn't set something explicitly, we could show an error. This would make it much easier to understand things.

const vectorQuery = new KnnQuery();
vectorQuery.field = "embedding";
vectorQuery.k = limit;
vectorQuery.query_vector = embeddings[0];
// const embedding_sum = vectorQuery.query_vector.reduce((sum, value) => sum + value, 0);
const segmentQuery = new SearchQuery();
segmentQuery.knn = vectorQuery;

const segmentRequest = new SearchRequest();
segmentRequest.table = SEGMENT_INDEX;
segmentRequest.query = segmentQuery;
segmentRequest.offset = 0;
segmentRequest.limit = limit * 3;

const segmentResults = await searchApi.search(segmentRequest);

// Get unique webpage IDs from matching segments
const webpageIds = [...new Set(segmentResults.hits.hits.map(hit => hit._source.webpage_id))];

[Nick-S-2018]

The second suggestion would require considerable changes in auto-generated code which we also would have to implement in all our clients. We already used that approach in the past, but eventually decided to abandon it.

As for the other ones, I think, there shouldn't be major obstacles to implement them.