Merge #707

707: Introduce finitePagination setting: remove default fetch of 200 documents r=bidoubiwa a=bidoubiwa

Fixes: #569 
fixes: #521 

## Previously
By default instant-meilisearch used to fetch 200 documents on every search request. This was needed to ensure we can provide a finite pagination when the user used the `pagination` widget (with the 1, 2, 3, at the bottom of the screen). 


## Issue 

This impacted the speed of the search considerably. Indeed meilisearch is designed to retrieve the most relevant documents, but not all the relevant documents. The more you ask, the more the search engine has to process. There is no difference regarding the search time between retrieving 6 or 7 documents, but between 6 and 200 documents to every request, the impact can be huge.

It also impacted users using the `infiniteHits` widget as well while it is not necessary for them. The `infiniteHits` settings does not need to showcase a start and an end to the pagination. It only has to enable the `show more` button whenever there are more hits after the current showcased ones. 


## New behavior


This PR introduces a new parameter: `finitePagination`.

`finitePagination` has a default value of `false`, when set to false, each search requests a limit: `(hitsPerPage + 1) * (page + 1)`
- [hitsPerPage](https://github.com/meilisearch/instant-meilisearch#-hitsperpage) (parameter of `instantsearch` and default to `6`) is the number of results on each page, or added on the current page on each `show more` click in case of infiniteHits. In the case if infiniteHits, we need to be able to know if there are more hits to come to enable or disable the `show more` button. Thus we add `1` to `hitsPerPage` so that if Meilisearch returns `hitsPerPage  + 1` documents we know it has more.
- page: behind the scene value tracking the current page the user is on. For example `1` if you clicked once on `load more`. Because it starts at `0`, we add `+1` for the multiplication. 

The pagination will stop at `paginationTotalHits` (default 200). 

When settings `finitePagination` to `true`, one search request will ask for a limit of `paginationTotalHits` as limit. Thus in the default case, one search request will return `200` documents. 
This number can be changed with `paginationTotalHits`

The following will make one request of `7` documents as `hitsPerPage` is at 6 by default. 
```js
instantMeiliSearch(host, key, {
   finitePagination: false // default false
   paginationTotalHits: 60 // default 200
})
```

You can [change hitsPerPage](https://github.com/meilisearch/instant-meilisearch#-hitsperpage).

The following will make one request of 200 documents
```js
instantMeiliSearch(host, key, {
   finitePagination: true // default false
   paginationTotalHits: 60 // default 200
})

```





Co-authored-by: Charlotte Vermandel <[email protected]>

Author: meili-bors[bot]

README.md

@@ -80,6 +80,7 @@ const searchClient = instantMeiliSearch(
 
 - [`placeholderSearch`](#placeholder-search): Enable or disable placeholder search (default: `true`).
 - [`paginationTotalHits`](#pagination-total-hits): Maximum total number of hits to create a finite pagination (default: `200`).
+- [`finitePagination`](#finite-pagination): Used to work with the [`pagination`](#-pagination) widget (default: `false`) .
 - [`primaryKey`](#primary-key): Specify the primary key of your documents (default `undefined`).
 - [`keepZeroFacets`](#keep-zero-facets): Show the facets value even when they have 0 matches (default `false`).
 
@@ -111,17 +112,31 @@ When placeholder search is set to `false`, no results appears when searching on
 
 ### Pagination total hits
 
-The total (and finite) number of hits you can browse during pagination when using the [pagination widget](https://www.algolia.com/doc/api-reference/widgets/pagination/js/). If the pagination widget is not used, `paginationTotalHits` is ignored.<br>
+The total (and finite) number of hits (default: `200`) you can browse during pagination when using the [pagination widget](https://www.algolia.com/doc/api-reference/widgets/pagination/js/) or the [`infiniteHits` widget](#-infinitehits). If none of these widgets are used, `paginationTotalHits` is ignored.<br>
 
-Which means that, with a `paginationTotalHits` default value of 200, and `hitsPerPage` default value of 20, you can browse `paginationTotalHits / hitsPerPage` => `200 / 20 = 10` pages during pagination. Each of the 10 pages containing 20 results.<br>
+For example, using the `infiniteHits` widget, and a `paginationTotalHits` of 9. On the first search request 6 hits are shown, by clicking a second time on `load more` only 3 more hits are added. This is because `paginationTotalHits` is `9`.
 
-The default value of `hitsPerPage` is set to `20` but it can be changed with [`InsantSearch.configure`](https://www.algolia.com/doc/api-reference/widgets/configure/js/#examples).<br>
+Usage:
 
 ```js
-{ paginationTotalHits : 20 } // default: 200
+{ paginationTotalHits: 50 } // default: 200
 ```
 
-⚠️ Meilisearch is not designed for pagination and this can lead to performances issues, so the usage of the pagination widget is not encouraged. However, the `paginationTotalHits` parameter lets you implement this pagination with less performance issue as possible: depending on your dataset (the size of each document and the number of documents) you might decrease the value of `paginationTotalHits`.<br>
+`hitsPerPage` has a value of `6` by default and can [be customized](#-hitsperpage).
+
+### Finite Pagination
+
+Finite pagination is used when you want to add a numbered pagination at the bottom of your hits (for example: `< << 1, 2, 3 > >>`).
+To be able to know the amount of page numbers you have, a search is done requesting `paginationTotalHits` documents (default: `200`).
+With the amount of documents returned, instantsearch is able to render the correct amount of numbers in the pagination widget.
+
+Example:
+
+```js
+{ finitePagination: true } // default: false
+```
+
+⚠️ Meilisearch is not designed for pagination and this can lead to performances issues, so the usage `finitePagination` but also of the pagination widgets are not recommended.<br>
 More information about Meilisearch and the pagination [here](https://github.com/meilisearch/documentation/issues/561).
 
 ### Primary key

jest.config.js

@@ -6,6 +6,15 @@ module.exports = {
     'jest-watch-typeahead/filename',
     'jest-watch-typeahead/testname',
   ],
+  collectCoverage: true,
+  coveragePathIgnorePatterns: [
+    'cypress/',
+    'playgrounds/',
+    'scripts',
+    'templates',
+    'tests',
+    '__tests__',
+  ],
   projects: [
     {
       displayName: 'build',

package.json

@@ -4,6 +4,7 @@
   "private": false,
   "description": "The search client to use Meilisearch with InstantSearch.",
   "scripts": {
+    "clear_jest": "jest --clearCache",
     "cleanup": "shx rm -rf dist/",
     "test:watch": "yarn test --watch",
     "test": "jest --runInBand --selectProjects dom --selectProjects node",

src/adapter/search-request-adapter/__tests__/search-params.tests.ts

@@ -3,8 +3,9 @@ import { adaptSearchParams } from '../search-params-adapter'
 test('Adapt basic SearchContext ', () => {
   const searchParams = adaptSearchParams({
     indexUid: 'test',
-    paginationTotalHits: 20,
+    pagination: { paginationTotalHits: 20, page: 0, hitsPerPage: 6 },
     defaultFacetDistribution: {},
+    finitePagination: false,
   })
   expect(searchParams.attributesToHighlight).toContain('*')
   expect(searchParams.attributesToHighlight?.length).toBe(1)
@@ -13,10 +14,11 @@ test('Adapt basic SearchContext ', () => {
 test('Adapt SearchContext with filters, sort and no geo rules ', () => {
   const searchParams = adaptSearchParams({
     indexUid: 'test',
-    paginationTotalHits: 20,
+    pagination: { paginationTotalHits: 20, page: 0, hitsPerPage: 6 },
     facetFilters: [['genres:Drama', 'genres:Thriller'], ['title:Ariel']],
     sort: 'id < 1',
     defaultFacetDistribution: {},
+    finitePagination: false,
   })
 
   expect(searchParams.filter).toStrictEqual([
@@ -31,11 +33,12 @@ test('Adapt SearchContext with filters, sort and no geo rules ', () => {
 test('Adapt SearchContext with filters, sort and geo rules ', () => {
   const searchParams = adaptSearchParams({
     indexUid: 'test',
-    paginationTotalHits: 20,
+    pagination: { paginationTotalHits: 20, page: 0, hitsPerPage: 6 },
     facetFilters: [['genres:Drama', 'genres:Thriller'], ['title:Ariel']],
     insideBoundingBox: '0,0,0,0',
     sort: 'id < 1',
     defaultFacetDistribution: {},
+    finitePagination: false,
   })
 
   expect(searchParams.filter).toStrictEqual([
@@ -51,10 +54,11 @@ test('Adapt SearchContext with filters, sort and geo rules ', () => {
 test('Adapt SearchContext with only facetFilters and geo rules ', () => {
   const searchParams = adaptSearchParams({
     indexUid: 'test',
-    paginationTotalHits: 20,
+    pagination: { paginationTotalHits: 20, page: 0, hitsPerPage: 6 },
     facetFilters: [['genres:Drama', 'genres:Thriller'], ['title:Ariel']],
     insideBoundingBox: '0,0,0,0',
     defaultFacetDistribution: {},
+    finitePagination: false,
   })
 
   expect(searchParams.filter).toEqual([
@@ -69,10 +73,11 @@ test('Adapt SearchContext with only facetFilters and geo rules ', () => {
 test('Adapt SearchContext with only sort and geo rules ', () => {
   const searchParams = adaptSearchParams({
     indexUid: 'test',
-    paginationTotalHits: 20,
+    pagination: { paginationTotalHits: 20, page: 0, hitsPerPage: 6 },
     insideBoundingBox: '0,0,0,0',
     sort: 'id < 1',
     defaultFacetDistribution: {},
+    finitePagination: false,
   })
 
   expect(searchParams.filter).toEqual(['_geoRadius(0.00000, 0.00000, 0)'])
@@ -84,12 +89,97 @@ test('Adapt SearchContext with only sort and geo rules ', () => {
 test('Adapt SearchContext with no sort and no filters and geo rules ', () => {
   const searchParams = adaptSearchParams({
     indexUid: 'test',
-    paginationTotalHits: 20,
+    pagination: { paginationTotalHits: 20, page: 0, hitsPerPage: 6 },
     insideBoundingBox: '0,0,0,0',
     defaultFacetDistribution: {},
+    finitePagination: false,
   })
 
   expect(searchParams.filter).toEqual(['_geoRadius(0.00000, 0.00000, 0)'])
   expect(searchParams.attributesToHighlight).toContain('*')
   expect(searchParams.attributesToHighlight?.length).toBe(1)
 })
+
+test('Adapt SearchContext with finite pagination', () => {
+  const searchParams = adaptSearchParams({
+    indexUid: 'test',
+    pagination: { paginationTotalHits: 20, page: 0, hitsPerPage: 6 },
+    insideBoundingBox: '0,0,0,0',
+    defaultFacetDistribution: {},
+    finitePagination: true,
+  })
+
+  expect(searchParams.limit).toBe(20)
+})
+
+test('Adapt SearchContext with finite pagination on a later page', () => {
+  const searchParams = adaptSearchParams({
+    indexUid: 'test',
+    pagination: { paginationTotalHits: 20, page: 10, hitsPerPage: 6 },
+    insideBoundingBox: '0,0,0,0',
+    defaultFacetDistribution: {},
+    finitePagination: true,
+  })
+
+  expect(searchParams.limit).toBe(20)
+})
+
+test('Adapt SearchContext with finite pagination and pagination total hits lower than hitsPerPage', () => {
+  const searchParams = adaptSearchParams({
+    indexUid: 'test',
+    pagination: { paginationTotalHits: 4, page: 0, hitsPerPage: 6 },
+    insideBoundingBox: '0,0,0,0',
+    defaultFacetDistribution: {},
+    finitePagination: true,
+  })
+
+  expect(searchParams.limit).toBe(4)
+})
+
+test('Adapt SearchContext with no finite pagination', () => {
+  const searchParams = adaptSearchParams({
+    indexUid: 'test',
+    pagination: { paginationTotalHits: 20, page: 0, hitsPerPage: 6 },
+    insideBoundingBox: '0,0,0,0',
+    defaultFacetDistribution: {},
+    finitePagination: false,
+  })
+
+  expect(searchParams.limit).toBe(7)
+})
+
+test('Adapt SearchContext with no finite pagination on page 2', () => {
+  const searchParams = adaptSearchParams({
+    indexUid: 'test',
+    pagination: { paginationTotalHits: 20, page: 1, hitsPerPage: 6 },
+    insideBoundingBox: '0,0,0,0',
+    defaultFacetDistribution: {},
+    finitePagination: false,
+  })
+
+  expect(searchParams.limit).toBe(13)
+})
+
+test('Adapt SearchContext with no finite pagination on page higher than paginationTotalHits', () => {
+  const searchParams = adaptSearchParams({
+    indexUid: 'test',
+    pagination: { paginationTotalHits: 20, page: 40, hitsPerPage: 6 },
+    insideBoundingBox: '0,0,0,0',
+    defaultFacetDistribution: {},
+    finitePagination: false,
+  })
+
+  expect(searchParams.limit).toBe(20)
+})
+
+test('Adapt SearchContext with no finite pagination and pagination total hits lower than hitsPerPage', () => {
+  const searchParams = adaptSearchParams({
+    indexUid: 'test',
+    pagination: { paginationTotalHits: 4, page: 0, hitsPerPage: 6 },
+    insideBoundingBox: '0,0,0,0',
+    defaultFacetDistribution: {},
+    finitePagination: false,
+  })
+
+  expect(searchParams.limit).toBe(4)
+})

src/adapter/search-request-adapter/search-params-adapter.ts

@@ -59,13 +59,27 @@ export function adaptSearchParams(
 
   const placeholderSearch = searchContext.placeholderSearch
   const query = searchContext.query
-  const paginationTotalHits = searchContext.paginationTotalHits
 
-  // Limit
-  if ((!placeholderSearch && query === '') || paginationTotalHits === 0) {
+  // Pagination
+  const { pagination } = searchContext
+
+  // Limit based on pagination preferences
+  if (
+    (!placeholderSearch && query === '') ||
+    pagination.paginationTotalHits === 0
+  ) {
     meiliSearchParams.limit = 0
+  } else if (searchContext.finitePagination) {
+    meiliSearchParams.limit = pagination.paginationTotalHits
   } else {
-    meiliSearchParams.limit = paginationTotalHits
+    const limit = (pagination.page + 1) * pagination.hitsPerPage + 1
+    // If the limit is bigger than the total hits accepted
+    // force the limit to that amount
+    if (limit > pagination.paginationTotalHits) {
+      meiliSearchParams.limit = pagination.paginationTotalHits
+    } else {
+      meiliSearchParams.limit = limit
+    }
   }
 
   const sort = searchContext.sort

src/adapter/search-request-adapter/search-resolver.ts

@@ -23,18 +23,26 @@ export function SearchResolver(cache: SearchCacheInterface) {
       searchParams: MeiliSearchParams,
       client: MeiliSearch
     ): Promise<MeiliSearchResponse<Record<string, any>>> {
-      // Create key with relevant informations
+      const { pagination } = searchContext
+
+      // In case we are in a `finitePagination`, only one big request is made
+      // containing a total of max the paginationTotalHits (default: 200).
+      // Thus we dont want the pagination to impact the cache as every
+      // hits are already cached.
+      const paginationCache = searchContext.finitePagination ? {} : pagination
+
+      // Create cache key containing a unique set of search parameters
       const key = cache.formatKey([
         searchParams,
         searchContext.indexUid,
         searchContext.query,
+        paginationCache,
       ])
-      const entry = cache.getEntry(key)
+      const cachedResponse = cache.getEntry(key)
 
-      // Request is cached.
-      if (entry) return entry
+      // Check if specific request is already cached with its associated search response.
+      if (cachedResponse) return cachedResponse
 
-      // Cache filters: todo components
       const facetsCache = extractFacets(searchContext, searchParams)
 
       // Make search request

src/adapter/search-response-adapter/pagination-adapter.ts

@@ -1,5 +1,3 @@
-import { SearchContext, PaginationContext } from '../../types'
-
 /**
  * Slice the requested hits based on the pagination position.
  *
@@ -21,19 +19,3 @@ export function adaptPagination(
   const start = page * hitsPerPage
   return hits.slice(start, start + hitsPerPage)
 }
-
-/**
- * @param  {AlgoliaMultipleQueriesQuery} searchRequest
- * @param  {Context} options
- * @returns {SearchContext}
- */
-export function createPaginationContext(
-  searchContext: SearchContext
-): PaginationContext {
-  return {
-    paginationTotalHits: searchContext.paginationTotalHits || 200,
-    hitsPerPage:
-      searchContext.hitsPerPage === undefined ? 20 : searchContext.hitsPerPage, // 20 is the Meilisearch's default limit value. `hitsPerPage` can be changed with `InsantSearch.configure`.
-    page: searchContext?.page || 0, // default page is 0 if none is provided
-  }
-}

src/adapter/search-response-adapter/search-response-adapter.ts

@@ -5,7 +5,6 @@ import type {
 } from '../../types'
 import { ceiledDivision } from '../../utils'
 import { adaptHits } from './hits-adapter'
-import { createPaginationContext } from './pagination-adapter'
 
 /**
  * Adapt search response from Meilisearch
@@ -23,26 +22,25 @@ export function adaptSearchResponse<T>(
   const searchResponseOptionals: Record<string, any> = {}
 
   const facets = searchResponse.facetsDistribution
+  const { pagination } = searchContext
 
   const exhaustiveFacetsCount = searchResponse?.exhaustiveFacetsCount
   if (exhaustiveFacetsCount) {
     searchResponseOptionals.exhaustiveFacetsCount = exhaustiveFacetsCount
   }
 
-  const paginationContext = createPaginationContext(searchContext)
-
   const nbPages = ceiledDivision(
     searchResponse.hits.length,
-    paginationContext.hitsPerPage
+    pagination.hitsPerPage
   )
-  const hits = adaptHits(searchResponse.hits, searchContext, paginationContext)
+  const hits = adaptHits(searchResponse.hits, searchContext, pagination)
 
   const exhaustiveNbHits = searchResponse.exhaustiveNbHits
   const nbHits = searchResponse.nbHits
   const processingTimeMs = searchResponse.processingTimeMs
   const query = searchResponse.query
 
-  const { hitsPerPage, page } = paginationContext
+  const { hitsPerPage, page } = pagination
 
   // Create response object compliant with InstantSearch
   const adaptedSearchResponse = {

src/client/instant-meilisearch-client.ts

@@ -11,7 +11,7 @@ import {
   adaptSearchParams,
   SearchResolver,
 } from '../adapter'
-import { createSearchContext } from './contexts'
+import { createSearchContext } from '../contexts'
 import { SearchCache, cacheFirstFacetsDistribution } from '../cache/'
 
 /**
@@ -79,7 +79,7 @@ export function instantMeiliSearch(
         throw new Error(e)
       }
     },
-    searchForFacetValues: async function (_) {
+    searchForFacetValues: async function (_: any) {
       return await new Promise((resolve, reject) => {
         reject(
           new Error('SearchForFacetValues is not compatible with Meilisearch')

src/contexts/index.ts

@@ -0,0 +1 @@
+export { createSearchContext } from './search-context'