Merge #538

538: Add entries query option in plugin configuration r=bidoubiwa a=bidoubiwa

# Pull Request

## Related issue
Fixes #536 
Fixes #263 
Fixes #85

## What does this PR do?

A lot of users were asking for more control over what entries are indexed in Meilisearch and how.
For example:
- the batch sizes which has a default value of 500.
- the `locale` support to fetch all entries of every language

This PR adds a new setting `entriesQuery` that lets you set exactly how Meilisearch should fetch documents from your database. These options are provided directly by the `findMany` API of strapi ([see doc](https://docs.strapi.io/developer-docs/latest/developer-resources/database-apis-reference/entity-service/crud.html#findmany)).

Further explanation on how it works isn provided in the README of this PR.

## ❌ Breaking changes

The setting `populateEntryRule` in the plugin configuration is removed in favor of `populate` in `entriesQuery`.

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Charlotte Vermandel <[email protected]>

Author: meili-bors[bot]

README.md

@@ -202,7 +202,7 @@ Settings:
 - [🪄 Transform entries](#-transform-entries)
 - [🤚 Filter entries](#-filter-entries)
 - [🏗 Add Meilisearch settings](#-add-meilisearch-settings)
-- [👥 Populate entry rule](#-populate-entry-rule)
+- [🔎 Entries query](#🔎-entries-query)
 
 ### 🏷 Custom index name
 
@@ -373,59 +373,33 @@ module.exports = {
 
 [See resources](./resources/meilisearch-settings) for more settings examples.
 
-### 👥 Populate entry rule
+### 🔎 Entries query
 
-Content-types in Strapi may have relationships with other content-types (ex: `restaurant` can have a many-to-many relation with `category`). To ensure that these links are fetched and added to an entry correctly from your Strapi database, the correct populate rule must be provided ([see documentation](https://docs-next.strapi.io/developer-docs/latest/developer-resources/database-apis-reference/entity-service/populate.html#basic-populating)).
+When indexing a content type to Meilisearch, the plugin has to fetch the documents from your database. With `entriesQuery` it is possible to specify some options that should be applied during the fetching of the entries.
+The options you can set are described in the [`findMany` documentation](https://docs.strapi.io/developer-docs/latest/developer-resources/database-apis-reference/entity-service/crud.html#findmany) of Strapi. However, we do not accept any changes on the `start` parameter.
 
-To communicate the populate rule, use the `populateEntryRule` setting on the according content-type in the plugin's settings.
+If you are using the [🌍 Internationalization (i18n)](https://docs.strapi.io/developer-docs/latest/plugins/i18n.html) plugin, an additional field `locale` can also be added in `entriesQuery`.
 
 **For example**
 
-Imagine my `restaurant` content-type has a relation with a repeatable-component `repeatableComponent` that itself has a relationship with the content-type `categories`.
-
-The following population will ensure that a `restaurant` entry contains even the most nested relation.
+For example, if you want your documents to be fetched in batches of `1000` you specify it in the `entriesQuery` option.
 
 ```js
 module.exports = {
   meilisearch: {
     config: {
       restaurant: {
-        populateEntryRule: ['repeatableComponent.categories', 'categories'],
+        entriesQuery: {
+          limit: 1000
+        }
       }
     }
   },
 }
 ```
 
-by providing this, the following is indexed in Meilisearch:
-
-```json
-  {
-    "id": "restaurant-1",
-    "title": "The slimmy snail",
-    // ... other restaurant fields
-    "repeatableComponent": [
-      {
-        "id": 1,
-        "title": "my repeatable component 1"
-        "categories": [
-          {
-            "id": 3,
-            "name": "Asian",
-            // ... other category fields
-          },
-          {
-            "id": 2,
-            "name": "Healthy",
-            // ... other category fields
-          }
-        ],
+[See resources](./resources/entries-query) for more entriesQuery examples.
 
-      }
-    ],
-
-  }
-```
 
 ### 🕵️‍♀️ Start Searching <!-- omit in toc -->
 

resources/entries-query/locale.js

@@ -0,0 +1,10 @@
+// All entries in every language are indexed in Meilisearch.
+module.exports = {
+  meilisearch: {
+    config: {
+      restaurant: {
+        locale: 'all',
+      },
+    },
+  },
+}

resources/entries-query/populate.js

@@ -0,0 +1,36 @@
+module.exports = {
+  meilisearch: {
+    config: {
+      restaurant: {
+        entriesQuery: {
+          populate: ['repeatableComponent.categories', 'categories'],
+        },
+      },
+    },
+  },
+}
+
+// The following document structure is indexed in Meilisearch
+// {
+//   id: 'restaurant-1',
+//   title: 'The slimmy snail',
+//   // ... other restaurant fields
+//   repeatableComponent: [
+//     {
+//       id: 1,
+//       title: 'my repeatable component 1',
+//       categories: [
+//         {
+//           id: 3,
+//           name: 'Asian',
+//           // ... other category fields
+//         },
+//         {
+//           id: 2,
+//           name: 'Healthy',
+//           // ... other category fields
+//         },
+//       ],
+//     },
+//   ],
+// }

resources/entries-query/publication-state.js

@@ -0,0 +1,10 @@
+// Both published and draft entries are added in Meilisearch
+module.exports = {
+  meilisearch: {
+    config: {
+      restaurant: {
+        publicationState: 'preview',
+      },
+    },
+  },
+}

server/__mocks__/meilisearch.js

@@ -0,0 +1,42 @@
+const addDocumentsMock = jest.fn(() => 10)
+const updateDocumentsMock = jest.fn(() => 10)
+const updateSettingsMock = jest.fn(() => 10)
+const deleteDocuments = jest.fn(() => {
+  return [{ taskUid: 1 }, { taskUid: 2 }]
+})
+const getIndexes = jest.fn(() => {
+  return { results: [{ uid: 'my_restaurant' }, { uid: 'restaurant' }] }
+})
+
+const getTasks = jest.fn(() => {
+  return {
+    results: [
+      { uid: 1, status: 'enqueued', indexUid: 'restaurant' },
+      { uid: 2, status: 'processed', indexUid: 'restaurant' },
+      { uid: 3, status: 'enqueued', indexUid: 'about' },
+    ],
+  }
+})
+
+const getStats = jest.fn(() => {
+  return { numberOfDocuments: 1, isIndexing: false, fieldDistribution: {} }
+})
+
+const mockIndex = jest.fn(() => ({
+  addDocuments: addDocumentsMock,
+  updateDocuments: updateDocumentsMock,
+  updateSettings: updateSettingsMock,
+  deleteDocuments,
+  getStats,
+}))
+
+// @ts-ignore
+const mock = jest.fn().mockImplementation(() => {
+  return {
+    getIndexes,
+    index: mockIndex,
+    getTasks,
+  }
+})
+
+module.exports = { MeiliSearch: mock }