Semantic Search

The semantic search module provides scalable functionality to enhance the search capabilities using semantic understanding.

The IntelliNode SemanticSearch module offers two key features:

1. Semantic Search: This applies contextual search operations by converting documents into vectors and applying semantic understanding using multiple providers such as OpenAI and Cohere.

2. Semantic Search Paging: For larger datasets, apply the semantic search in iterations, comparing the top 'n' matches at a time. This ensures efficient memory usage and faster execution time even for large-scale data.

Semantic Search

1. Start by importing the SemanticSearch from IntelliNode:

   const { SemanticSearch } = require('intellinode');

2. Specify the search variables:

• pivotItem: This term serves as the reference string in the search.

• searchArray: This is an array of items or phrases that the system will search through to find the best semantic matches to the pivot item.

• numberOfMatches: the number of top matches you want to get.

  const pivotItem = 'Hello from OpenAI!';
  const searchArray = ['Greetings from OpenAI!', 'Bonjour de OpenAI!', 'Hola desde OpenAI!'];
  const numberOfMatches = 2;

3. Instantiate SemanticSearch with the desired model's API key and provider (openai or cohere):

   const apiKey = '<your-provider-key>'
   const provider = 'openai' // or 'cohere'
   const search = new SemanticSearch(apiKey, provider);

4. Call the getTopMatches function to get the scores of the array indices:

 const results = await search.getTopMatches(pivotItem, searchArray, numberOfMatches);
 console.log('Semantic Search Results:', results);

6. To get only top matches content from the scores results you can use:

   console.log('top matches:', search.filterTopMatches(results, searchArray));

Semantic Search Pagination

For larger data sets, IntelliNode provides a SemanticSearchPaging feature. This allows memory-efficient semantic search by working in iterations and keeping the top 'n' matches.

1. Import the required SemanticSearchPaging and SupportedEmbedModels from IntelliNode:

   const { SemanticSearchPaging, SupportedEmbedModels } = require('intellinode');

2. Instantiation of SemanticSearchPaging remains the same as above but takes the number of desired matches and the pivot item in the constructor:

• pivotItem: This term serves as the reference string in the search.

• numberOfMatches: the number of top matches you want to get.

  const search = new SemanticSearchPaging(apiKey, provider, pivotItem, numberOfMatches);

3. You can add new arrays in your search memory using addNewData function:

   // this is only a simulation of data load, in a real app you will have a loop on your data source provider.
   
   const searchArray1 = ['Greetings from IntelliNode!', 'Bonjour de IntelliNode!', '来自 IntelliNode 的问候！'];
   const searchArray2 = ['Saudações do IntelliNode!', 'Hola desde IntelliNode!', 'Groeten van IntelliNode!'];
   
   await search.addNewData(searchArray1);
   await search.addNewData(searchArray2);

4. Finally, to get your top matches, call getCurrentTopMatches:

   const results = await search.getCurrentTopMatches();
   console.log('Semantic Search Results:', results);

• Output example:

  [{ text: 'Greetings from IntelliNode!', score: 0.9966673636144003 },
  { text: 'Hola desde IntelliNode!', score: 0.9596313682845611 }]

Remember to wrap these functions inside an async function to handle promise returns properly.

Visit the following links to access these sample codes:

• Sample code for SemanticSearch.
• Sample code for SemanticSearchPaging.