Make WebResearchRetriever class agnostic of the search engine

[francisco-perez-sorrosal]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

The current implementation of the WebResearchRetriever class is very tied to the Google Search API
as it uses directly the GoogleSearchAPIWrapper in the search attribute

I think it would be nice to have a more generic implementation of the WebResearchRetriever that can be used with other search engines like DuckDuckGo or Bing.

Motivation

I had to use the `WebResearchRetriever but I found out that it's very tied to Google Search. I didn't want to use Google as it requires API KEYS, so I had to end up hacking the injection of the DDG Wrapper directly in a personal project to use the DuckDuckGo Wrapper with the retriever. DuckDuckGo for example doesn't require any API keys (unlike Google) and it's free to use and useful when prototyping or testing. The proposal would avoid those kind of dirty hacks that are not maintainable nor portable by allowing to inject the a compatible wrapper. With the current hack I did I was able to check that the retriever works good with other search engine wrappers like DDG too.

Proposal (If applicable)

The scaffolding of what I propose is this:

1. Create a new file (e.g. search_base.py) in the langchain community utilities package, with the SearchAPIWrapperBase class that defines the interface for the search API wrappers.

from abc import ABC, abstractmethod
from typing import Any, Dict, List, Optional

from pydantic import BaseModel, Extra


class SearchAPIWrapperBase(BaseModel, ABC):
    """Abstract base class for search API wrappers.
    e.g. GoogleSearchAPIWrapper, DuckDuckGoSearchAPIWrapper..."""

    class Config:
        extra = Extra.forbid

    search_engine_name: str = Field(..., description="Search engine name")

    @abstractmethod
    def run(self, query: str) -> str:
        """Run query through the search engine and return concatenated results."""
        pass

    @abstractmethod
    def results(self, query: str, max_results: int, **search_params: Any) -> List[Dict[Any, Any]]:
        """Run query through the search engine and return metadata.

        Args:
            query: The query to search for.
            max_results: The maximum number of results to return.
            **search_params: Additional search arguments specific to each implementation.

        Returns:
            A list of dictionaries with metadata about each result.
        """
        pass

2. Make it visible in the __init__.py of the utilities package.
3. Import it the web_research.py and make the WebResearchRetriever to use the SearchAPIWrapperBase instead of the GoogleSearchAPIWrapper and
   eliminate all the specific references to Google search as it would be generic.

from langchain_community.utilities import SearchAPIWrapperBase
...
class WebResearchRetriever(BaseRetriever):
    """Search retriever."""

    # Inputs
    vectorstore: VectorStore = Field(
        ..., description="Vector store for storing web pages"
    )
    llm_chain: LLMChain
    search: SearchAPIWrapperBase = Field(..., description="Search API Wrapper")
    ...

4. Adapt also the prompts in the web_research.py to be search engine agnostic.

DEFAULT_LLAMA_SEARCH_PROMPT = PromptTemplate(
    input_variables=["question", "search_engine"],
    template="""<<SYS>> \n You are an assistant tasked with improving {search_engine} search \
results. \n <</SYS>> \n\n [INST] Generate THREE {search_engine} search queries that \
are similar to this question. The output should be a numbered list of questions \
and each should have a question mark at the end: \n\n {question} [/INST]""",
)

DEFAULT_SEARCH_PROMPT = PromptTemplate(
    input_variables=["question", "search_engine"]],
    template="""You are an assistant tasked with improving {search_engine} search \
results. Generate THREE {search_engine} search queries that are similar to \
this question. The output should be a numbered list of questions and each \
should have a question mark at the end: {question}""",
)

# And later in the code where the prompts are used:

    def _get_relevant_documents(
        self,
        query: str,
        *,
        run_manager: CallbackManagerForRetrieverRun,
    ) -> List[Document]:

        # Get search questions
        search_eng_name = self.search.search_engine_name
        logger.info(f"Generating questions for {search_eng_name} Search ...")
        result = self.llm_chain({"question": query, "search_engine": search_eng_name})

5. In the wrappers, initialize the attribute (search_engine_name) and adapt the methods to the signatures of the abstract class. For example, in the case of the DuckDuckGoSearchAPIWrapper:

    search_engine_name: str = "DuckDuckGo"
    ...
    def results(
        self, query: str, max_results: int, **search_params: Any
    ) -> List[Dict]:
        """Run query through DuckDuckGo and return metadata.
        ...
        source = search_params.get('source', None) or self.source
        ...

6. Test it

Am I missing something? If you are interested in this approach, I can help you to implement it, I have a WIP which I can turn into a PR that I can share with you to validate the idea.

Thanks for your attention!!!