Minor prompt variation causes model to fail retrieving information from the web

[ahmedkooli]

Hey ! I am having trouble fetching information from the web. I noticed that a very slight change in my prompt can cause the whole web search to fail. My use-case consists in fetching information on a company given its name and its website URL. Here's one of the most simple prompts examples:

# Prompts
company_name = "Association Delphinea"
website_url = "https://associationdelphinea.com"

user_prompt_1 = f"""
Find detailed information on the following company:
Company name: `{company_name}`
Company website URL: `{website_url}`
"""

user_prompt_2 = f"""
Find detailed information on the following company:
Company name: `{company_name}`
Company website URL: `{website_url}`

Describe your findings.
"""

Strangely enough, with user_prompt_1 Perplexity outputs very thorough information about the searched company. However things change completely with user_prompt_2, the outputs basically starting like this:

I couldn't find any specific information about a company named "Association Delphinea" or its website at `https://associationdelphinea.com`/. The search results provided do not mention this
company, and there is no available data on it in the public domain based on the search results.

Also, I am not using any parameter in my API request.

# API call
import requests

url = "https://api.perplexity.ai/chat/completions"

payload = {
    "model": "sonar",
    "messages": [
        {
            "role": "user",
            "content": user_prompt_1
        }
    ]
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.request("POST", url, json=payload, headers=headers)

I am stuck here as it seems that a very small additional instruction destroys the web search. Any kind of help would be kindly appreciated, thanks a lot !

[james-pplx]

Hey @ahmedkooli ! This is a very interesting phenomenon you're coming across here, which goes back to general best practices of prompt engineering, which at times may be more art than science :) .

Why I believe this happens

At the high level, our retrieval system operates in two key phases:

1. Query Parsing & Web Search – The model extracts relevant terms and runs a search.
2. Response Generation – The model synthesizes retrieved data into a coherent response.

When using user_prompt_1, Sonar focuses purely on retrieving information about the company (via the desired web search). However, adding "Describe your findings." in user_prompt_2 might be shifting the model’s focus away from direct retrieval and toward a more generative response, which can lead to it relying more on internal knowledge.

What I suggest to fix it

Try one of these approaches:

1. Explicitly instruct the Model to search before summarizing
   Instead of "Describe your findings.", use:

   user_prompt_2 = f"""
   Perform a web search to find detailed information on the following company:
   Company name: `{company_name}`
   Company website URL: `{website_url}`
   
   Summarize the key details found from the search.
   """

   This reinforces the retrieval step before synthesis by explicitly asking Sonar to search the web.

2. Use a model designed for research
   Since your use case involves in-depth company research, consider using sonar-deep-research instead of sonar.

   • sonar is a lightweight model optimized for quick queries.
   • sonar-deep-research is designed for exhaustive web searches, reading multiple sources, and synthesizing insights into comprehensive reports.

You can read about all our models here.

Example API request using sonar-deep-research:

payload = {
    "model": "sonar-deep-research",
    "messages": [{"role": "user", "content": user_prompt_1}]
}

3. Use separate API Calls for retrieval and summarization
   I would only use this as a last resort as it would double your costs. If the model isn't retrieving properly in one step, first fetch search results, then summarize them in a second request:

   # Step 1: Retrieve raw search results
   payload = {
       "model": "sonar",
       "messages": [{"role": "user", "content": f"Search for `{company_name}` at `{website_url}` and provide raw findings."}]
   }
   response = requests.post(url, json=payload, headers=headers)
   search_results = response.json()["choices"][0]["message"]["content"]
   
   # Step 2: Summarize the findings
   payload = {
       "model": "sonar",
       "messages": [{"role": "user", "content": f"Summarize the following search results:\n{search_results}"}]
   }
   response = requests.post(url, json=payload, headers=headers)
   print(response.json()["choices"][0]["message"]["content"])

[ahmedkooli]

Thank you @james-pplx for your quick and interesting reply :) I tried your first prompt and other variations but it's not helping. sonar-deep-search however is able to find the answer but is much slower, which is why I'll go with the third option which is what I had implemented as a fallback (I value time over money for this use-case).

Anyway, it's indeed closer to art than science... thanks again !

[james-pplx]

My pleasure helping you, @ahmedkooli ! We would also really appreciate your feedback on our interaction if this was helpful. Thanks for being a valued Sonar customer!