Skip to content

Fix perplexity baseline to actually search the web#92

Open
nbalepur wants to merge 2 commits intoallenai:mainfrom
nbalepur:perplexity_fix
Open

Fix perplexity baseline to actually search the web#92
nbalepur wants to merge 2 commits intoallenai:mainfrom
nbalepur:perplexity_fix

Conversation

@nbalepur
Copy link
Copy Markdown
Contributor

Adding logic so the perplexity baseline can search the web, namely by:

  1. Setting search_mode parameter to "" by default
  2. Adding reasoning_effort as another parameter in the solver
  3. Making an extra solver between Perplexity's and Gemini's formatted_solver that adds the references to the end of the prompt (called add_perplexity_references)

With this repo, I re-ran the scores for SQA and Litqa2, respectively:
image
image

Which are quite similar to what I shared earlier on slack for these two benchmarks (some variance but nothing crazy imo):

image image

Copy link
Copy Markdown
Contributor

@amanpreet692 amanpreet692 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants