Skip to content

Releases: o19s/quepid

v8.5.0 - Changes coming fast and furious!

15 Jan 01:53
@epugh epugh
v8.5.0
1ba948d
This commit was signed with the committer’s verified signature.
epugh Eric Pugh
GPG key ID: E54EAD255F4A9678
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v8.5.0 - Changes coming fast and furious! Latest
Latest

Changes are coming fast and furious!

🔧 HTTP Request Infrastructure Overhaul

We've completely refactored how Quepid handles external HTTP requests by introducing a shiny new HttpClientService that consolidates all the scattered HTTP logic that was previously duplicated across ProxyController, DownloadPage, and MapperWizardService. This isn't just about cleaning up code (though we love that!) - it brings some sweet new capabilities too. The Mapper Wizard now supports custom headers and basic authentication credentials, making it much more flexible when fetching search results from different endpoints. Plus, we've made the proxy controller smarter about preserving content types for HTML responses.

📝 Judgment Explanations

Judges can now add explanations to their individual judgments! This is a simple but powerful addition that lets evaluators provide context about why they rated a particular result the way they did. This is useful for training your LLM as a Judge.

🔄 Case Evaluation API Integration

We've modernized how case evaluations are triggered by now supporting an case evaluation in response to an API call. This triggers a background job that executes all queries in the case against the search endpoint, collects the results, and calculates scores.

🎯 Smarter Rating Calculations

The way Quepid averages multiple judgments into final ratings got a major upgrade! We've implemented David Fisher's optimistic/pessimistic logic (you can read all about it in the official docs). The system now handles edge cases better, especially when you have just one or two judges rating the same result. No more weird averaging artifacts!

📚 Customizable Judgement Guidelines for a Book

Books now support custom judgement guidelines! While we have always had a generic set of guidelines to help a judge in rating, many teams want to provide their own specific flavour. The guidelines are written in markdown formatting too, so you can make them as detailed and pretty as you want.

🔍 Custom Search API Row Limits

Updated how Custom Search API work to ensure that row limits properly apply to custom search APIs. This was a behind-the-scenes fix to make sure that when you set limits on how many results to display, those limits are actually respected by the search backend. No more 40 results when you say you want 10.

What's Changed

  • Smarter averaging of ratings by @epugh in #1597
  • Let a user override the scorer guidelines for a book by @epugh in #1598
  • Run case as api call by @epugh in #1606
  • Add explanation to single judgment by @epugh in #1609
  • Refactor external requests to share logic by @epugh in #1605

Full Changelog: v8.4.0...v8.5.0

Contributors

epugh
Assets 2
Loading

8.4.0 - Two big things! And three nice things!

07 Jan 21:28
@epugh epugh
v8.4.0
af4dceb
This commit was signed with the committer’s verified signature.
epugh Eric Pugh
GPG key ID: E54EAD255F4A9678
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
8.4.0 - Two big things! And three nice things!

  1. We introduced a Search API Mapper Wizard that simplifies creating your Javascript "data mapping" functions to convert custom search api responses (including HTML and JSON formats) to those that Quepid expects. It even has a LLM assist for generating the code if you want. Thanks @mikehendo for testing this feature.

  2. We now support MULTIPLE Snapshots being Diffed! Comparing current search to various other points in time is one of the most common analysis that you want to do, but our Snapshot support has been, ahem, rough, to say the least. We completely redid all the logic, and now you can diff multiple snapshots at the same time. Oh, and we fixed all the little bugs folks had been experiencing.

And the nice things...

  1. We revamped how Books are configured so that they are simpler and easier to set up. No need to pick a "selection strategy" for how many judges you wil have or think about what scale to use in rating anymore. Plus we cleaned up some data flows between making judgements in a book

  2. We introduced rate limiting as an attribute of a Search Endpoint. THis ensures that if you need to constrain how fast you are sending the queires you can, to deal with AP rate limits, or just to reduce load on your search engine if you are testing in a live environment! See a video walkthrough.

  3. Search Endpoints have a lot of settings that maybe folks shouldn't poke around in. You can now set SEARCH_ENDPOINT_VIEWS_ADMIN_ONLY=true to prevent regular users from poking around. Thanks to @mikehendo for making the request.

  • Resolved mix of QUEPID_HOST and QUEPID_DOMAIN in favour of QUEPID_DOMAIN for specifying the domain of Quepid. Used in links generated in emails. 51cea26 by @epugh.
  • Option to prevent casual users from seeing search endpoint details. #1592 by @epugh fixes #1565 by @mikehendo.
  • Introduced configurable rate limiting to keep proxied search endpoints from getting overwhelmed by @epugh in #1587.
  • Added LLM-powered generation of data mapper Javascript to speed up custom API onboarding by @epugh in #1285.
  • Swapped to released oas_rails gem and cleaned up OpenAPI references (bumped to 3.4.8) by @epugh in #1589.
  • Propshaft replaces Sprockets, plus importmap cleanups for a leaner asset pipeline by @epugh in #1540.
  • Snapshot diffing is sturdier (again) so comparing runs behaves under edge cases by @epugh in #1566.
  • Books now carry their own scale and default to the multiple judges strategy, reducing scorer coupling by @epugh in #1563 and #1561.
  • Home page now shows annotations, and bulk judging picked up tests plus safer delete handling by @epugh in #1488.
  • Dependencies: bcrypt 3.1.21, rubocop 1.82.1, faraday-retry 2.4.0, bootsnap 1.20.x, pagy 43.2.2, scout_apm 6.0.2, selenium-webdriver 4.39.0, omniauth-rails_csrf_protection 2.0.1, annotaterb 4.20.0, dotenv 3.2.0, database_consistency 2.1.1 by @dependabot in assorted PRs (#1586, #1578, #1580, #1579, #1577, #1573, #1574, #1575, #1562, #1567, #1558, #1555, #1564).

What's Changed

Full Changelog: v8.3.7...v8.4.0

Contributors

epugh, dependabot, and mikehendo
Loading

8.3.7

04 Dec 02:23
@epugh epugh
v8.3.7
40cadd6

Choose a tag to compare

View all tags
8.3.7

What's Changed

  • Fixed issue in Bulk Judging when you paginate and you don't have enough docs left due to rating activity to get results by @epugh in #1554. Thanks @david-fisher for finding this!
  • Default to Ollama and Qwen model for local LLM as a Judge when doing development by @epugh in #1553

And a million dependency updates

Full Changelog: v8.3.6...v8.3.7

Contributors

epugh, david-fisher, and 2 other contributors
Loading

8.3.6 -- 2025-11-20

20 Nov 13:39
@epugh epugh
v8.3.6
459cf09
This commit was signed with the committer’s verified signature.
epugh Eric Pugh
GPG key ID: E54EAD255F4A9678
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
8.3.6 -- 2025-11-20

8.3.6 -- 2025-11-20

  • LLMService builds the wrong URL for Google’s Gemini (OpenAI-compatible) API. Thanks @oskrocha for reporting #1519 and @lauzel for fixing in #1521.

Sorry it took a few weeks to release Quepid with the fix!

What's Changed

New Contributors

Full Changelog: v8.3.5...v8.3.6

Contributors

oskrocha and lauzel
Loading

8.3.5 -- 2025-10-24

27 Oct 11:44
@epugh epugh
v8.3.5
d75233f
This commit was signed with the committer’s verified signature.
epugh Eric Pugh
GPG key ID: E54EAD255F4A9678
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
8.3.5 -- 2025-10-24

8.3.5 -- 2025-10-24

There were a number of 8.3.x releases as we worked through some debugging. 8.3.5 though is the one to use!

  • Query Doc Pair query_text should be case sensitive, the same way Query query_text is, but it wasn't. So you couldn't have two queries with differing case like "Nike" and "nike" in a Book. Thanks @david-fisher for finding #1512 and fixed in #1513.

  • Query Doc Pairs for a book are synced when you run a linked Case in the background! Previously we sent all the data in one huge JSON file that was processed immediately. Not we submit in chunks of 100 queries, and the data is loaded in the background async to the book, smoothing the workflow. This happens automagically for you, no need to click the "populate book" button either in the Judgements modal. #1496 by @epugh.

  • Fix bulk judgements UI under nested deployment routes. Pin docker image to debian Bookworm for now. #1495

What's Changed

Full Changelog: v8.3.0...v8.3.5

Contributors

epugh, david-fisher, and dependabot
Loading

v8.3.0 - now with bulk judging!

06 Oct 14:44
@epugh epugh
v8.3.0
29da6a0
This commit was signed with the committer’s verified signature.
epugh Eric Pugh
GPG key ID: E54EAD255F4A9678
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v8.3.0 - now with bulk judging!

8.3.0 -- 2025-10-06

Features

  • Bulk Judging UI added for Books. Want to quickly judge lots of documents for a set of queries in context of each other? Bulk Judging UI is here. You can pick a specific depth, say only results in the top 3. Or you can search for specific query patterns. Results are presented in random order to counteract position bias in judging. You can also provide an explanation. #1490 by @epugh, thanks Joelle for inspiring this to finally happen.

  • Added ability to Archive a book, similar to how you can Archive a case. This lets you hide books that you aren't currently using. #1487

  • Developers using Docker no longer need Ruby to be installed locally. Scripts have been converted to Bash. #1479

Improvements

  • Improved syncing of Books and Cases by making it a back ground process if you have more than 50 queries. Deals with the front end becomign sluggish and timing out when it takes a long time to sync. #1487

  • Simplify book creation by defaulting to Multiple Raters choice. #1487

  • All the updates across all the dependencies! #1479

  • Finally giving up on smart handling of HTTPS/HTTP protocol IN Quepid. We tried for a long time to make Quepid aware of a search endpoints TLS setting because the browsers these days don't want you to interact across the protocols. If Quepid was on https and the search endpoint was an http url, then we prompted to reload the browser app into the matching http url. This didn't work well when Quepid is behind a proxy, as they typically take of this. Also, we now have an actual built in capability for Quepid server to send queries to an endpoint, and it doesn't care if it's http or https. #1427 by @epugh to remove this. Fixes #1410 by @brucks24.

Bugs Addressed

  • Restored Annotation view in the Case screens Scores over time chart. Fixed some visualization issues IN that chart too. #1479

What's Changed

Full Changelog: v8.2.1...v8.3.0

Contributors

epugh, dependabot, and brucks24
Loading

8.2.1 -- 2025-08-25

25 Aug 19:15
@epugh epugh
v8.2.1
34ff451
This commit was signed with the committer’s verified signature.
epugh Eric Pugh
GPG key ID: E54EAD255F4A9678
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
8.2.1 -- 2025-08-25

@david-fisher did some testing and found a couple of regressions:

  • Fixed navbar drop down on Admin pages. #1459
  • Version numbers not showing up in the Dockerized version of Quepid. 632b73c
  • Integrate into entrypoint.sh handling Quepid assests in a nested directory like www.mycompany.com/quepid-prod/.

@jesigger fixed the dropdowns for recent cases and books! #1415

Lastly, nicer docs on deploying Quepid locally: https://quepid-docs.dev.o19s.com/2/quepid/61/how-to-deploy-quepid-locally.

8.2.1 -- 2025-08-25

What's Changed

New Contributors

Full Changelog: v8.2.0...v8.2.1

Contributors

epugh, jesiggar, and 3 other contributors
Loading

8.2.0 -- 2025-07-23 - Welcome quepidapp.com as the domain name!

23 Jul 19:28
@epugh epugh
v8.2.0
c61a478
This commit was signed with the committer’s verified signature.
epugh Eric Pugh
GPG key ID: E54EAD255F4A9678
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
8.2.0 -- 2025-07-23 - Welcome quepidapp.com as the domain name!

What's Changed

  • The biggest thing folks may notice is that we've moved from the domain name quepid.com to quepidapp.com. This has required lots of updates in documentation, build tooling, and various links scattered around the Quepid codebase.

  • Lots of refinement to the Continuous Experimentation support released in 8.1.0. Quepid will run cases marked "Nightly" and store the score. It also stores the last FIVE snapshots including document fields for additional analysis. We are limiting Snapshots to 5 to make sure we don't destroy the database ;-), and could evaluate that later. When you load the case, we still rerun all the queries again, however there is some preliminary work about just loading the last snapshot for these nightly cases: #1323.

  • Lots of improvements around the LLM-as-a-Judge. Encrypted LLM keys in the database, nicer UI when testing out prompts, ability to evaluate images in the judging process.

  • Brand new approach to documenting APIs! OpenAPI 3.1 specification compliant docs, and you can test out API calls right there in Quepid, or get the curl equivalent. Thank you Apipie for your previous service, and I'm excited about the OasRails project. The lead developer, Andrés has been great to work with.

Features

  • Nightly Evaluation Support. #1288 and #1289.

  • You can now generate an API key for a user from the commandline: bundle exec thor user:add_api_key EMAIL. Thanks @frutik for the contribution in #1408

  • Nightly build and push to DockerHub of Quepid image. Thanks @frutik for #1369.

  • A new Scorer! Welcome NDCG_CUT@10! NDCG is properly calculated on all the documents that have been rated for a query. However, often we want to only look at the top 10, and cut off evaluation there, ignoring all the rest. That is when you want NDCG_CUT@10. If you had been using NDCG@10 then you may want to swap.

  • API to get the latest snapshot for a case. Supporting a custom evaluation frontend built by @sjarmak. Thanks @david-fisher for encouragement. #1312 by @epugh.

Improvements

Bugs Addressed

  • When comparing snapshots, if you had basic auth set up then those credentials were not passed in. Thanks @kiratraynor for finding this, fixed in #1411 by @epugh.

  • Judgements in a Book don't HAVE to have a user id, so fix the UI: 178594f by @epugh.

  • Fix Quepid API to support the Jupyter Inter Rater Reliablity notebook. #1294 by @epugh.

What's Changed

Read more

Contributors

epugh, frutik, and 9 other contributors
Loading

8.1.0 - Polished version of 8!

10 Mar 15:09
@epugh epugh
v8.1.0
fa0156b
This commit was signed with the committer’s verified signature.
epugh Eric Pugh
GPG key ID: E54EAD255F4A9678
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
8.1.0 - Polished version of 8!

8.1.0 -- 2024-02-28

We've had lots of folks trying new features of Quepid, and that has led to some nice polish.
This is the release for people who don't want the bleeding edge version ;-).

What's Changed

  • Special characters like $ and @ in Basic Auth passwords are now supported. Thanks to @atarora for opening #1245 and the debugging work from @david-fisher.
  • UX Work.
  • We have added filtering and pagination to all of the various screens like Seach Endpoints, Communal Scorers, Users, Judgements etc. This makes these pages load MUCH faster and you can search for the data you need.
  • Consistency across web pages. Outside of the core AngularJS app, we had a mish mash of UI's. Now they all have the same style header, footer, and UX treatment using card layouts.
  • Fixed the nDCG calculation to look at ALL ratings, not just the top k when doing calculation.
  • Fixed a goof where the new ERR scorer wasn't added as a communal scorer for everyone.
  • LLM as a Judge is better. Cleaned up the refine UI for prompt development.
  • Fix a bug where the Judge couldn't be launched from the UI.

Details

Full Changelog: v8.0.1...v8.1.0

Contributors

epugh, david-fisher, and 3 other contributors
Loading

8.0.1 means now you can deploy Quepid 8!

19 Feb 20:09
@epugh epugh
v8.0.1
4931635
This commit was signed with the committer’s verified signature.
epugh Eric Pugh
GPG key ID: E54EAD255F4A9678
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
8.0.1 means now you can deploy Quepid 8!

8.0.1 -- 2024-02-19

Whelp, that didn't take long!

We broke support for running Quepid on non standard ports like :3000. #1229 by @epugh fixes this.

What's Changed

Full Changelog: v8.0.0...v8.0.1

Contributors

epugh and dependabot
Loading