evaluate NVIDIA-Nemotron-Nano-9B-v2 as a reranker

[sahel-sh]

evaluate the model as a listwise reranker out of the box with and without thinking.
compare to qwen3 or some other model that supports thinking on/off.

bonus: have throughput or latency comparisons too.

[sahel-sh]

use vllm for serving both models

[Raghav0005]

I'll be working on this task.

[sahel-sh]

Thanks @Raghav0005
it is yours

[r-aya]

Hi Sahel! Would also love to hop on as well. Happy to work with you on this, @Raghav0005

[sahel-sh]

It is actually good to compare two sets of runs. @r-aya please work independently, and compare results with @Raghav0005 afterwards.

[sahel-sh]

reproducibility is important in these types of work

[sahel-sh]

@r-aya
these are from my chat with @Raghav0005 , you might find them useful as well:

Hi Raghov, if you have not onboarded on rankllm, please do that first. It will give you a good starting point. Then look into the demo folder to get an idea of how it works (https://github.com/castorini/rank_llm/blob/55d5c1f4a49080e92af1b19a184ea57982ae9f8b/src/rank_llm/demo/rerank_qwen-r1-distill.py for the thinking version,

rank_llm/src/rank_llm/demo/experimental_results.py

Line 44 in 55d5c1f

RankListwiseOSLLM(

for non-thinking). The outcomes should be a PR with a similar demo that ranks with the nvidia model and results in the issues. You need to read more about the model though, for example you need a hint in the system prompt to toggle thinking for this model. Look into dl19 for now. Add your results to the git hub issue.

11:30
Some debug tips: work with one query only at the beginning rather than all retrieved results (

rank_llm/src/rank_llm/demo/experimental_results.py

Line 113 in 55d5c1f

retrieved_results = Retriever.from_dataset_with_prebuilt_index(dataset, k=100)

) and save the invocation history (

rank_llm/src/rank_llm/demo/experimental_results.py

Line 116 in 55d5c1f

kwargs = {"populate_invocations_history": True}

). Go through the saved prompts and completions, once it makes sense you can run for the entire dataset.

[Raghav0005]

I used the dataset dl19.
Results with a thinking prompt template:
ndcg_cut_1 all 0.5814
ndcg_cut_5 all 0.5240
ndcg_cut_10 all 0.5085

Ran with command:
python src/rank_llm/scripts/run_rank_llm.py --model_path nvidia/NVIDIA-Nemotron-Nano-9B-v2 --dataset dl19 --top_k_candidates 100 --prompt_template_path src/rank_llm/rerank/prompt_templates/nemotron_thinking_template.yaml --context_size 4096 --retrieval_method bm25

Results with non-thinking template:
ndcg_cut_1 all 0.5543
ndcg_cut_5 all 0.5282
ndcg_cut_10 all 0.5100

Ran with command:
python src/rank_llm/scripts/run_rank_llm.py --model_path nvidia/NVIDIA-Nemotron-Nano-9B-v2 --dataset dl19 --top_k_candidates 100 --prompt_template_path src/rank_llm/rerank/prompt_templates/nemotron_nonthinking_template.yaml --context_size 4096 --retrieval_method bm25

An interesting thing to note is that results do depend largely on the prompt template being used.

cc @r-aya @sahel-sh

[Raghav0005]

Added a PR for the demo with VLLM. The results for thinking/non-thinking with baseline and Nvidia are included in the PR.
Edit: I used the same prompt templates as Raya for consistency.

[sahel-sh]

@Raghav0005 your results in #298 (comment) are different from those reported in the PR: #313 (comment)
Please update one to be matching the other.
For the record my results for the nonthinking is close to what you have in the PR.
I already mentioned the issue with the thinking version, you need to pass the is_thinking=True to the model coordinator to allow for max output tokens>100