Can LLM Annotations Replace User Clicks for Learning to Rank?

This repo implements the source code of our paper "Can LLM Annotations Replace User Clicks for Learning to Rank?".

Dataset

We utilized the publicly available dataset TianGong-ST and a proprietary dataset Baidu-Click collected from the Baidu search engine.

Usage

• Run different LLM-based annotation methods

1. Given the annotation criterion:

    cd llm_annotation
    sh run.sh (pointwise: annotation.py; listwise: annotation_list.py & grade = 'multi')

2. Direclty select:

    cd llm_annotation
    sh run.sh (annotation_list.py & grade = 'binary')

3. Directly rank:

    cd llm_rank
    sh run_list.sh

• Train models using different annotation

1. LLM Annotation-Supervised Training
    
    cd llm_click
    sh run_llm.sh

2. Click Supervised Training (Unbiased Learning to Rank)

    cd llm_click
    sh run_click.sh

3. Hybrid Training

    cd llm_click
    1) Frequency-Aware Multi-Objective Learning: sh run_multi.sh
    2) Data Scheduling: sh run_multi_stage_freq.sh