docs: update index page with new features #51
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,11 +19,12 @@ OpenJudge unifies evaluation metrics and reward signals into a single, standardi | |
| - **Quality Assurance:** Built for reliability. Every grader comes with benchmark datasets and pytest integration for immediate quality validation. <a href="https://huggingface.co/datasets/agentscope-ai/OpenJudge" class="feature-link" target="_blank"> View Benchmark Datasets<span class="link-arrow">→</span></a> | ||
|
|
||
| + **Flexible Grader Building**: Choose the build method that fits your requirements: | ||
| - **Customization:** Clear requirements, but no existing grader? If you have explicit rules or logic, use our Python interfaces or Prompt templates to quickly define your own grader. <a href="building_graders/create_custom_graders/" class="feature-link">Custom Grader Development Guide <span class="link-arrow">→</span></a> | ||
| - **Zero-shot Rubrics Generation:** Not sure what criteria to use, and no labeled data yet? Just provide a task description and optional sample queries—the LLM will automatically generate evaluation rubrics for you. Ideal for rapid prototyping. <a href="building_graders/generate_rubrics_as_graders/#simple-rubric-zero-shot-generation" class="feature-link">Zero-shot Rubrics Generation Guide <span class="link-arrow">→</span></a> | ||
| - **Data-driven Rubrics Generation:** Ambiguous requirements, but have few examples? Use the GraderGenerator to automatically summarize evaluation Rubrics from your annotated data, and generate a llm-based grader. <a href="building_graders/generate_rubrics_as_graders/#iterative-rubric-data-driven-generation" class="feature-link">Data-driven Rubrics Generation Guide <span class="link-arrow">→</span></a> | ||
| - **Training Judge Models:** Massive data and need peak performance? Use our training pipeline to train a dedicated Judge Model. This is ideal for complex scenarios where prompt-based grading falls short. <a href="building_graders/training_judge_models/" class="feature-link">Train Judge Models <span class="link-arrow">→</span></a> | ||
|
|
||
| + **Easy Integration**: Using mainstream observability platforms like **LangSmith** or **Langfuse**? We offer seamless integration to enhance their evaluators and automated evaluation capabilities. We're also building integrations with training frameworks like **verl**. | ||
|
|
||
| </div> | ||
|
|
||
|
|
@@ -33,26 +34,38 @@ OpenJudge unifies evaluation metrics and reward signals into a single, standardi | |
|
|
||
| <div class="card-grid"> | ||
|
|
||
| <a href="applications/zero_shot_evaluation/" class="feature-card-sm"> | ||
| <div class="card-header"> | ||
| <img src="https://unpkg.com/lucide-static@latest/icons/zap.svg" class="card-icon card-icon-general"> | ||
| <h3>Zero-Shot Evaluation</h3> | ||
| </div> | ||
| <p class="card-desc"> | ||
| <b>Compare models without test data:</b> Generate queries, collect responses, and rank via pairwise evaluation. | ||
| </p> | ||
| </a> | ||
|
|
||
| <a href="get_started/evaluate_ai_agents/" class="feature-card-sm"> | ||
| <div class="card-header"> | ||
| <img src="https://unpkg.com/lucide-static@latest/icons/bot.svg" class="card-icon card-icon-agent"> | ||
| <h3>Evaluate An AI Agent</h3> | ||
| </div> | ||
| <p class="card-desc"> | ||
| <b>Agent lifecycle evaluation:</b> Assess response, trajectory, tool usage, planning, memory, and reflection. | ||
| </p> | ||
| </a> | ||
|
|
||
| <a href="get_started/build_reward/" class="feature-card-sm"> | ||
| <div class="card-header"> | ||
| <img src="https://unpkg.com/lucide-static@latest/icons/brain-circuit.svg" class="card-icon card-icon-tool"> | ||
| <h3>Build Rewards for Training</h3> | ||
| </div> | ||
| <p class="card-desc"> | ||
| <b>Quality reward signals:</b> Aggregate graders with custom weighting for model alignment. | ||
| </p> | ||
| </a> | ||
|
|
||
|
|
||
|
|
||
|
Comment on lines
+67
to
+68
Contributor
There was a problem hiding this comment. These empty lines appear to be unnecessary and can be removed for better code cleanliness. |
||
| </div> | ||
|
|
||
|
|
||
|
|
@@ -141,41 +154,49 @@ OpenJudge unifies evaluation metrics and reward signals into a single, standardi | |
|
|
||
| <a href="building_graders/generate_rubrics_as_graders/" class="feature-card-sm"> | ||
| <div class="card-header"> | ||
| <img src="https://unpkg.com/lucide-static@latest/icons/sparkles.svg" class="card-icon card-icon-data"> | ||
| <h3>Generate Rubrics</h3> | ||
| </div> | ||
| <p class="card-desc"> | ||
| <b>Auto-generate evaluation criteria.</b> Use Zero-Shot generation from task descriptions, or Data-Driven generation to learn rubrics from labeled preference data. | ||
| </p> | ||
| </a> | ||
|
|
||
| <a href="building_graders/training_judge_models/" class="feature-card-sm"> | ||
| <div class="card-header"> | ||
| <img src="https://unpkg.com/lucide-static@latest/icons/scale.svg" class="card-icon card-icon-integration"> | ||
| <h3>Train Judge Models</h3> | ||
| </div> | ||
| <p class="card-desc"> | ||
| <b>Massive data and need peak performance?</b> Train dedicated judge models using SFT, Bradley-Terry, or GRPO. Supports both scalar rewards and generative evaluation with reasoning. | ||
| </p> | ||
| </a> | ||
|
|
||
| </div> | ||
|
|
||
| ### Integrations | ||
|
|
||
| <div class="card-grid"> | ||
|
|
||
| <a href="integrations/langsmith/" class="feature-card"> | ||
| <div class="card-header"> | ||
| <img src="https://unpkg.com/lucide-static@latest/icons/telescope.svg" class="card-icon card-icon-integration"> | ||
| <h3>LangSmith</h3> | ||
| </div> | ||
| <p class="card-desc"> | ||
| Build external evaluation pipelines for LangSmith. Wrap OpenJudge graders as LangSmith evaluators and run batch evaluations with GradingRunner. | ||
| </p> | ||
| </a> | ||
|
|
||
| <a href="integrations/langfuse/" class="feature-card"> | ||
| <div class="card-header"> | ||
| <img src="https://unpkg.com/lucide-static@latest/icons/activity.svg" class="card-icon card-icon-data"> | ||
| <h3>Langfuse</h3> | ||
| </div> | ||
| <p class="card-desc"> | ||
| Fetch traces from Langfuse, evaluate with OpenJudge graders, and push scores back. Supports batch processing and score aggregation. | ||
| </p> | ||
| </a> | ||
|
|
||
| <div class="feature-card-wip"> | ||
| <div class="card-header"> | ||
|
|
@@ -184,7 +205,7 @@ OpenJudge unifies evaluation metrics and reward signals into a single, standardi | |
| <span class="badge-wip">🚧 Work in Progress</span> | ||
| </div> | ||
| <p class="card-desc"> | ||
| Directly integrate into training loops such as <strong>VERL</strong>. Use Graders as high-quality reward functions for fine-tuning to align models effectively. | ||
| </p> | ||
| </div> | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -44,7 +44,7 @@ | |
| /* Three column cards */ | ||
| .feature-card-sm { | ||
| flex: 1 1 30%; | ||
| min-width: 280px; | ||
| text-decoration: none; | ||
| color: inherit; | ||
| border: 1px solid var(--md-default-fg-color--lightest, #e0e0e0); | ||
|
|
@@ -101,7 +101,7 @@ | |
|
|
||
| .card-header h3 { | ||
| margin: 0 !important; | ||
| font-size: 15px; | ||
| font-weight: 600; | ||
| white-space: nowrap !important; | ||
| display: inline !important; | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For better readability and consistency, it's good practice to add a space before the arrow emoji (👉).