feat:llama3.1-8b accuracy by attafosu · Pull Request #100 · mlcommons/endpoints

attafosu · 2026-01-20T06:49:12Z

What does this PR do?

Adds config for accuracy run
Requires #97

Rouge scores:

Legacy-loadgen ref: {"rouge1": 38.78, "rouge2": 15.91, "rougeL": 24.5, "rougeLsum": 35.79, "gen_len": 8167644, "gen_num": 13368}
Endpoint-client: {"rouge1": 39.96, "rouge2": 16.61, "rougeL": 25.23, "rougeLsum": 36.87, "gen_len": 8095741, "gen_num": 13368}

Type of change

Bug fix
New feature
Documentation update
Refactor/cleanup

Related issues

Resolves #56

Testing

Tests added/updated
All tests pass locally
Manual testing completed

Checklist

Code follows project style
Pre-commit hooks pass
Documentation updated (if needed)

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

github-actions · 2026-01-20T06:49:21Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

gemini-code-assist · 2026-01-20T06:49:31Z

Summary of Changes

Hello @attafosu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the evaluation framework for the Llama 3.1-8B model by integrating dedicated accuracy testing capabilities. It updates existing benchmark configurations to align with '.jsonl' dataset formats, fine-tunes generation parameters for consistency, and refines runtime and client settings to improve the reliability and efficiency of both performance and accuracy measurements. The changes are designed to provide a more robust system for assessing the model's summarization quality using ROUGE metrics.

Highlights

Accuracy Configuration: Introduced new configurations to enable accuracy runs for the Llama 3.1-8B model, specifically for ROUGE score evaluation.
Dataset Format Update: Standardized dataset file extensions from '.json' to '.jsonl' across relevant scripts and configurations.
Parameter Tuning: Adjusted model parameters, such as setting 'temperature' to '0.0' in both offline and online configurations for deterministic generation.
Runtime and Client Adjustments: Modified runtime durations, increased client workers, and added 'n_samples_to_issue' in both offline and online configurations for more controlled and efficient benchmarking.
Online Load Pattern Change: Switched the online load pattern from 'poisson' to 'concurrency' with a target concurrency of 128.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

Copilot

Pull request overview

This PR adds configuration for running accuracy benchmarks on the Llama 3.1-8B model using the CNN/DailyMail dataset. The changes update the example configurations to support accuracy evaluation with ROUGE scoring and adjust various runtime parameters for proper accuracy measurement.

Changes:

Updated model and dataset configurations to support accuracy evaluation with ROUGE metrics
Changed temperature from 1.0 to 0.0 for deterministic generation during accuracy runs
Modified dataset file format from .json to .jsonl across all configurations

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
examples/05_Llama3.1-8B_Example/online_llama3_8b_cnn.yaml	Added accuracy dataset configuration with ROUGE evaluation, updated load pattern to concurrency-based, and changed file format to jsonl
examples/05_Llama3.1-8B_Example/offline_llama3_8b_cnn.yaml	Updated to use jsonl format, adjusted runtime durations, and increased worker count
examples/05_Llama3.1-8B_Example/download_cnndm.py	Changed output file extension from .json to .jsonl

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

examples/05_Llama3.1-8B_Example/online_llama3_8b_cnn.yaml

examples/05_Llama3.1-8B_Example/offline_llama3_8b_cnn.yaml

gemini-code-assist

Code Review

This pull request adds configuration for Llama3.1-8B accuracy runs. The changes are generally good, updating file formats to JSONL, adjusting model parameters for deterministic output, and setting up datasets for accuracy and performance tests. My review focuses on improving the portability and clarity of the example configuration files. I've suggested making the model path more generic and refactoring the dataset configuration to remove redundancy and use a more descriptive name.

examples/05_Llama3.1-8B_Example/online_llama3_8b_cnn.yaml

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/dataset_manager/predefined/cnndailymail/__init__.py

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-22T00:24:34Z

src/inference_endpoint/dataset_manager/predefined/cnndailymail/__init__.py

+        force_regenerate: bool = False,
+    ) -> "Dataset":
+        transforms = (transforms or []) + cls.PRESETS.llama3()
+        df = cls.generate(force=force_regenerate, datasets_dir=datasets_dir)
+        return cls(df, transforms=transforms, repeats=num_repeats)


The transforms parameter is being overridden by appending preset transforms, which prevents callers from fully controlling the transform pipeline. Consider either applying presets first (so caller transforms can override) or making the preset application optional via a parameter.

Suggested change

force_regenerate: bool = False,

) -> "Dataset":

transforms = (transforms or []) + cls.PRESETS.llama3()

df = cls.generate(force=force_regenerate, datasets_dir=datasets_dir)

return cls(df, transforms=transforms, repeats=num_repeats)

apply_presets: bool = True,

force_regenerate: bool = False,

) -> "Dataset":

if apply_presets:

final_transforms: list[Transform] = (transforms or []) + cls.PRESETS.llama3()

else:

final_transforms = transforms or []

df = cls.generate(force=force_regenerate, datasets_dir=datasets_dir)

return cls(df, transforms=final_transforms, repeats=num_repeats)

…/__init__.py Sanitize logger Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/dataset_manager/factory.py

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-22T08:20:47Z

src/inference_endpoint/dataset_manager/predefined/cnndailymail/__init__.py

+    ) -> "Dataset":
+        transforms = cls.PRESETS.llama3() + (transforms or [])


The preset transforms are hardcoded to llama3(), but the function accepts a transforms parameter that could conflict with or duplicate the preset transforms. Consider either making the preset configurable or documenting that custom transforms will be appended after the llama3 preset.

Suggested change

) -> "Dataset":

transforms = cls.PRESETS.llama3() + (transforms or [])

preset_transforms: list[Transform] | None = None,

) -> "Dataset":

# By default, use the llama3 preset and append any user-provided transforms.

# Callers can override or disable presets by passing `preset_transforms`.

if transforms is None:

transforms = []

if preset_transforms is None:

preset_transforms = cls.PRESETS.llama3()

transforms = preset_transforms + transforms

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

attafosu · 2026-01-22T16:51:16Z

@nvzhihanj @arekay-nv please review when you get the chance. thx

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

arekay-nv · 2026-01-25T21:16:17Z

src/inference_endpoint/dataset_manager/predefined/cnndailymail/__init__.py

+        force_regenerate: bool = False,
+    ) -> "Dataset":
+        transforms = (transforms or []) + cls.PRESETS.llama3()
+        df = cls.generate(force=force_regenerate, datasets_dir=datasets_dir)
+        return cls(df, transforms=transforms, repeats=num_repeats)


examples/05_Llama3.1-8B_Example/offline_llama3_8b_cnn.yaml

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Add 8b acc config to yaml

e5b5d1a

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot AI review requested due to automatic review settings January 20, 2026 06:49

github-actions bot requested review from arekay-nv and nvzhihanj January 20, 2026 06:49

Copilot AI reviewed Jan 20, 2026

View reviewed changes

examples/05_Llama3.1-8B_Example/online_llama3_8b_cnn.yaml Outdated Show resolved Hide resolved

examples/05_Llama3.1-8B_Example/online_llama3_8b_cnn.yaml Outdated Show resolved Hide resolved

examples/05_Llama3.1-8B_Example/offline_llama3_8b_cnn.yaml Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Jan 20, 2026

View reviewed changes

examples/05_Llama3.1-8B_Example/online_llama3_8b_cnn.yaml Outdated Show resolved Hide resolved

examples/05_Llama3.1-8B_Example/online_llama3_8b_cnn.yaml Outdated Show resolved Hide resolved

attafosu and others added 2 commits January 19, 2026 22:54

Fix checkpoint path

1288f96

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Apply suggestions from code review

fec6d16

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings January 20, 2026 06:58

Copilot AI reviewed Jan 20, 2026

View reviewed changes

attafosu marked this pull request as ready for review January 20, 2026 07:04

attafosu requested a review from a team as a code owner January 20, 2026 07:05

Merge branch 'main' into feat/attafosu/llama3.1-8b-acc

1f9e4c2

attafosu marked this pull request as draft January 21, 2026 16:49

Add cnn_dailymail to predefined datasets

8a3fc04

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot AI review requested due to automatic review settings January 21, 2026 21:58

Copilot AI reviewed Jan 21, 2026

View reviewed changes

src/inference_endpoint/dataset_manager/predefined/cnndailymail/__init__.py Outdated Show resolved Hide resolved

src/inference_endpoint/dataset_manager/predefined/cnndailymail/__init__.py Show resolved Hide resolved

attafosu added 2 commits January 21, 2026 14:19

Consolidate dataloading for predefined datasets

75cae8a

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Return all rouge scores

109a247

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot AI review requested due to automatic review settings January 22, 2026 00:23

Copilot AI reviewed Jan 22, 2026

View reviewed changes

Update src/inference_endpoint/dataset_manager/predefined/cnndailymail…

936e202

…/__init__.py Sanitize logger Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings January 22, 2026 00:32

Copilot AI reviewed Jan 22, 2026

View reviewed changes

src/inference_endpoint/dataset_manager/factory.py Outdated Show resolved Hide resolved

attafosu added 2 commits January 21, 2026 16:44

Re-order transforms for cnndaily downloader

0071fb3

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Update endpoints entry in 8b offline config

db3063e

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot AI review requested due to automatic review settings January 22, 2026 08:20

attafosu marked this pull request as ready for review January 22, 2026 08:20

Copilot AI reviewed Jan 22, 2026

View reviewed changes

attafosu added 2 commits January 22, 2026 00:29

Use cnn_dailymail for offline config

0c601cb

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Update Llama3.1-8b README.md

f481e47

Copilot AI review requested due to automatic review settings January 22, 2026 08:33

Copilot AI reviewed Jan 22, 2026

View reviewed changes

attafosu added 2 commits January 23, 2026 08:32

Merge branch 'main' into feat/attafosu/llama3.1-8b-acc

3ca5c44

Sanitize cnn_dailymail presets

94ba72d

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

Copilot AI review requested due to automatic review settings January 23, 2026 17:43

Copilot AI reviewed Jan 23, 2026

View reviewed changes

arekay-nv approved these changes Jan 25, 2026

View reviewed changes

Sync cnn_dailymail dataset to current template

a2d878c

Signed-off-by: attafosu <thomas.atta-fosu@intel.com>

attafosu merged commit eb135b1 into main Jan 26, 2026
4 checks passed

github-actions bot locked and limited conversation to collaborators Jan 26, 2026

attafosu deleted the feat/attafosu/llama3.1-8b-acc branch January 26, 2026 16:31

		) -> "Dataset":
		transforms = cls.PRESETS.llama3() + (transforms or [])

-    ) -> "Dataset":
-        transforms = cls.PRESETS.llama3() + (transforms or [])
+        preset_transforms: list[Transform] | None = None,
+    ) -> "Dataset":
+        # By default, use the llama3 preset and append any user-provided transforms.
+        # Callers can override or disable presets by passing `preset_transforms`.
+        if transforms is None:
+            transforms = []
+        if preset_transforms is None:
+            preset_transforms = cls.PRESETS.llama3()
+        transforms = preset_transforms + transforms

Conversation

attafosu commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Type of change

Related issues

Testing

Checklist

Uh oh!

github-actions bot commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Jan 20, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

arekay-nv Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

attafosu commented Jan 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

arekay-nv Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

attafosu commented Jan 20, 2026 •

edited

Loading

github-actions bot commented Jan 20, 2026 •

edited

Loading