Skip to content

Conversation

@psiddh
Copy link
Contributor

@psiddh psiddh commented Jan 9, 2026

Summary:
Both tests were flaky because LLM outputs are non-deterministic with the default temperature of 0.8, which uses RNG-based sampling with a time-based seed. Setting temperature=0 enables greedy argmax decoding, eliminating randomness and making assertions on generated text reliable.

This is consistent with how other LLM tests and production runners in the codebase handle determinism (e.g., test_text_decoder_runner.cpp, test_sampler.cpp, and QNN/QAI Hub runners).

This fixes 5 flaky tests

{F1984490494}

Reviewed By: shoumikhin

Differential Revision: D90361187

Summary:
Both tests were flaky because LLM outputs are non-deterministic with the default temperature of 0.8, which uses RNG-based sampling with a time-based seed. Setting temperature=0 enables greedy argmax decoding, eliminating randomness and making assertions on generated text reliable.

This is consistent with how other LLM tests and production runners in the codebase handle determinism (e.g., test_text_decoder_runner.cpp, test_sampler.cpp, and QNN/QAI Hub runners).


This fixes 5 flaky tests

 {F1984490494}

Reviewed By: shoumikhin

Differential Revision: D90361187
Copilot AI review requested due to automatic review settings January 9, 2026 18:18
@pytorch-bot
Copy link

pytorch-bot bot commented Jan 9, 2026

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16517

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 8c9e574 with merge base 847d70d (image):

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 9, 2026
@meta-codesync
Copy link

meta-codesync bot commented Jan 9, 2026

@psiddh has exported this pull request. If you are a Meta employee, you can view the originating Diff in D90361187.

@github-actions
Copy link

github-actions bot commented Jan 9, 2026

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes flaky LLM tests by setting temperature=0 in the generation configuration, enabling deterministic greedy decoding instead of random sampling. This eliminates non-deterministic behavior that was causing test failures when assertions checked for specific words in generated text.

Key changes:

  • Added temperature=0 to both generation calls in testPhi4
  • Added temperature=0 to the generation call in testVoxtral

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
extension/llm/apple/ExecuTorchLLM/tests/TextRunnerTest.swift Set temperature=0 in both generation calls of testPhi4 to ensure deterministic output when asserting "paris" is in the generated text
extension/llm/apple/ExecuTorchLLM/tests/MultimodalRunnerTest.swift Set temperature=0 in testVoxtral's generation call to ensure deterministic output when asserting "tattoo" is in the generated text

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@psiddh psiddh requested a review from shoumikhin January 9, 2026 18:21
@psiddh psiddh merged commit 10f72fc into pytorch:main Jan 9, 2026
151 of 168 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants