Skip to content

Comments

feat: deterministic seed derivation#13

Open
kjnasdfkjnsadfkjnds wants to merge 8 commits intogonka-ai:mainfrom
kjnasdfkjnsadfkjnds:feat/sequence-check-vllm
Open

feat: deterministic seed derivation#13
kjnasdfkjnsadfkjnds wants to merge 8 commits intogonka-ai:mainfrom
kjnasdfkjnsadfkjnds:feat/sequence-check-vllm

Conversation

@kjnasdfkjnsadfkjnds
Copy link

@kjnasdfkjnsadfkjnds kjnasdfkjnsadfkjnds commented Dec 10, 2025

Purpose

This PR implements deterministic seed derivation for inference validation to prevent pre-fill attacks.

When a user provides a seed, the system now derives a unique deterministic run_seed using:

SHA256(user_seed || inference_id)

This ensures:

  • Each inference run has a unique RNG state even with the same user seed.
  • Attackers cannot pre-compute RNG states across requests.
  • Validators can verify that the returned token sequence was actually sampled from the claimed probability distribution.

Changes

  • Add run_seed to ChatCompletionLogProbs response.
  • Implement compute_derived_seed() and derive_seed_from_run_seed() for deterministic seed conversion.
  • Update sampling flow to use derived_seed instead of raw seed.
  • Forward run_seed through non-streaming and streaming responses.
  • Add full unit test coverage for seed derivation behavior.
  • Add support for overriding run_seed during validation replays.

This closes the gap that previously allowed pre-fill attacks where an attacker could sample tokens with a cheap model but report logprobs from an expensive model.

Test Plan

Test response includes run_seed:
curl http://localhost:8000/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "model-name",
"messages": [{"role": "user", "content": "test"}],
"seed": 42,
"logprobs": true,
"top_logprobs": 5
}' | jq '.choices[0].logprobs.run_seed'

Test Result

Response correctly includes run_seed:
{
"choices": [{
"logprobs": {
"content": [...],
"run_seed": "332f3b7181f6ec2b4c8a9f1e..."
}
}]
}

Same seed + same inference_id produces identical sequences across multiple runs, confirming deterministic sampling.

@github-actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@kjnasdfkjnsadfkjnds kjnasdfkjnsadfkjnds changed the title Feat/sequence check vllm feat: deterministic seed derivation Dec 10, 2025
@kjnasdfkjnsadfkjnds kjnasdfkjnsadfkjnds marked this pull request as ready for review December 10, 2025 12:11
@tamazgadaev
Copy link

The deterministic seed is needed to reproduce the procedure of sampling during the validation. Are there any tests that the same seed gives the same sampling from the probability distribution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants