feat: deterministic seed derivation#13
feat: deterministic seed derivation#13kjnasdfkjnsadfkjnds wants to merge 8 commits intogonka-ai:mainfrom
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
|
The deterministic seed is needed to reproduce the procedure of sampling during the validation. Are there any tests that the same seed gives the same sampling from the probability distribution? |
Purpose
This PR implements deterministic seed derivation for inference validation to prevent pre-fill attacks.
When a user provides a seed, the system now derives a unique deterministic
run_seedusing:This ensures:
Changes
run_seedtoChatCompletionLogProbsresponse.compute_derived_seed()andderive_seed_from_run_seed()for deterministic seed conversion.derived_seedinstead of rawseed.run_seedthrough non-streaming and streaming responses.run_seedduring validation replays.This closes the gap that previously allowed pre-fill attacks where an attacker could sample tokens with a cheap model but report logprobs from an expensive model.
Test Plan
Test response includes run_seed:
curl http://localhost:8000/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "model-name",
"messages": [{"role": "user", "content": "test"}],
"seed": 42,
"logprobs": true,
"top_logprobs": 5
}' | jq '.choices[0].logprobs.run_seed'
Test Result
Response correctly includes run_seed:
{
"choices": [{
"logprobs": {
"content": [...],
"run_seed": "332f3b7181f6ec2b4c8a9f1e..."
}
}]
}
Same seed + same inference_id produces identical sequences across multiple runs, confirming deterministic sampling.