feat: inference validation by kjnasdfkjnsadfkjnds · Pull Request #486 · gonka-ai/gonka

kjnasdfkjnsadfkjnds · 2025-12-10T11:37:27Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR
The test plan
The test results
(Optional) Documentation updates

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS ABOVE HAVE BEEN CONSIDERED.

Purpose

This PR introduces two core validation mechanisms for decentralized inference security according to this proposal - https://github.com/gonka-ai/gonka/blob/main/proposals/inference-validation/inference-validation.md

1. Stage 1 Sequence Check

Implements deterministic verification that each generated token in a submitted sequence appears in the corresponding top-k tokens returned in the inference artifact. This ensures executers cannot pre-generate sequences with a cheaper model and later compute probabilities with a different (claimed) model.

Sequence check logic:

Extract run_seed and top-k tokens from the inference response.
Align generated tokens with the top-k lists using the shared seed.
Validate that every generated token is included in its respective top-k list.
Fail validation if any mismatch occurs.

This closes the primary pre-fill attack vector for Stage 1.

2. Distribution Check with Live vLLM Inference

Test Plan

Make sure to build vllm with the run_seed changes - gonka-ai/vllm#13

Execution environment was validated on both local and server GPUs to ensure consistent results:

Local GPU:

NVIDIA GeForce RTX 4060 Ti

Server GPU:

NVIDIA GeForce RTX 4070

Both environments successfully executed the vLLM-based distribution check.

cd decentralized-api
go test -v ./internal/validation/

Tests cover:

Sequence check (valid, invalid, tampered, seed mismatch, length mismatch)
Sequence check on real vLLM data
Distribution check with local vLLM startup and live inference

Test Results

Sequence Check Tests

All sequence check tests pass:

❯ go test -v ./internal/validation 
=== RUN   TestValidation
    validation_test.go:52: Validation result: &{{1 []} 1}
--- PASS: TestValidation (0.00s)
=== RUN   TestValidationQuant
    validation_test.go:72: Validation result: &{{1 []} 0.9668894246685018}
--- PASS: TestValidationQuant (0.00s)
=== RUN   TestSequenceCheckValidTokensInTopK
--- PASS: TestSequenceCheckValidTokensInTopK (0.00s)
=== RUN   TestSequenceCheckInvalidTokenNotInTopK
--- PASS: TestSequenceCheckInvalidTokenNotInTopK (0.00s)
=== RUN   TestSequenceCheckEmptySeed
--- PASS: TestSequenceCheckEmptySeed (0.00s)
=== RUN   TestSequenceCheckLengthMismatch
--- PASS: TestSequenceCheckLengthMismatch (0.00s)
=== RUN   TestSequenceCheckRealData1
    validation_test.go:200: Sequence check passed for 20 tokens
--- PASS: TestSequenceCheckRealData1 (0.00s)
=== RUN   TestSequenceCheckRealData2
    validation_test.go:241: Sequence check passed for 25 tokens
--- PASS: TestSequenceCheckRealData2 (0.00s)
=== RUN   TestSequenceCheckTamperedToken
--- PASS: TestSequenceCheckTamperedToken (0.00s)
=== RUN   TestSequenceCheckWrongSeed
--- PASS: TestSequenceCheckWrongSeed (0.00s)
=== RUN   TestDistributionCheckWithVLLM
    validation_test.go:329: Skipping vLLM distribution check test since it requires vLLM server
--- SKIP: TestDistributionCheckWithVLLM (0.00s)
PASS
ok  	decentralized-api/internal/validation	0.049s

Distribution Check Tests (Using Live vLLM)

=== RUN   TestDistributionCheckWithVLLM
    validation_test.go:367: Testing distribution check with model: facebook/opt-125m
    validation_test.go:368: vLLM URL: http://localhost:8080
    validation_test.go:378: Running first inference...
2025/12/10 10:27:44 INFO Setting up vLLM for model subsystem=Validation model=facebook/opt-125m dtype=auto additional_args="[--max-model-len 2048 --gpu-memory-utilization 0.5 --chat-template \"<|im_start|>{{ role }}\\n{{ content }}<|im_end|>\"]"
2025/12/10 10:27:44 INFO Model set successfully subsystem=Validation model=facebook/opt-125m
2025/12/10 10:29:02 INFO vLLM started successfully subsystem=Validation message="vLLM started successfully"
2025/12/10 10:29:04 INFO vLLM is ready subsystem=Validation model=facebook/opt-125m
    validation_test.go:384: First inference completed with status: completed
    validation_test.go:387: DEBUG: logits1=map[choices:[map[finish_reason:length index:0 logprobs:map[content:[map[bytes:[92] logprob:-0.02168002724647522 token:\ top_logprobs:[map[bytes:[92] logprob:-0.02168002724647522 token:\] map[bytes:[10] logprob:-4.131054878234863 token:
        ] map[bytes:[124] logprob:-5.990429878234863 token:|]]] map[bytes:[110] logprob:-7.390948667307384e-06 token:n top_logprobs:[map[bytes:[110] logprob:-7.390948667307384e-06 token:n] map[bytes:[117] logprob:-12.875007629394531 token:u] map[bytes:[114] logprob:-13.046882629394531 token:r]]] map[bytes:[60] logprob:-0.005468409974128008 token:< top_logprobs:[map[bytes:[60] logprob:-0.005468409974128008 token:<] map[bytes:[10] logprob:-5.989843368530273 token:
        ] map[bytes:[60 47] logprob:-6.630468368530273 token:</]]] map[bytes:[124] logprob:-0.0014393694000318646 token:| top_logprobs:[map[bytes:[124] logprob:-0.0014393694000318646 token:|] map[bytes:[33] logprob:-7.173314571380615 token:!] map[bytes:[118] logprob:-8.204564094543457 token:v]]] map[bytes:[105 109] logprob:-0.00021324267436284572 token:im top_logprobs:[map[bytes:[105 109] logprob:-0.00021324267436284572 token:im] map[bytes:[109] logprob:-9.343963623046875 token:m] map[bytes:[73 77] logprob:-10.297088623046875 token:IM]]] map[bytes:[95] logprob:-9.536738616588991e-07 token:_ top_logprobs:[map[bytes:[95] logprob:-9.536738616588991e-07 token:_] map[bytes:[45] logprob:-14.015625953674316 token:-] map[bytes:[105 109] logprob:-17.328125 token:im]]] map[bytes:[101 110 100] logprob:-0.11217606067657471 token:end top_logprobs:[map[bytes:[101 110 100] logprob:-0.11217606067657471 token:end] map[bytes:[115 116 97 114 116] logprob:-2.377800941467285 token:start] map[bytes:[98 101 103 105 110] logprob:-4.955925941467285 token:begin]]] map[bytes:[124] logprob:-0.0005323661607690156 token:| top_logprobs:[map[bytes:[124] logprob:-0.0005323661607690156 token:|] map[bytes:[10] logprob:-8.203657150268555 token:
        ] map[bytes:[43] logprob:-9.625532150268555 token:+]]] map[bytes:[62] logprob:-0.0002693767019081861 token:> top_logprobs:[map[bytes:[62] logprob:-0.0002693767019081861 token:>] map[bytes:[124] logprob:-9.406518936157227 token:|] map[bytes:[10] logprob:-9.484643936157227 token:
        ]]] map[bytes:[92] logprob:-0.0018736441852524877 token:\ top_logprobs:[map[bytes:[92] logprob:-0.0018736441852524877 token:\] map[bytes:[10] logprob:-6.53312349319458 token:
        ] map[bytes:[124] logprob:-9.048748970031738 token:|]]] map[bytes:[110] logprob:-9.536738616588991e-07 token:n top_logprobs:[map[bytes:[110] logprob:-9.536738616588991e-07 token:n] map[bytes:[114] logprob:-14.953125953674316 token:r] map[bytes:[117] logprob:-15.312500953674316 token:u]]] map[bytes:[60] logprob:-0.0015754204941913486 token:< top_logprobs:[map[bytes:[60] logprob:-0.0015754204941913486 token:<] map[bytes:[60 47] logprob:-7.282825469970703 token:</] map[bytes:[10] logprob:-7.532825469970703 token:
        ]]] map[bytes:[124] logprob:-0.0002858230145648122 token:| top_logprobs:[map[bytes:[124] logprob:-0.0002858230145648122 token:|] map[bytes:[118] logprob:-9.031536102294922 token:v] map[bytes:[33] logprob:-9.047161102294922 token:!]]] map[bytes:[105 109] logprob:-5.113947918289341e-05 token:im top_logprobs:[map[bytes:[105 109] logprob:-5.113947918289341e-05 token:im] map[bytes:[109] logprob:-11.140676498413086 token:m] map[bytes:[73 77] logprob:-11.156301498413086 token:IM]]] map[bytes:[95] logprob:-3.576278118089249e-07 token:_ top_logprobs:[map[bytes:[95] logprob:-3.576278118089249e-07 token:_] map[bytes:[45] logprob:-14.9375 token:-] map[bytes:[95 123] logprob:-18.703125 token:_{]]] map[bytes:[101 110 100] logprob:-0.018947675824165344 token:end top_logprobs:[map[bytes:[101 110 100] logprob:-0.018947675824165344 token:end] map[bytes:[115 116 97 114 116] logprob:-4.065822601318359 token:start] map[bytes:[98 101 103 105 110] logprob:-6.565822601318359 token:begin]]] map[bytes:[124] logprob:-2.396077979938127e-05 token:| top_logprobs:[map[bytes:[124] logprob:-2.396077979938127e-05 token:|] map[bytes:[10] logprob:-11.43752384185791 token:
        ] map[bytes:[92] logprob:-12.03127384185791 token:\]]] map[bytes:[62] logprob:-5.185469490243122e-05 token:> top_logprobs:[map[bytes:[62] logprob:-5.185469490243122e-05 token:>] map[bytes:[62 60] logprob:-10.843801498413086 token:><] map[bytes:[124] logprob:-11.750051498413086 token:|]]] map[bytes:[92] logprob:-0.0004153581103309989 token:\ top_logprobs:[map[bytes:[92] logprob:-0.0004153581103309989 token:\] map[bytes:[10] logprob:-8.000415802001953 token:
        ] map[bytes:[32 92] logprob:-11.031665802001953 token: \]]] map[bytes:[110] logprob:-7.152555099310121e-07 token:n top_logprobs:[map[bytes:[110] logprob:-7.152555099310121e-07 token:n] map[bytes:[114] logprob:-15.062500953674316 token:r] map[bytes:[117] logprob:-15.531250953674316 token:u]]] map[bytes:[60] logprob:-0.0008304246584884822 token:< top_logprobs:[map[bytes:[60] logprob:-0.0008304246584884822 token:<] map[bytes:[60 47] logprob:-7.76645565032959 token:</] map[bytes:[10] logprob:-8.51645565032959 token:
        ]]] map[bytes:[124] logprob:-0.0001411338453181088 token:| top_logprobs:[map[bytes:[124] logprob:-0.0001411338453181088 token:|] map[bytes:[33] logprob:-9.765766143798828 token:!] map[bytes:[118] logprob:-9.781391143798828 token:v]]] map[bytes:[105 109] logprob:-1.966933996300213e-05 token:im top_logprobs:[map[bytes:[105 109] logprob:-1.966933996300213e-05 token:im] map[bytes:[73 77] logprob:-11.984395027160645 token:IM] map[bytes:[109] logprob:-12.078145027160645 token:m]]] map[bytes:[95] logprob:-2.3841855067985307e-07 token:_ top_logprobs:[map[bytes:[95] logprob:-2.3841855067985307e-07 token:_] map[bytes:[45] logprob:-15.125 token:-] map[bytes:[95 123] logprob:-19.203125 token:_{]]] map[bytes:[101 110 100] logprob:-0.002434982219710946 token:end top_logprobs:[map[bytes:[101 110 100] logprob:-0.002434982219710946 token:end] map[bytes:[115 116 97 114 116] logprob:-6.127435207366943 token:start] map[bytes:[98 101 103 105 110] logprob:-8.377434730529785 token:begin]]] map[bytes:[124] logprob:-5.722029527532868e-06 token:| top_logprobs:[map[bytes:[124] logprob:-5.722029527532868e-06 token:|] map[bytes:[10] logprob:-13.046880722045898 token:
        ] map[bytes:[92] logprob:-13.250005722045898 token:\]]] map[bytes:[62] logprob:-2.5987286790041253e-05 token:> top_logprobs:[map[bytes:[62] logprob:-2.5987286790041253e-05 token:>] map[bytes:[62 60] logprob:-11.359400749206543 token:><] map[bytes:[62 40] logprob:-12.875025749206543 token:>(]]] map[bytes:[92] logprob:-0.00011777184408856556 token:\ top_logprobs:[map[bytes:[92] logprob:-0.00011777184408856556 token:\] map[bytes:[10] logprob:-9.265742301940918 token:
        ] map[bytes:[32 92] logprob:-11.843867301940918 token: \]]] map[bytes:[110] logprob:-5.960462772236497e-07 token:n top_logprobs:[map[bytes:[110] logprob:-5.960462772236497e-07 token:n] map[bytes:[114] logprob:-15.406250953674316 token:r] map[bytes:[117] logprob:-15.671875953674316 token:u]]] map[bytes:[60] logprob:-0.00041940953815355897 token:< top_logprobs:[map[bytes:[60] logprob:-0.00041940953815355897 token:<] map[bytes:[60 47] logprob:-8.266044616699219 token:</] map[bytes:[92] logprob:-9.516044616699219 token:\]]] map[bytes:[124] logprob:-5.090107151772827e-05 token:| top_logprobs:[map[bytes:[124] logprob:-5.090107151772827e-05 token:|] map[bytes:[33] logprob:-10.73442554473877 token:!] map[bytes:[118] logprob:-11.10942554473877 token:v]]] map[bytes:[105 109] logprob:-1.0609570381348021e-05 token:im top_logprobs:[map[bytes:[105 109] logprob:-1.0609570381348021e-05 token:im] map[bytes:[73 77] logprob:-12.53126049041748 token:IM] map[bytes:[109] logprob:-12.65626049041748 token:m]]] map[bytes:[95] logprob:-3.576278118089249e-07 token:_ top_logprobs:[map[bytes:[95] logprob:-3.576278118089249e-07 token:_] map[bytes:[45] logprob:-15.03125 token:-] map[bytes:[42] logprob:-19.46875 token:*]]] map[bytes:[101 110 100] logprob:-0.00020525732543319464 token:end top_logprobs:[map[bytes:[101 110 100] logprob:-0.00020525732543319464 token:end] map[bytes:[115 116 97 114 116] logprob:-8.672080039978027 token:start] map[bytes:[98 101 103 105 110] logprob:-10.359580039978027 token:begin]]] map[bytes:[124] logprob:-1.9073468138230965e-06 token:| top_logprobs:[map[bytes:[124] logprob:-1.9073468138230965e-06 token:|] map[bytes:[10] logprob:-14.250001907348633 token:
        ] map[bytes:[92] logprob:-14.343751907348633 token:\]]] map[bytes:[62] logprob:-1.6331539882230572e-05 token:> top_logprobs:[map[bytes:[62] logprob:-1.6331539882230572e-05 token:>] map[bytes:[62 60] logprob:-11.781266212463379 token:><] map[bytes:[62 62] logprob:-12.968766212463379 token:>>]]] map[bytes:[92] logprob:-4.1483970562694594e-05 token:\ top_logprobs:[map[bytes:[92] logprob:-4.1483970562694594e-05 token:\] map[bytes:[10] logprob:-10.296916007995605 token:
        ] map[bytes:[32 92] logprob:-12.765666007995605 token: \]]] map[bytes:[110] logprob:-5.960462772236497e-07 token:n top_logprobs:[map[bytes:[110] logprob:-5.960462772236497e-07 token:n] map[bytes:[117] logprob:-15.312500953674316 token:u] map[bytes:[114] logprob:-15.578125953674316 token:r]]] map[bytes:[60] logprob:-0.00025138078490272164 token:< top_logprobs:[map[bytes:[60] logprob:-0.00025138078490272164 token:<] map[bytes:[60 47] logprob:-8.687751770019531 token:</] map[bytes:[92] logprob:-10.140876770019531 token:\]]] map[bytes:[124] logprob:-2.5152843591058627e-05 token:| top_logprobs:[map[bytes:[124] logprob:-2.5152843591058627e-05 token:|] map[bytes:[33] logprob:-11.406274795532227 token:!] map[bytes:[118] logprob:-11.968774795532227 token:v]]] map[bytes:[105 109] logprob:-6.556489552167477e-06 token:im top_logprobs:[map[bytes:[105 109] logprob:-6.556489552167477e-06 token:im] map[bytes:[73 77] logprob:-12.875006675720215 token:IM] map[bytes:[109] logprob:-13.234381675720215 token:m]]] map[bytes:[95] logprob:-2.3841855067985307e-07 token:_ top_logprobs:[map[bytes:[95] logprob:-2.3841855067985307e-07 token:_] map[bytes:[45] logprob:-15.390625 token:-] map[bytes:[42] logprob:-19.703125 token:*]]] map[bytes:[101 110 100] logprob:-3.2305197237292305e-05 token:end top_logprobs:[map[bytes:[101 110 100] logprob:-3.2305197237292305e-05 token:end] map[bytes:[115 116 97 114 116] logprob:-10.578157424926758 token:start] map[bytes:[98 101 103 105 110] logprob:-12.000032424926758 token:begin]]] map[bytes:[124] logprob:-9.536738616588991e-07 token:| top_logprobs:[map[bytes:[124] logprob:-9.536738616588991e-07 token:|] map[bytes:[10] logprob:-14.843750953674316 token:
        ] map[bytes:[92] logprob:-15.140625953674316 token:\]]] map[bytes:[62] logprob:-1.3708974620385561e-05 token:> top_logprobs:[map[bytes:[62] logprob:-1.3708974620385561e-05 token:>] map[bytes:[62 60] logprob:-11.92188835144043 token:><] map[bytes:[62 62] logprob:-12.85938835144043 token:>>]]] map[bytes:[92] logprob:-2.52720492426306e-05 token:\ top_logprobs:[map[bytes:[92] logprob:-2.52720492426306e-05 token:\] map[bytes:[10] logprob:-10.796899795532227 token:
        ] map[bytes:[32 92] logprob:-13.250024795532227 token: \]]] map[bytes:[110] logprob:-5.960462772236497e-07 token:n top_logprobs:[map[bytes:[110] logprob:-5.960462772236497e-07 token:n] map[bytes:[117] logprob:-15.109375953674316 token:u] map[bytes:[114] logprob:-15.765625953674316 token:r]]] map[bytes:[60] logprob:-0.00016807096835691482 token:< top_logprobs:[map[bytes:[60] logprob:-0.00016807096835691482 token:<] map[bytes:[60 47] logprob:-9.062667846679688 token:</] map[bytes:[92] logprob:-10.609542846679688 token:\]]] map[bytes:[124] logprob:-1.7762025890988298e-05 token:| top_logprobs:[map[bytes:[124] logprob:-1.7762025890988298e-05 token:|] map[bytes:[33] logprob:-11.640643119812012 token:!] map[bytes:[60] logprob:-12.500018119812012 token:<]]] map[bytes:[105 109] logprob:-4.768360213347478e-06 token:im top_logprobs:[map[bytes:[105 109] logprob:-4.768360213347478e-06 token:im] map[bytes:[73 77] logprob:-12.953129768371582 token:IM] map[bytes:[109] logprob:-13.734379768371582 token:m]]] map[bytes:[95] logprob:-1.1920928244535389e-07 token:_ top_logprobs:[map[bytes:[95] logprob:-1.1920928244535389e-07 token:_] map[bytes:[45] logprob:-15.796875 token:-] map[bytes:[42] logprob:-19.921875 token:*]]] map[bytes:[101 110 100] logprob:-9.059865078597795e-06 token:end top_logprobs:[map[bytes:[101 110 100] logprob:-9.059865078597795e-06 token:end] map[bytes:[115 116 97 114 116] logprob:-11.937508583068848 token:start] map[bytes:[98 101 103 105 110] logprob:-13.046883583068848 token:begin]]] map[bytes:[124] logprob:-8.344646857949556e-07 token:| top_logprobs:[map[bytes:[124] logprob:-8.344646857949556e-07 token:|] map[bytes:[10] logprob:-15.046875953674316 token:
        ] map[bytes:[92] logprob:-15.406250953674316 token:\]]] map[bytes:[62] logprob:-1.1324817933200393e-05 token:> top_logprobs:[map[bytes:[62] logprob:-1.1324817933200393e-05 token:>] map[bytes:[62 60] logprob:-12.109386444091797 token:><] map[bytes:[62 62] logprob:-12.953136444091797 token:>>]]] map[bytes:[92] logprob:-1.8715683836489916e-05 token:\ top_logprobs:[map[bytes:[92] logprob:-1.8715683836489916e-05 token:\] map[bytes:[10] logprob:-11.062519073486328 token:
        ] map[bytes:[32 92] logprob:-13.765644073486328 token: \]]] map[bytes:[110] logprob:-7.152555099310121e-07 token:n top_logprobs:[map[bytes:[110] logprob:-7.152555099310121e-07 token:n] map[bytes:[117] logprob:-15.031250953674316 token:u] map[bytes:[10] logprob:-15.765625953674316 token:
        ]]] map[bytes:[60] logprob:-0.00016378014697693288 token:< top_logprobs:[map[bytes:[60] logprob:-0.00016378014697693288 token:<] map[bytes:[60 47] logprob:-9.109539031982422 token:</] map[bytes:[10] logprob:-10.515789031982422 token:
        ]]] map[bytes:[124] logprob:-1.3708974620385561e-05 token:| top_logprobs:[map[bytes:[124] logprob:-1.3708974620385561e-05 token:|] map[bytes:[33] logprob:-11.85938835144043 token:!] map[bytes:[60] logprob:-12.64063835144043 token:<]]] map[bytes:[105 109] logprob:-3.6954811548639555e-06 token:im top_logprobs:[map[bytes:[105 109] logprob:-3.6954811548639555e-06 token:im] map[bytes:[73 77] logprob:-13.156253814697266 token:IM] map[bytes:[109] logprob:-14.031253814697266 token:m]]] map[bytes:[95] logprob:-1.1920928244535389e-07 token:_ top_logprobs:[map[bytes:[95] logprob:-1.1920928244535389e-07 token:_] map[bytes:[45] logprob:-15.875 token:-] map[bytes:[42] logprob:-20.03125 token:*]]] map[bytes:[101 110 100] logprob:-2.7418097943154862e-06 token:end top_logprobs:[map[bytes:[101 110 100] logprob:-2.7418097943154862e-06 token:end] map[bytes:[115 116 97 114 116] logprob:-13.29687786102295 token:start] map[bytes:[98 101 103 105 110] logprob:-14.03125286102295 token:begin]]] map[bytes:[124] logprob:-5.960462772236497e-07 token:| top_logprobs:[map[bytes:[124] logprob:-5.960462772236497e-07 token:|] map[bytes:[10] logprob:-15.390625953674316 token:
        ] map[bytes:[32 124] logprob:-15.546875953674316 token: |]]] map[bytes:[62] logprob:-1.0013530300057027e-05 token:> top_logprobs:[map[bytes:[62] logprob:-1.0013530300057027e-05 token:>] map[bytes:[62 60] logprob:-12.203134536743164 token:><] map[bytes:[62 62] logprob:-13.062509536743164 token:>>]]] map[bytes:[92] logprob:-1.9311717551317997e-05 token:\ top_logprobs:[map[bytes:[92] logprob:-1.9311717551317997e-05 token:\] map[bytes:[10] logprob:-11.015644073486328 token:
        ] map[bytes:[32 92] logprob:-14.000019073486328 token: \]]] map[bytes:[110] logprob:-7.152555099310121e-07 token:n top_logprobs:[map[bytes:[110] logprob:-7.152555099310121e-07 token:n] map[bytes:[117] logprob:-14.937500953674316 token:u] map[bytes:[10] logprob:-15.968750953674316 token:
        ]]] map[bytes:[60] logprob:-8.77341881277971e-05 token:< top_logprobs:[map[bytes:[60] logprob:-8.77341881277971e-05 token:<] map[bytes:[60 47] logprob:-9.70321273803711 token:</] map[bytes:[10] logprob:-11.23446273803711 token:
        ]]] map[bytes:[124] logprob:-1.1801649634435307e-05 token:| top_logprobs:[map[bytes:[124] logprob:-1.1801649634435307e-05 token:|] map[bytes:[33] logprob:-11.953136444091797 token:!] map[bytes:[60] logprob:-12.765636444091797 token:<]]] map[bytes:[105 109] logprob:-2.622600959512056e-06 token:im top_logprobs:[map[bytes:[105 109] logprob:-2.622600959512056e-06 token:im] map[bytes:[73 77] logprob:-13.43750286102295 token:IM] map[bytes:[109] logprob:-14.40625286102295 token:m]]] map[bytes:[95] logprob:-1.1920928244535389e-07 token:_ top_logprobs:[map[bytes:[95] logprob:-1.1920928244535389e-07 token:_] map[bytes:[45] logprob:-16.0625 token:-] map[bytes:[42] logprob:-20.0625 token:*]]] map[bytes:[101 110 100] logprob:-1.311301275563892e-06 token:end top_logprobs:[map[bytes:[101 110 100] logprob:-1.311301275563892e-06 token:end] map[bytes:[115 116 97 114 116] logprob:-14.218750953674316 token:start] map[bytes:[98 101 103 105 110] logprob:-14.765625953674316 token:begin]]] map[bytes:[124] logprob:-3.576278118089249e-07 token:| top_logprobs:[map[bytes:[124] logprob:-3.576278118089249e-07 token:|] map[bytes:[10] logprob:-15.703125 token:
        ] map[bytes:[32 124] logprob:-15.71875 token: |]]] map[bytes:[62] logprob:-7.510157047363464e-06 token:> top_logprobs:[map[bytes:[62] logprob:-7.510157047363464e-06 token:>] map[bytes:[62 60] logprob:-12.421882629394531 token:><] map[bytes:[62 62] logprob:-13.328132629394531 token:>>]]] map[bytes:[92] logprob:-1.3112935448589269e-05 token:\ top_logprobs:[map[bytes:[92] logprob:-1.3112935448589269e-05 token:\] map[bytes:[10] logprob:-11.42188835144043 token:
        ] map[bytes:[62] logprob:-14.42188835144043 token:>]]] map[bytes:[110] logprob:-9.536738616588991e-07 token:n top_logprobs:[map[bytes:[110] logprob:-9.536738616588991e-07 token:n] map[bytes:[117] logprob:-14.390625953674316 token:u] map[bytes:[10] logprob:-15.671875953674316 token:
        ]]] map[bytes:[60] logprob:-6.878139538457617e-05 token:< top_logprobs:[map[bytes:[60] logprob:-6.878139538457617e-05 token:<] map[bytes:[60 47] logprob:-9.921943664550781 token:</] map[bytes:[10] logprob:-11.562568664550781 token:
        ]]] map[bytes:[124] logprob:-1.0251946150674485e-05 token:| top_logprobs:[map[bytes:[124] logprob:-1.0251946150674485e-05 token:|] map[bytes:[33] logprob:-12.07813549041748 token:!] map[bytes:[60] logprob:-12.85938549041748 token:<]]] map[bytes:[105 109] logprob:-2.3841830625315197e-06 token:im top_logprobs:[map[bytes:[105 109] logprob:-2.3841830625315197e-06 token:im] map[bytes:[73 77] logprob:-13.484376907348633 token:IM] map[bytes:[109] logprob:-14.640626907348633 token:m]]] map[bytes:[95] logprob:-1.1920928244535389e-07 token:_ top_logprobs:[map[bytes:[95] logprob:-1.1920928244535389e-07 token:_] map[bytes:[45] logprob:-15.90625 token:-] map[bytes:[42] logprob:-19.984375 token:*]]] map[bytes:[101 110 100] logprob:-8.344646857949556e-07 token:end top_logprobs:[map[bytes:[101 110 100] logprob:-8.344646857949556e-07 token:end] map[bytes:[115 116 97 114 116] logprob:-14.484375953674316 token:start] map[bytes:[98 101 103 105 110] logprob:-15.062500953674316 token:begin]]] map[bytes:[124] logprob:-3.576278118089249e-07 token:| top_logprobs:[map[bytes:[124] logprob:-3.576278118089249e-07 token:|] map[bytes:[32 124] logprob:-15.6875 token: |] map[bytes:[10] logprob:-15.78125 token:
        ]]]] run_seed:c923a716b21c5a1eb9671b40771f9b74c1f4c69a559e99092f619244d40b6368] message:map[content:\n<|im_end|>\n<|im_end|>\n<|im_end|>\n<|im_end|>\n<|im_end|>\n<|im_end|>\n<|im_end|>\n<|im_end|>\n<|im_end| reasoning_content:<nil> role:assistant tool_calls:[]] stop_reason:<nil>]] created:1.765355344e+09 id:chatcmpl-67b71d2f2c884db7a54480f7d2783864 kv_transfer_params:<nil> model:facebook/opt-125m object:chat.completion prompt_logprobs:<nil> usage:map[completion_tokens:80 prompt_tokens:16 prompt_tokens_details:<nil> total_tokens:96]], err=<nil>
    validation_test.go:392: Extracted 80 tokens from first inference
    validation_test.go:394: First token: \ (logprob: -0.0217, top_logprobs: 3)
    validation_test.go:398: Running second inference with same parameters...
    validation_test.go:404: Second inference completed with status: completed
    validation_test.go:411: Extracted 80 tokens from second inference
    validation_test.go:414: Extracted 80 tokens from file response
    validation_test.go:421:
        === Comparing vLLM inference 1 vs inference 2 ===
    validation_test.go:423: Distribution comparison result: *validation.SimilarityValidationResult
    validation_test.go:426: Similarity score (vLLM 1 vs 2): 1.000000
    validation_test.go:427: Is successful (>0.99): true
    validation_test.go:436:
        === Comparing vLLM inference 1 vs file response ===
    validation_test.go:439: Similarity score (vLLM 1 vs file): 0.999404
    validation_test.go:447:
        === Comparing vLLM inference 2 vs file response ===
    validation_test.go:450: Similarity score (vLLM 2 vs file): 0.999404
    validation_test.go:459:
        First 3 tokens comparison:
    validation_test.go:461:   Position 0:
    validation_test.go:462:     vLLM 1: \ (-0.0217)
    validation_test.go:463:     vLLM 2: \ (-0.0217)
    validation_test.go:465:     File:   \ (-0.0217)
    validation_test.go:461:   Position 1:
    validation_test.go:462:     vLLM 1: n (-0.0000)
    validation_test.go:463:     vLLM 2: n (-0.0000)
    validation_test.go:465:     File:   n (-0.0000)
    validation_test.go:461:   Position 2:
    validation_test.go:462:     vLLM 1: < (-0.0055)
    validation_test.go:463:     vLLM 2: < (-0.0055)
    validation_test.go:465:     File:   < (-0.0055)
--- PASS: TestDistributionCheckWithVLLM (80.88s)
PASS

Distribution check logs confirm:

vLLM fully initializes
logits are collected
validation passes across both GPU environments

(Optional) Documentation Update

No documentation update required. The validation logic is fully internal and does not change the public API.

…o vllm client

feat: distribution check

tamazgadaev · 2025-12-15T22:47:37Z

Here is the short recap of the review with the main points:

Validator container and validator package aren't necessary.
We don't need the separate container for inference validation, we can deploy vllm by sending a /inference/up command to MLnode. Inference and validation are executed using the same vllm endpoint, the difference is in sampling and enforced_tokens.
A variation of distribution check is already implemented and it is basically the current form of validation, so we don't need to implement or test it.
checkSequenceFromArtifact is a simplified sampling check, that this imrovement is intended to target.
the core idea is to implement the check: given tokens are indeed sampled from the given logprobs with the given seed. I didn't find the test, that evaluates that.

strange-dv and others added 13 commits December 3, 2025 17:20

feat: sequence check validator

baaa3d8

test with real vllm responses

a8c7b7d

test with invalid seed

452a042

implement basic functionality of distribution check. Bind validator t…

6e2ab82

…o vllm client

Remove large file from repository

6653073

clean up

1e50378

fix tests

09c7664

fix test data formatting

d8f90e8

stop vllm after validator inference

6a88ff0

clean up

a50c600

clean up

ce86a0e

return poetry.lock and skip faling test

ea0c2c7

Merge pull request #3 from ZpokenWeb3/feat/distribution-check

46aac77

feat: distribution check

kjnasdfkjnsadfkjnds marked this pull request as draft December 10, 2025 11:38

kjnasdfkjnsadfkjnds marked this pull request as ready for review December 10, 2025 12:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

feat: inference validation #486

feat: inference validation #486
kjnasdfkjnsadfkjnds wants to merge 13 commits intogonka-ai:mainfrom
ZpokenWeb3:feat/sequence-check-validator

kjnasdfkjnsadfkjnds commented Dec 10, 2025

Uh oh!

tamazgadaev commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

Conversation

kjnasdfkjnsadfkjnds commented Dec 10, 2025

Essential Elements of an Effective PR Description Checklist

Purpose

1. Stage 1 Sequence Check

2. Distribution Check with Live vLLM Inference

Test Plan

Test Results

Sequence Check Tests

Distribution Check Tests (Using Live vLLM)

(Optional) Documentation Update

Uh oh!

tamazgadaev commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants