[Bug]: AutoDeploy: track nano perf regression

[lucaslie]

System Info

see https://nvidia.slack.com/archives/C08T55LHSG4/p1765573240272469

Tracking a nano3 perf regression in the commit range 3156f2e...a6263a1 (3156f2e to a6263a1)

[lucaslie]

ISL/OSL at concurrency 384:
a6263a1

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃                               Metric ┃       avg ┃       min ┃       max ┃       p99 ┃       p90 ┃       p50 ┃      std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│             Time to First Token (ms) │  2,018.97 │    277.04 │ 12,197.08 │ 11,917.80 │  7,499.42 │    789.14 │ 2,994.24 │
│            Time to Second Token (ms) │    222.11 │      0.01 │  3,275.25 │  1,000.56 │    220.80 │    209.51 │   222.63 │
│                 Request Latency (ms) │ 86,231.29 │ 74,731.40 │ 96,475.55 │ 96,038.63 │ 91,258.49 │ 85,948.02 │ 3,489.99 │
│             Inter Token Latency (ms) │     84.48 │     76.83 │     96.05 │     87.15 │     86.60 │     84.77 │     2.10 │
│     Output Token Throughput Per User │     11.84 │     10.41 │     13.02 │     12.93 │     12.27 │     11.80 │     0.31 │
│                    (tokens/sec/user) │           │           │           │           │           │           │          │
│      Output Sequence Length (tokens) │    997.84 │    874.00 │  1,011.00 │  1,003.00 │  1,001.00 │    999.00 │     5.79 │
│       Input Sequence Length (tokens) │  1,000.00 │  1,000.00 │  1,000.00 │  1,000.00 │  1,000.00 │  1,000.00 │     0.00 │
│ Output Token Throughput (tokens/sec) │  4,433.11 │       N/A │       N/A │       N/A │       N/A │       N/A │      N/A │
│    Request Throughput (requests/sec) │      4.44 │       N/A │       N/A │       N/A │       N/A │       N/A │      N/A │
│             Request Count (requests) │  1,920.00 │       N/A │       N/A │       N/A │       N/A │       N/A │      N/A │
└──────────────────────────────────────┴───────────┴───────────┴───────────┴───────────┴───────────┴───────────┴──────────┘

vs
3156f2e

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃                               Metric ┃       avg ┃       min ┃       max ┃       p99 ┃       p90 ┃       p50 ┃      std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│             Time to First Token (ms) │  2,452.90 │    193.69 │ 10,686.31 │ 10,504.80 │  7,111.35 │    940.70 │ 2,621.66 │
│            Time to Second Token (ms) │    161.11 │      0.01 │  3,236.45 │    534.23 │    163.19 │    155.55 │   207.46 │
│                 Request Latency (ms) │ 35,990.70 │ 28,507.92 │ 45,142.39 │ 44,825.08 │ 41,314.43 │ 35,220.54 │ 3,381.38 │
│             Inter Token Latency (ms) │     33.63 │     27.95 │     39.68 │     37.21 │     34.56 │     34.20 │     1.62 │
│     Output Token Throughput Per User │     29.81 │     25.20 │     35.78 │     35.43 │     31.85 │     29.24 │     1.56 │
│                    (tokens/sec/user) │           │           │           │           │           │           │          │
│      Output Sequence Length (tokens) │    998.18 │    868.00 │  1,010.00 │  1,003.00 │  1,001.00 │    999.00 │     4.63 │
│       Input Sequence Length (tokens) │  1,000.00 │  1,000.00 │  1,000.00 │  1,000.00 │  1,000.00 │  1,000.00 │     0.00 │
│ Output Token Throughput (tokens/sec) │ 10,607.09 │       N/A │       N/A │       N/A │       N/A │       N/A │      N/A │
│    Request Throughput (requests/sec) │     10.63 │       N/A │       N/A │       N/A │       N/A │       N/A │      N/A │
│             Request Count (requests) │  1,920.00 │       N/A │       N/A │       N/A │       N/A │       N/A │      N/A │
└──────────────────────────────────────┴───────────┴───────────┴───────────┴───────────┴───────────┴───────────┴──────────┘

[lucaslie]

Bisected to 02edb19 which introduces the regression

[lucaslie]

Tracked via https://nvbugspro.nvidia.com/bug/5738737 and temporarily reverted in #10002