Investigate NanoV3 perf sweep with tp=2,4,8

[galagam]

Initial perf sweep yields poor results. Investigate reason.

vLLM sweep looks fine (tp=8 is broken on vLLM ToT)

[galagam]

I was recording average metrics, but std dev is appears to be high in some scenarios.
See below for concurrency 80 (bad) and concurrency 96 (good).
Specifically TTFT, TTST and ITL - avg value vs mean value and std dev.

tp=2, conc=80

                                              NVIDIA AIPerf | LLM Metrics                                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃                               Metric ┃      avg ┃      min ┃       max ┃       p99 ┃       p90 ┃      p50 ┃      std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│             Time to First Token (ms) │   676.49 │   169.58 │  8,816.90 │  8,726.08 │    898.82 │   461.01 │ 1,171.80 │
│            Time to Second Token (ms) │   235.74 │     0.01 │  8,349.68 │  8,347.30 │    112.41 │    89.40 │ 1,161.28 │
│                 Request Latency (ms) │ 8,287.81 │ 5,400.48 │ 14,810.18 │ 14,807.70 │ 14,612.67 │ 6,677.24 │ 3,170.04 │
│             Inter Token Latency (ms) │     7.52 │     5.01 │     14.83 │     14.41 │     13.97 │     6.03 │     3.08 │
│     Output Token Throughput Per User │   148.37 │    67.42 │    199.45 │    197.23 │    175.41 │   165.85 │    38.15 │
│                    (tokens/sec/user) │          │          │           │           │           │          │          │
│      Output Sequence Length (tokens) │ 1,013.45 │   988.00 │  1,035.00 │  1,029.01 │  1,023.00 │ 1,013.00 │     7.26 │
│       Input Sequence Length (tokens) │ 1,000.00 │ 1,000.00 │  1,000.00 │  1,000.00 │  1,000.00 │ 1,000.00 │     0.00 │
│ Output Token Throughput (tokens/sec) │ 9,736.67 │      N/A │       N/A │       N/A │       N/A │      N/A │      N/A │
│    Request Throughput (requests/sec) │     9.61 │      N/A │       N/A │       N/A │       N/A │      N/A │      N/A │
│             Request Count (requests) │   400.00 │      N/A │       N/A │       N/A │       N/A │      N/A │      N/A │
└──────────────────────────────────────┴──────────┴──────────┴───────────┴───────────┴───────────┴──────────┴──────────┘

tp=2, conc=96

                                            NVIDIA AIPerf | LLM Metrics                                             
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┓
┃                               Metric ┃       avg ┃      min ┃      max ┃      p99 ┃      p90 ┃      p50 ┃    std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━┩
│             Time to First Token (ms) │    577.41 │   200.61 │ 1,208.26 │ 1,197.89 │   900.15 │   542.15 │ 207.49 │
│            Time to Second Token (ms) │     87.43 │     0.02 │   466.39 │   465.60 │    96.57 │    91.19 │  60.12 │
│                 Request Latency (ms) │  7,323.34 │ 6,350.38 │ 8,048.44 │ 8,018.19 │ 7,986.04 │ 7,084.22 │ 482.16 │
│             Inter Token Latency (ms) │      6.66 │     5.77 │     7.80 │     7.70 │     7.35 │     6.49 │   0.49 │
│     Output Token Throughput Per User │    150.93 │   128.19 │   173.27 │   172.02 │   161.85 │   154.10 │  10.80 │
│                    (tokens/sec/user) │           │          │          │          │          │          │        │
│      Output Sequence Length (tokens) │  1,014.01 │   989.00 │ 1,041.00 │ 1,035.00 │ 1,024.00 │ 1,014.00 │   7.69 │
│       Input Sequence Length (tokens) │  1,000.00 │ 1,000.00 │ 1,000.00 │ 1,000.00 │ 1,000.00 │ 1,000.00 │   0.00 │
│ Output Token Throughput (tokens/sec) │ 13,244.53 │      N/A │      N/A │      N/A │      N/A │      N/A │    N/A │
│    Request Throughput (requests/sec) │     13.06 │      N/A │      N/A │      N/A │      N/A │      N/A │    N/A │
│             Request Count (requests) │    480.00 │      N/A │      N/A │      N/A │      N/A │      N/A │    N/A │
└──────────────────────────────────────┴───────────┴──────────┴──────────┴──────────┴──────────┴──────────┴────────┘

Need to find a technical solution for capturing nsys traces for the problem iterations. Capturing the whole run results in a very large file that causes nsys UI to crash.

[galagam]

@MrGeva please take a look

[MrGeva]

Tested on latest main f7e2456
Adding cuda batch sizes of the spikes concurrency values solves the spikes: 40, 80, 192 in TP2

graph after adding these batch sizes:

graph before adding the batch sizes:

<style> </style>

max_concurrency	TPOT avg	user TPS avg	output TPS avg
1	3.054701	327.3643	315.6486
4	3.149085	317.5526	1194.687
8	3.249131	307.7746	2327.192
16	3.484378	286.9952	4240.998
24	3.682067	271.5866	6031.339
32	4.35079	229.8433	6799.645
40	4.576732	218.4965	7974.748
48	5.10684	195.8158	8699.812
64	5.185264	192.8542	11259.21
80	6.126727	163.2193	11671.05
96	7.074527	141.3522	12447.03
128	7.673896	130.3119	13006.02
192	10.97392	91.12517	13210.15
224	12.79147	78.1771	13779.94

closing this issue

[MrGeva]

Need to:
○ Plot with vllm
○ Why do we have this convex plot? Maybe compute boundness
○ We should not shard the shared experts - comment out the proj up and down