Skip to content

Commit 00da186

Browse files
authored
Select the correct device (arch) on benchmark dashboard(s) (#6993)
There is a bug when a device is select where all the different arch(s) of the same device are showing up too. For example, selecting `cuda (NVIDIA B200)` would also query the results from `cuda (NVIDIA H100)` and `cuda (NVIDIA A100)` in addition to B200. The problem here is that we are using a single device drop-down list to query both device and arch, and this confuses the ClickHouse queries. The fix here is to parse the content from the device drop-down list to set the device and the arch names correctly. This basically unravels [getDeviceArch](https://github.com/pytorch/test-infra/blob/bcc20e4fcae874ed65f8a5036ab6b8989a93aefc/torchci/components/benchmark/llms/components/LLMsSummaryPanel.tsx#L33-L40) into its components ### Testing The current prod dashboards show all arch(s) of the same devices, for example: * [vLLM](https://hud.pytorch.org/benchmark/llms?startTime=Tue%2C%2005%20Aug%202025%2002%3A07%3A52%20GMT&stopTime=Tue%2C%2012%20Aug%202025%2002%3A07%3A52%20GMT&granularity=day&lBranch=main&lCommit=bd3db7f46965bfc979734a6d4b50cf96184c10d8&rBranch=main&rCommit=458e74eb907f96069e6d8a4f3c9f457001fef2ea&repoName=vllm-project%2Fvllm&benchmarkName=&modelName=All%20Models&backendName=All%20Backends&modeName=All%20Modes&dtypeName=All%20DType&deviceName=cuda%20(NVIDIA%20B200)&archName=All%20Platforms) shows B200, H100, and A100 * [ExecuTorch](https://hud.pytorch.org/benchmark/llms?startTime=Tue%2C%2005%20Aug%202025%2002%3A08%3A58%20GMT&stopTime=Tue%2C%2012%20Aug%202025%2002%3A08%3A58%20GMT&granularity=day&lBranch=main&lCommit=123fe91a5bd4e582f0d660926ec9fea620f5c9c5&rBranch=main&rCommit=8e858570a2dba922142cad667f7ba84f82e2859a&repoName=pytorch%2Fexecutorch&benchmarkName=&modelName=All%20Models&backendName=All%20Backends&modeName=All%20Modes&dtypeName=All%20DType&deviceName=Apple%20iPhone%2015%20(iOS%2018.0)&archName=All%20Platforms) shows both iOS 18.0 and 18.5 The preview dashboards show only the selected device (arch), for example: * [vLLM B200](https://torchci-git-fork-huydhn-fix-device-selection-fbopensource.vercel.app/benchmark/llms?startTime=Tue%2C%2005%20Aug%202025%2002%3A10%3A13%20GMT&stopTime=Tue%2C%2012%20Aug%202025%2002%3A10%3A13%20GMT&granularity=day&lBranch=main&lCommit=f825c6bd22133a8b2242457069f59654a2ae401b&rBranch=main&rCommit=458e74eb907f96069e6d8a4f3c9f457001fef2ea&repoName=vllm-project%2Fvllm&benchmarkName=&modelName=All%20Models&backendName=All%20Backends&modeName=All%20Modes&dtypeName=All%20DType&deviceName=cuda%20(NVIDIA%20B200)&archName=All%20Platforms) * [ExecuTorch iOS 18.0](https://torchci-git-fork-huydhn-fix-device-selection-fbopensource.vercel.app/benchmark/llms?startTime=Tue%2C%2005%20Aug%202025%2002%3A10%3A38%20GMT&stopTime=Tue%2C%2012%20Aug%202025%2002%3A10%3A38%20GMT&granularity=day&lBranch=main&lCommit=123fe91a5bd4e582f0d660926ec9fea620f5c9c5&rBranch=main&rCommit=8e858570a2dba922142cad667f7ba84f82e2859a&repoName=pytorch%2Fexecutorch&benchmarkName=&modelName=All%20Models&backendName=All%20Backends&modeName=All%20Modes&dtypeName=All%20DType&deviceName=Apple%20iPhone%2015%20(iOS%2018.0)&archName=All%20Platforms) --------- Signed-off-by: Huy Do <[email protected]>
1 parent d4b2e6a commit 00da186

File tree

1 file changed

+24
-2
lines changed

1 file changed

+24
-2
lines changed

torchci/lib/benchmark/llms/utils/llmUtils.ts

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,31 @@ export function getLLMsBenchmarkPropsQueryParameter(props: LLMsBenchmarkProps) {
5858
dtypes = [props.dtypeName];
5959
}
6060

61+
const deviceName =
62+
props.deviceName === DEFAULT_DEVICE_NAME ? "" : props.deviceName;
63+
const archName = props.archName === DEFAULT_ARCH_NAME ? "" : props.archName;
64+
65+
let device = "";
66+
let arch = "";
67+
if (archName === "") {
68+
// All the dashboards currently put device and arch into the same field in
69+
// device (arch) format, i.e. cuda (NVIDIA B200). So, we need to extract
70+
// the arch name here to use it in the query
71+
const deviceArchRegex = new RegExp("^(?<device>.+)\\s+\\((?<arch>.+)\\)$");
72+
const m = deviceName.match(deviceArchRegex);
73+
74+
device =
75+
m !== null && m.groups !== undefined ? m.groups.device : deviceName;
76+
arch = m !== null && m.groups !== undefined ? m.groups.arch : archName;
77+
} else {
78+
// If both device and arch are set, we just need to use them as they are
79+
device = deviceName;
80+
arch = archName;
81+
}
82+
6183
const queryParams = {
62-
arch: props.archName === DEFAULT_ARCH_NAME ? "" : props.archName,
63-
device: props.deviceName === DEFAULT_DEVICE_NAME ? "" : props.deviceName,
84+
arch: arch,
85+
device: device,
6486
mode: props.modeName === DEFAULT_MODE_NAME ? "" : props.modeName,
6587
dtypes: dtypes,
6688
excludedMetrics: EXCLUDED_METRICS,

0 commit comments

Comments
 (0)