Skip to content

Improving quantized matmul performance by devectorizing shader. #15274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 22, 2025
Merged

Improving quantized matmul performance by devectorizing shader. #15274

merged 1 commit into from
Oct 22, 2025
+36 −22

Conversation

trivedivivek
Copy link
Contributor

Summary:
This diff improves the performance of quantized matrix multiplication by devectorizing the shader.

An example modification is shown below:

// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}

Differential Revision: D85023829

@trivedivivek trivedivivek requested a review from SS-JIA as a code owner October 20, 2025 15:46
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Oct 20, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15274

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures, 2 Unrelated Failures

As of commit 29e995a with merge base 82611e9 (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 20, 2025
@meta-codesync Meta CodeSync
Copy link

meta-codesync bot commented Oct 20, 2025

@trivedivivek has exported this pull request. If you are a Meta employee, you can view the originating Diff in D85023829.

@trivedivivek trivedivivek added the release notes: vulkan Changes to the Vulkan backend delegate label Oct 20, 2025
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 20, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
4913210 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Differential Revision: D85023829
@trivedivivek trivedivivek force-pushed the export-D85023829 branch 2 times, most recently from 4913210 to b831b18 Compare October 20, 2025 22:09
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 20, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
b831b18 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
88f7aa0 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
1a04777 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
29a03da 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
81e68f6 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
d4b474d 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
cf198e1 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
3f8c735 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 22, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
17fc715 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
29e995a 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
@meta-codesync meta-codesync bot merged commit 788ef2f into pytorch:main Oct 22, 2025
138 of 149 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SS-JIA SS-JIA SS-JIA approved these changes

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported meta-exported release notes: vulkan Changes to the Vulkan backend delegate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@trivedivivek @SS-JIA