Skip to content

Conversation

hipudding
Copy link
Collaborator

Make sure to read the contributing guidelines before submitting a PR

@hipudding hipudding marked this pull request as ready for review August 20, 2025 01:42
@github-actions github-actions bot added ggml changes relating to the ggml tensor library for machine learning Ascend NPU issues specific to Ascend NPUs labels Aug 20, 2025
@hipudding hipudding requested review from Copilot and ggerganov August 20, 2025 03:18
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes the rope cache implementation in the CANN backend by changing the initialization strategy from pre-allocating a fixed-size cache to dynamically allocating memory only when needed.

  • Changes the initial max_prompt_length from a hardcoded value of 65536 to 0
  • Removes the upfront allocation of sin/cos cache memory
  • Adds proper memory deallocation before reallocating when cache needs to grow

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
ggml/src/ggml-cann/common.h Initialize max_prompt_length to 0 instead of 65536
ggml/src/ggml-cann/aclnn_ops.cpp Remove upfront cache allocation and add proper cleanup before reallocation

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

if (ctx.sin_ptr != nullptr)
ACL_CHECK(aclrtFree(ctx.sin_ptr));
if (ctx.cos_ptr != nullptr)
ACL_CHECK(aclrtFree(ctx.cos_ptr));
Copy link

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The conditional check for ctx.cos_ptr != nullptr is missing braces around the ACL_CHECK(aclrtFree(ctx.cos_ptr)) call on line 2226. This creates a logic error where the free operation is always executed regardless of the null check.

Suggested change
ACL_CHECK(aclrtFree(ctx.cos_ptr));
if (ctx.cos_ptr != nullptr) {
ACL_CHECK(aclrtFree(ctx.cos_ptr));
}

Copilot uses AI. Check for mistakes.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fine because there's only one line in if.

@hipudding
Copy link
Collaborator Author

This optimize has already done in #15501

@hipudding hipudding closed this Aug 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ascend NPU issues specific to Ascend NPUs ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant