[Performance]: Online service test: Sending the same request twice consecutively does not generate BLOCK files, but the prefix cache hit rate will change

[soilderone]

Proposal to improve performance

No response

Report of performance regression

I am testing vllm online service on 910b3 with vllm-ascend 0.9.2 rc1 and ucm v0.1.0. When I conduct offline tests, BLOCK files can be generated.But when I conduct online tests, BLOCK files can not be generated, but the prefix cache hit rate changed.

The following is my startup command：

ASCEND_RT_VISIBLE_DEVICES=6 vllm serve /app/model/Qwen2.5-14B-Instruct \
	--served-model-name vllm_nfs_offload \
	--max-model-len 4096 \
	--tensor-parallel-size 1 \
	--gpu-memory-utilization 0.8 \
	--trust-remote-code \
	--port 5006 \
	--kv-transfer-config '{
    "kv_connector": "UCMConnector",
    "kv_connector_module_path": "ucm.integration.vllm.ucm_connector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "ucm_connectors": [
        {
          "ucm_connector_name": "UcmNfsStore",
          "ucm_connector_config": {
            "storage_backends": "/app/storage",
            "use_direct": false
          }
        }
      ],
      "ucm_sparse_config": {
        "KVStarMultiStep": {
          "init_window_sz": 1,
          "local_window_sz": 2,
          "sparse_ratio": 0.25,
          "retrieval_stride": 8,
          "blk_repre_dim_prune_ratio": 0.25,
          "blk_repre_inner_token_merge": 2
        }
      }
    }
  }'

The following is part of the log output of the first request：

The following is part of the log output of the second request：

The content under storage_backends：

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

[qyh111]

1、Can you provide your prompt?
2、 you should add --no-enable-prefix-caching,otherwise it will not go though remote storage load/save

[soilderone]

Here is my prompt:
以下是我的个人资料和系统指令内容：[客户张伟（会员ID：VIP88001），于2018年5月10日注册成为本公司“恒星级”铂金会员，现已连续五年保持最高等级，其个人档案显示，张先生出生于1975年，主要居住地为上海市徐汇区，联系电话已更新为最新的139****1234。其年度消费总额近三年均稳定在人民币50,000元以上，累计消费服务次数达260次，属于高价值活跃客户。目前账户积分余额为35,890点，其中有12,500点将在今年12月31日到期清零，急需提醒兑换。张先生主要的消费偏好集中在金融理财顾问服务（占比60%）和高端健康管理套餐（占比30%），尤其是对“全球视野投资”和“定制化抗衰老体检”两项服务反馈良好。根据最新的《恒星级会员权益条款V3.0》，张先生有权享受所有产品的最高折扣率（15%），一名专属高级顾问的全天候服务，以及每年一次的免费豪华行政套房升级权益。请注意，张先生上一次积分兑换发生在去年10月，兑换了一张洲际商务舱升舱券。系统建议本次服务中重点提醒其即将到期的积分，并推荐兑换最新的家庭亲子旅行服务包。]。请根据我的个人资料总结我的权益。

As mentioned in the document（nfs_store.md）：

[qyh111]

1、How about remove sparse related config and try again?If you just want to use prefix cache. Cause our patch for meet some problem.
2、You're right, in production，we will use both use hbm prefix cache, which is faster and remote storage prefix cache, which provides larger space . But if you want to test ucm first .i think it is better to add --no-enable-prefix-caching

[soilderone]

1.When removing sparse related config, it can generate BLOCK files.
2.When i add --no-enable-prefix-caching, the prefix cache hit rate change to 0. There is no load process in the log

[qyh111]

The prefix cache hit rate reported by vLLM refers to the hit rate inside HBM. Since --no-enable-prefix-caching is enabled, all HBM hit rates become 0, and every prefix-cache lookup will fall back to external storage.

If you want to observe the hit rate of the external storage, you can search for the keyword “hit external” in the logs.