pd disaggregation PREFILL instance llmdatadist init failed

[AlvL1225]

I20251110 15:36:33.291344 280651 llm_data_dist_transfer.cpp:44] Create LlmDataDistTransfer for prefill instance.
F20251110 15:36:33.327589 280651 llm_data_dist_transfer.cpp:75] Check failed: ret == LLM_SUCCESS Initialize LlmDataList failed, ret = 1343270917
*** Check failure stack trace: ***
@ 0x6124dc google::LogMessage::SendToLog()
@ 0x60edf0 google::LogMessage::Flush()
@ 0x612abc google::LogMessageFatal::~LogMessageFatal()
@ 0x1241fc8 xllm::LlmDataDistTransfer::initialize()
@ 0xa9e7ac xllm::WorkerImpl::allocate_kv_cache_with_transfer()
@ 0xa9d3d4 _ZN5folly6detail8function14FunctionTraitsIFvvEE9callSmallIZN4xllm10WorkerImpl37allocate_kv_cache_with_transfer_asyncEmRKSt6vectorIS8_IlSaIlEESaISA_EEEUlvE_EEvRNS1_4DataE
@ 0x1cacb50 xllm::ThreadPool::internal_loop()
@ 0xffff8148314c execute_native_thread_routine
@ 0xffff793dce4c (unknown)
@ 0xffff7944391c (unknown)
@ (nil) (unknown)

[JimHsiung]

Hello, could you please provide your xllm image version and the startup script so that we can identify the issue?

[AlvL1225]

Hello, could you please provide your xllm image version and the startup script so that we can identify the issue?

Hi, I tried all from main branch and I launched both prefill and decode instances successfully.
However I cannot process one single request:

My scripts:

etcd
./etcd --listen-peer-urls 'http://localhost:2390' --listen-client-urls 'http://localhost:2389' --advertise-client-urls 'http://localhost:2391'

xllm-service
export ENABLE_DECODE_RESPONSE_TO_SERVICE=true
cd xllm-service/build/xllm_service && ./xllm_master_serving --etcd_addr="127.0.0.1:2389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=./models/Qwen/Qwen3-0___6B

prefill
./xllm/core/server/xllm --model=./models/Qwen/Qwen3-0___6B
--port=8010
--devices="npu:2"
--master_node_addr="127.0.0.1:18888"
--enable_prefix_cache=false
--enable_chunked_prefill=false
--enable_disagg_pd=true
--instance_role=PREFILL
--max_memory_utilization 0.85
--etcd_addr=127.0.0.1:2389
--transfer_listen_port=26000
--disagg_pd_port=7777
--node_rank=0
--nnodes=1

decode
./xllm/core/server/xllm --model=./models/Qwen/Qwen3-0___6B
--port=8020
--devices="npu:1"
--master_node_addr="127.0.0.1:18898"
--enable_prefix_cache=false
--enable_chunked_prefill=false
--enable_disagg_pd=true
--instance_role=DECODE
--etcd_addr=127.0.0.1:2389
--transfer_listen_port=26100
--disagg_pd_port=7787
--node_rank=0
--nnodes=1

curl script
curl http://localhost:28888/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "Qwen3-0___6B",
"max_tokens": 100,
"temperature": 0.0,
"stream": true,
"messages": [
{
"role": "system",
"content": "Hello good guy"
},
{
"role": "user",
"content": "hello xllm"
}
]
}'

Error Info

W20251113 14:33:02.674055 238318 disagg_pd_scheduler.cpp:204] Failed to find channel to instance: 172.19.0.28:8020, try to create channel now.
terminate called after throwing an instance of 'c10::Error'
what(): Backend doesn't support synchronizing streams.
Exception raised from synchronizeStream at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:193 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffffb0d3d898 in /usr/local/lib64/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0xec (0xffffb0cf6400 in /usr/local/lib64/python3.11/site-packages/torch/lib/libc10.so)
frame #2: + 0x78f1ec (0xffffaed8f1ec in /usr/local/libtorch_npu/lib/libtorch_npu.so)
frame #3: ./xllm/core/server/xllm() [0x24d4b94]
frame #4: ./xllm/core/server/xllm() [0xad06fc]
frame #5: ./xllm/core/server/xllm() [0xad2874]
frame #6: ./xllm/core/server/xllm() [0xac6984]
frame #7: ./xllm/core/server/xllm() [0xb46e70]
frame #8: ./xllm/core/server/xllm() [0xb4a88c]
frame #9: ./xllm/core/server/xllm() [0x1ccd4e0]
frame #10: + 0x19cd24 (0xffffb8e8cd24 in /usr/local/Ascend/ascend-toolkit/latest/lib64/libllm_datadist.so)
frame #11: + 0x7ce4c (0xffffae25ce4c in /usr/lib64/libc.so.6)
frame #12: + 0xe391c (0xffffae2c391c in /usr/lib64/libc.so.6)

F20251113 14:33:02.757658 238325 batch.cpp:113] Check failed: output_idx < num_seqs (0 vs. 0)
*** Check failure stack trace: ***
@ 0x6268cc google::LogMessage::SendToLog()
@ 0x6231e0 google::LogMessage::Flush()
@ 0x626eac google::LogMessageFatal::~LogMessageFatal()
@ 0xb12adc xllm::Batch::process_sample_output()
@ 0xa8ceb4 xllm::LLMEngine::step()
@ 0xa4a4d0 xllm::ContinuousScheduler::step()
@ 0xa59fc4 xllm::DisaggPDScheduler::step()
@ 0x5deb0c _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4xllm9LLMMaster3runEvEUlvE_EEEEE6_M_runEv
@ 0xffffb8e8cd24 execute_native_thread_routine
@ 0xffffae25ce4c (unknown)
@ 0xffffae2c391c (unknown)
@ (nil) (unknown)

[JimHsiung]

What kind of NPU card you are using? Additionally, could you please provide the error log information from the /root/ascend/log/debug/ directory?

[AlvL1225]

What kind of NPU card you are using? Additionally, could you please provide the error log information from the /root/ascend/log/debug/ directory?

I tried re-compile carefully and PD launched successfully, but xllm-service cannot get prefill or decode distances

W20251114 17:46:44.989130 251454 controller.cpp:1606] SIGINT was installed with 1
E20251114 17:47:21.875921 251414 types.h:253] json str:{"addrs":[""],"cluster_ids":[2017634262517310864],"dp_size":1,"k_cache_ids":[1],"name":"172.19.0.28:8010","rpc_address":"172.19.0.28:7777","ttft_profiling_data":[[20480,544],[10240,181],[5120,72],[2560,35],[1280,21],[640,17],[320,16],[160,16],[80,16],[40,19],[20,16],[10,16],[5,16],[2,16]],"type":1,"v_cache_ids":[2]}, parse to instancemetainfo error: [json.exception.out_of_range.403] key 'tpot_profiling_data' not found
E20251114 17:47:21.875972 251414 instance_mgr.cpp:370] pase json:{"addrs":[""],"cluster_ids":[2017634262517310864],"dp_size":1,"k_cache_ids":[1],"name":"172.19.0.28:8010","rpc_address":"172.19.0.28:7777","ttft_profiling_data":[[20480,544],[10240,181],[5120,72],[2560,35],[1280,21],[640,17],[320,16],[160,16],[80,16],[40,19],[20,16],[10,16],[5,16],[2,16]],"type":1,"v_cache_ids":[2]} error!
E20251114 17:47:22.312386 251414 types.h:253] json str:{"addrs":[""],"cluster_ids":[2017634262517310964],"dp_size":1,"k_cache_ids":[1],"name":"172.19.0.28:8010","rpc_address":"172.19.0.28:7787","ttft_profiling_data":[],"type":2,"v_cache_ids":[2]}, parse to instancemetainfo error: [json.exception.out_of_range.403] key 'tpot_profiling_data' not found
E20251114 17:47:22.312425 251414 instance_mgr.cpp:370] pase json:{"addrs":[""],"cluster_ids":[2017634262517310964],"dp_size":1,"k_cache_ids":[1],"name":"172.19.0.28:8010","rpc_address":"172.19.0.28:7787","ttft_profiling_data":[],"type":2,"v_cache_ids":[2]} error!