`tt-smi` -s causes System Hang/Device Fault in Kubernetes Environment (concurrent with `tt-topology`/`ttnn`)

[hongsu]

Description: We are operating in a Kubernetes environment. We monitor our devices by periodically executing the tt-smi -s command (e.g., via liveness probe, sidecar, or cronjob) to collect metrics and heartbeats.

However, we have observed a critical stability issue where the system either hangs or results in a Device Fault when tt-smi -s is executed while a Pod is running tt-topology or ttnn workloads.

Steps to Reproduce:

1. Deploy a Pod running a workload using ttnn or tt-topology in a Kubernetes cluster.
2. While the workload is active, execute tt-smi -s periodically (e.g., inside the same container, a sidecar, or via kubectl exec).
3. Observe that the process hangs or the device enters a fault state.

Expected Behavior: tt-smi commands should be able to run safely for monitoring purposes within a Kubernetes environment without interfering with active ttnn/tt-topology workloads.

Questions & Requests: We need a solution to continue monitoring the device status in our cluster without causing instability.

1. Alternative Monitoring Methods: Is there any other recommended way to collect metrics and heartbeats safely while tt-topology or ttnn is in use?
2. Fix Timeline: If this requires a fix in the library, driver, or device plugin, could you please provide an estimated timeline (ETA)?

Environment:

• Orchestrator: Kubernetes
• OS / Kernel: Ubuntu 22.04
• Machine: LoudBox

[sbansalTT]

May I ask what versions of the following you have on your system -

• tt-smi
• tt-kmd
• tt-topology
• ttnn
• fw versions flashed on the cards (can be found on tab 3 of tt-smi)

We have some support for this style of concurrent execution already merged in newer versions of these pieces of SW. So upgrading and trying again would be my first recommendation. (My assumption is that you are running on a WH loudbox.)

I am a bit confused by what you mean by "tt-topology workload" since tt-topology is a tool that is used to flash chip coordinates and cannot run any AI workloads.

[hongsu]

May I ask what versions of the following you have on your system -

• tt-smi
• tt-kmd
• tt-topology
• ttnn
• fw versions flashed on the cards (can be found on tab 3 of tt-smi)

We have some support for this style of concurrent execution already merged in newer versions of these pieces of SW. So upgrading and trying again would be my first recommendation. (My assumption is that you are running on a WH loudbox.)

I am a bit confused by what you mean by "tt-topology workload" since tt-topology is a tool that is used to flash chip coordinates and cannot run any AI workloads.

Hello, Thank you for the quick response and the recommendation regarding the upgrade.

Here are the specific versions currently running on my system (WH Loudbox):

• tt-smi : 3.0.39
• tt-kmd : 2.5.0
• tt-topology : 1.2.16
• ttnn : 0.1.dev500
• fw : 0.3.4.9

Regarding the confusion about the "tt-topology workload": To clarify, I used that term to refer to the operation of changing the chip layout. I was specifically referring to running the command: tt-topology -l mesh.