GPU statistics for Intel `xe` kernel driver devices (Intel Arc B-series cards and optionally A-series)

[coledeck]

Describe what you are trying to accomplish and why in non technical terms
Currently intel_gpu_top is polled for Intel GPU stats. That only supports the i915 kernel driver, but Arc battlemage (i.e. B580) cards use the xe kernel driver, so stats are unavailable. A-series cards can also be manually switched to xe for (in theory) better performance and would have the same issue, and it may become the default driver in the (mid to far) future for A-series.

Describe the solution you'd like
Use alternate stats polling method. Likely would need to be polling drm kernel interface or use another utility. See nvtop's code for example: https://github.com/Syllo/nvtop/blob/master/src/extract_gpuinfo_intel_xe.c

nvtop supports a crap ton of GPUs, maybe could just integrate that instead of dealing with polling everything ourselves? https://github.com/Syllo/nvtop/tree/master?tab=readme-ov-file#gpu-support

Describe alternatives you've considered
I'm okay with pulling up nvtop on my system manually - it's not a huge deal to be missing the stats in frigate - just wanted to put this out there as I'm not sure if you guys know about the lack of support for intel's own tools going forward with their new GPUs.

Additional context
Inference times with Arc B580 if you want to add to the wiki (running 0.17.0-beta2):
yolov9s 640x640 2026.0 base with openvino: ~9.0ms
yolov9s 320x320 2026.0 base with openvino: ~4.8ms

Going to be attempting to run 30 cameras with detection and all fancy features with this GPU. We'll see how it goes.

[NickM-27]

Is there a reason to believe intel_gpu_top won't support this driver in the future? Would seem odd to me that they wouldn't

[coledeck]

Is there a reason to believe intel_gpu_top won't support this driver in the future? Would seem odd to me that they wouldn't

Just looked into it, it looks like intel has switched to using a new CLI utility, gputop instead of intel_gpu_top. It's included in the same debian package and seems to work. In fact it doesn't even require CAP_PERFMON I think - it seems to work when e/g/ nvtop doesn't in an unprivileged LXC at least. Works on A-series and B-series.

intel-gpu-tools: /usr/bin/gputop

[coledeck]

See intel's comments here: https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/issues/153

intel_gpu_top uses several features that are i915-only. The tool that is driver-agnostic is gputop

[cmmh]

gputop and nvtop both show workloads for my Arc B50 battlemage. intel-gpu-top does not show anything. Haven't experimented with permissions, but I needed sudo for both gputop and nvtop outside of docker. I second OP's suggestion that perhaps nvtop is the way to pivot, although, as written, it doesn't have a convenient JSON output (but either does gputop).

[ukasz]

If we don't plan to support Xe gpus on intel_gpu_top, maybe we could at least print info to use gputop if there are no i915 cards, but xe module is loaded.

[mrthosi]

+1 vote for gputop. It works with XE2 out of the box, too. (not like nvtop)

[brontide]

The problem is that intel_gpu_top is formatted as json, gputop has no output in computer readable formats like json or csv.

    intel_gpu_top_command = [
        "timeout",
        "0.5s",
        "intel_gpu_top",
        "-J",
        "-o",
        "-",
        "-s",
        "1000",  # Intel changed this from seconds to milliseconds in 2024+ versions
    ]

Until Intel adds some way to collect the data with json or csv this would be hard to implement.

[brontide]

But nvtop does provide a json output that can be set to 1/s.

nvtop --loop -d 10

[mrthosi]

I use a docker overlay (installing compiled gputop and patching the services.py) for the last 17.x releases, and GPU statistics looks working with this code:

def get_intel_gpu_stats(intel_gpu_device: Optional[str]) -> Optional[dict[str, str]]:
    import re
    import shutil
    import string
    import subprocess as sp

    ansi_re = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")
    percent_re = re.compile(r"(\d+(?:\.\d+)?)%")

    def clean_text(text: str) -> str:
        text = ansi_re.sub("", text)
        text = text.replace("\r", "\n")
        text = "".join(
            ch for ch in text
            if ch in "\n\t" or ch in string.printable or ord(ch) >= 32
        )
        return text

    def extract_candidate_lines(text: str) -> list[str]:
        lines = []
        for raw_line in text.splitlines():
            line = raw_line.strip()
            if not line:
                continue
            if re.match(r"^\d+", line) and "%" in line:
                lines.append(line)
        return lines

    script_bin = shutil.which("script")
    timeout_bin = shutil.which("timeout")
    gputop_bin = shutil.which("gputop") or "/usr/local/bin/gputop"

    if not script_bin or not timeout_bin or not gputop_bin:
        logger.error("Unable to poll intel GPU stats: missing script/timeout/gputop")
        return None

    try:
        cmd = [
            script_bin,
            "-q",
            "-c",
            f"{timeout_bin} -s INT 1 {gputop_bin}",
            "/dev/stdout",
        ]
        p = sp.run(cmd, encoding="utf-8", capture_output=True)
    except Exception as e:
        logger.error(f"Unable to poll intel GPU stats: {e}")
        return None

    output = clean_text((p.stdout or "") + "\n" + (p.stderr or ""))
    lines = extract_candidate_lines(output)

    render = 0.0
    video = 0.0

    for line in lines:
        vals = percent_re.findall(line)
        if len(vals) < 5:
            continue

        rcs = float(vals[0])
        vcs = float(vals[1])
        vecs = float(vals[2])

        render = max(render, rcs)
        video = max(video, vcs + vecs)

    results: dict[str, str] = {}
    results["gpu"] = f"{round((render + video) / 2, 2)}%"
    results["mem"] = "-%"
    return results

Of course, parsing a JSON output would be safer...
(I cannot use nvtop with my xe2 (258V integrated) GPU, unfortunately.)