Skip to content

[PERF] Reduce global inference stalls from Tensor.Destroy lock granularity #44

@tazarov

Description

@tazarov

Summary

Tensor.Destroy() currently acquires ortCallMu.Lock() (global write lock) in /Users/tazarov/GolandProjects/onnx-purego/ort/tensor.go.

This is correct for safety but stalls unrelated Session.Run() calls across all sessions while any tensor is being released.

Context

Why this matters

In multi-session inference pipelines, batch/tensor rotation can cause avoidable global contention.

Proposed direction

  1. Design finer-grained lifetime tracking so releasing one tensor does not block unrelated in-flight runs.
  2. Keep environment teardown safety guarantees intact.
  3. Preserve pure-Go/no-CGO constraints.

Potential approaches:

  • Per-handle active-use tracking (reference counters), or
  • Session/value dependency tracking for targeted waits.

Acceptance criteria

  • Tensor release no longer globally blocks unrelated session inference.
  • Existing concurrency safety tests continue to pass.
  • New regression tests demonstrate reduced cross-session contention without unsafe releases.

Notes

This is a performance/scalability follow-up, not a correctness blocker for MVP.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions