Skip to content

feat(backend): locate-anything-cpp (open-vocabulary object detection via ggml)#10264

Merged
mudler merged 7 commits into
masterfrom
feat/locate-anything-backend
Jun 12, 2026
Merged

feat(backend): locate-anything-cpp (open-vocabulary object detection via ggml)#10264
mudler merged 7 commits into
masterfrom
feat/locate-anything-backend

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

What

Adds a new LocalAI backend locate-anything-cpp for open-vocabulary object detection (visual grounding): given an image and a free-text prompt, it returns labeled boxes. It wraps locate-anything.cpp - a C++/ggml port of NVIDIA's LocateAnything-3B - via its flat C ABI.

Same family as the existing detection backends: it mirrors backend/go/rfdetr-cpp/ almost exactly (Go + purego Dlopen, no cgo; a CMake MODULE lib static-links ggml into a per-CPU-variant .so) and implements the gRPC Detect RPC. Open-vocab means it consumes DetectOptions.prompt (the same field SAM3 uses).

How it maps

  • Detect(DetectOptions{Src, Prompt}) -> base64 image to temp file -> la_capi_locate_path(ctx, img, prompt, hybrid) -> DetectResponse{ Detection{X, Y, Width, Height, Confidence=1.0, ClassName=label} }. A prompt is required (open-vocabulary). la_capi exposes a label per box (no class-id/score/mask), so Confidence is reported as 1.0 (greedy detection is deterministic).
  • Static-links ggml so the .so only depends on libc/libstdc++/libgomp (bundled by the package step), like rfdetr-cpp.

Changes

  • backend/go/locate-anything-cpp/ - the backend (main.go, golocateanythingcpp.go, CMakeLists.txt, Makefile, run.sh, package.sh, + Load/Detect wire test).
  • .github/backend-matrix.yml - 9 build entries mirroring rfdetr-cpp (cuda-12/13, l4t-arm64, cpu, intel-sycl f16/f32, vulkan).
  • backend/index.yaml + core/gallery/importers/locate-anything.go (+ test) - gallery entry + model importer (routes locate-anything GGUFs to this backend).
  • Makefile - wired into test-extra.

Validation

Built locally (4 .so variants + the Go binary, CGO_ENABLED=0, go vet/gofmt clean). The Detect wire test passes end-to-end on the real model: gRPC LoadModel (q8_0 GGUF) -> Detect on a COCO street image with prompt person</c>car -> labeled boxes (TestDetect PASS, ~50s on CPU). Importer tests pass (go test ./core/gallery/importers/...).

Follow-ups

  • bump_deps.yaml: the daily auto-bump registration for LOCATEANYTHING_VERSION is not in this PR - the pushing token lacked workflow scope. It should be added to .github/workflows/bump_deps.yaml:
    - repository: "mudler/locate-anything.cpp"
      variable: "LOCATEANYTHING_VERSION"
      branch: "master"
      file: "backend/go/locate-anything-cpp/Makefile"
  • DCO: commits carry an Assisted-by: trailer per the AI-assistance policy; they need the human submitter's Signed-off-by to pass the DCO check.

AI-assisted (Claude Code, Opus 4.8); the human submitter owns, reviews, and is responsible for the change.

mudler added 5 commits June 11, 2026 23:04
…via la_capi)

A Go/purego backend wrapping locate-anything.cpp's la_capi C ABI, implementing
the gRPC Detect RPC: image + open-vocabulary text prompt -> labeled boxes.
Mirrors backend/go/rfdetr-cpp; static-links ggml into a per-CPU-variant .so.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
detections := make([]*pb.Detection, 0, n)
for i := int32(0); i < n; i++ {
var xyxy [4]float32 // x1, y1, x2, y2
if CapiGetDetectionBox(r.handle, i, uintptr(unsafe.Pointer(&xyxy[0]))) != 0 {
need := CapiGetDetectionLabel(r.handle, i, 0, 0)
if need > 0 {
buf := make([]byte, need)
CapiGetDetectionLabel(r.handle, i, uintptr(unsafe.Pointer(&buf[0])), need)
@localai-bot localai-bot force-pushed the feat/locate-anything-backend branch from 733b646 to 188cad1 Compare June 11, 2026 23:05
mudler added 2 commits June 11, 2026 23:08
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: mudler <mudler@localai.io>
…3B + image, runs Detect)

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: mudler <mudler@localai.io>
@mudler mudler merged commit 56cc4f6 into master Jun 12, 2026
66 checks passed
@mudler mudler deleted the feat/locate-anything-backend branch June 12, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants