Nav: guided on-site navigation

[functor-flow]

Nav: guided on-site navigation

full spec at spec

flowchart TD
  Task["SearchTask"]
  HistBefore["SearchHistory (before)"]
  StartUrl["starting URL (promising site)"]
  NavCtx["nav context (links/sections)"]
  Budget["nav budget (depth/steps)"]

  Gate{"nav gate: should we navigate?"}

  subgraph NavMod["Nav module (navigateSite)"]
    Init["setup navigator state"]
    Step["run navigation step"]
    Record["record step + visited URL"]
    Check{"more steps allowed?"}
  end

  HistAfter["SearchHistory (after)"]
  NewUrls["new URLs for Crawler"]

  Task --> Gate
  HistBefore --> Gate
  StartUrl --> Gate
  NavCtx --> Gate

  Gate -- "yes" --> Init
  Gate -- "no" --> HistAfter

  Budget --> Init
  Init --> Step
  Step --> Record
  Record --> Check

  Check -- "yes" --> Step
  Check -- "no" --> HistAfter

  Record --> NewUrls

Loading

Status: Navigation engine implemented; not yet connected to the evidence loop.

Role. When a site looks promising but a single page crawl is not enough, use the Occam + Playwright stack to click around a bit and discover more pages for Crawler → Embeddings.

Responsibilities.

• Take a SearchTask, the current SearchHistory, and a starting URL that Serper/Embeddings/Decision marked as promising.
• Before starting navigation, build a small navigation context (e.g. key links/sections from the current page) and run a simple LLM prompt (nav gate) that decides if deeper navigation is likely to surface the missing evidence.
• When the nav gate says "navigate", run a short, budgeted navigation session on that domain using the existing Occam navigator (depth/step limits, simple stop conditions).
• For each new URL discovered during navigation, call the Crawler boundary to fetch the page and then the Embeddings helpers to attach evidence, updating SearchHistory.
• Mark all navigation-derived visits in SearchHistory.pages with source = "nav", and keep enough depth/attempt information so Decision can see how hard we tried on that site.

---

TODO

• Define a Nav boundary function, e.g. navigateSite(task, history, startUrl, navBudget), that runs one bounded navigation session and returns the updated SearchHistory (and optionally a small navigation summary object).
• Design and implement a small nav-gate helper (e.g. shouldNavigate(task, history, navContext)) that builds the navigation context and calls a focused LLM prompt to decide whether navigation should run at all.
• Wire the existing Occam navigator into this boundary so that every URL it discovers flows through the same crawlUrl → attachEvidenceFromPage path as SERP URLs.
• Make sure SearchHistory captures basic navigation metadata (source "nav", depth/step counters) but hides Playwright/Occam internals from the rest of the pipeline.

[linear]

SWARM-69