Skip to content

Tools

Jump to bottom
Jdaie edited this page Jun 27, 2026 · 1 revision

LLM Tools

Whisplay AI Chatbot can expose function-calling tools to the configured LLM. When the selected LLM backend supports tool/function calling, the model can decide to call these tools during a conversation and use the returned result in its answer.

Tool support depends on the LLM provider and model. If a model does not support tools, disable tool calling for that provider in .env to avoid API errors.

Web Tools

Web tools let the LLM search for current information and read webpages. They are useful for questions about recent news, documentation, prices, releases, weather, or any information that may have changed after the model was trained.

By default, Whisplay registers these tools when WEB_SEARCH_ENABLED is not set to false:

Tool Purpose
web_search Search the web and return compact result titles and URLs.
fetch_webpage Fetch a webpage, extract readable text, and list links on the page.

web_search supports three search types:

search_type Backend
web DuckDuckGo HTML search
news Google News RSS, with DuckDuckGo fallback
sites Google Programmable Search when configured, with DuckDuckGo fallback

fetch_webpage can open a URL directly, or follow a link from the previous fetched page by link_text or link_index. This lets the model search first, open a result, then open a related link from that page.

Basic Configuration

# Enabled by default. Set false to disable all web tools.
WEB_SEARCH_ENABLED=true

# Web tools use the global proxy settings when present.
HTTPS_PROXY=http://127.0.0.1:7897
HTTP_PROXY=http://127.0.0.1:7897
ALL_PROXY=socks5://127.0.0.1:7897

# Timeout for each web/page request.
WEB_TOOL_TIMEOUT_MS=30000

# Search/page result limits.
WEB_SEARCH_MAX_RESULTS=5
WEB_PAGE_TEXT_MAX_CHARS=6000
WEB_PAGE_LINK_MAX_RESULTS=30

Legacy API Search Tools

Whisplay also keeps backward-compatible API-backed tools:

Tool Purpose
webSearch Search using Tavily, SerpAPI, Bing, or Google Custom Search.
webImageSearch Search images through the configured legacy provider.

These legacy tools are only registered when WEB_SEARCH_LEGACY_ENABLED=true or when a valid provider API key is configured.

WEB_SEARCH_LEGACY_ENABLED=true
WEB_SEARCH_PROVIDER=tavily
TAVILY_API_KEY=your_tavily_api_key

# Optional image search.
WEB_SEARCH_INCLUDE_IMAGES=true

Supported legacy providers:

Provider Required variables
tavily TAVILY_API_KEY
serp SERP_API_KEY
bing BING_SEARCH_API_KEY
google GOOGLE_SEARCH_API_KEY, GOOGLE_SEARCH_CX

Example Prompts

  • "Search the latest Raspberry Pi OS release notes."
  • "Open the first result and summarize the installation steps."
  • "Find recent news about Raspberry Pi AI HAT+."
  • "Search this site for LLM8850 setup instructions."

Hardness Command Tool

The Hardness command tool gives the LLM a restricted local command-line interface. It is designed for trusted local or edge deployments where the model may need to inspect files, check service status, query network state, run diagnostics, and follow multi-step operating procedures.

If the selected model is capable enough, Hardness can turn Whisplay from a simple voice chatbot into a lightweight local agent. With runCommand, checkCommand, stopCommand, listSkills, and readSkill, the model can inspect the device, gather context, read local instructions, run safe diagnostic commands, wait for background jobs, and continue reasoning from real command output.

It is disabled by default because it can execute shell commands on the device.

Warning: Enable Hardness only in trusted deployments. Even with the default restrictions, the model can run local commands and read local command output. Tool calls also add command results, status snapshots, and skill contents back into the conversation, which may significantly increase token usage and API cost.

When enabled, Whisplay registers:

Tool Purpose
runCommand Run a restricted shell command. Long-running commands continue in the background and return a job_id.
checkCommand Check a background command returned by runCommand.
stopCommand Stop a queued or running background command.
listSkills List local skill folders when skill tools are enabled.
readSkill Read a skill's SKILL.md content when skill tools are enabled.

Agent-Style Workflows

Hardness is most useful with models that can plan, call tools reliably, and continue from intermediate results. For example, a capable model can:

  • Diagnose why a service is not starting by checking logs, process state, disk space, network state, and config files.
  • Follow local skill instructions from SKILL.md before running commands.
  • Start a long-running command, receive a job_id, continue the conversation, and call checkCommand after the suggested delay.
  • Stop a queued or running background command if the task changes.
  • Combine web tools with local commands, such as reading online docs and then checking the local installation.

This agent-like behavior is only as reliable as the model and deployment context. Use narrow prompts, keep dangerous mode disabled, and review the commands it attempts to run when operating on important devices.

Security Model

With the default safety policy, commands are limited by:

  • An allowlist of common read/diagnostic commands such as pwd, ls, cat, head, tail, find, grep, rg, date, hostname, ip, ping, curl, mkdir, printf, echo, wc, du, df, ps, whoami, uname, sed, awk, sort, uniq, and cut.
  • Rejection of dangerous patterns such as sudo, su, rm, dd, mkfs, reboot/shutdown commands, destructive systemctl/service actions, command substitution, backgrounding, and multiline commands.
  • Write-path checks for redirection, mkdir, and curl -o. Writes are restricted to user or temp locations and refused under protected system paths.
  • Command length, concurrency, output truncation, and background-job limits.

Do not enable HARDNESS_COMMAND_ALLOW_DANGEROUS=true unless the model, prompt, users, network, and device are fully trusted.

Basic Configuration

# Disabled by default.
HARDNESS_COMMAND_TOOL_ENABLED=true

# Keep false for normal deployments.
HARDNESS_COMMAND_ALLOW_DANGEROUS=false

# Foreground wait before returning a background job_id.
HARDNESS_COMMAND_FOREGROUND_TIMEOUT_MS=10000

# Maximum number of command jobs running at the same time.
HARDNESS_COMMAND_MAX_CONCURRENT=2

# Suggested delay between background status checks.
HARDNESS_COMMAND_CHECK_AFTER_SECONDS=15

# Output handling.
HARDNESS_COMMAND_SPILL_CHARS=4000
HARDNESS_COMMAND_RETURN_CHARS=1200
HARDNESS_COMMAND_TEMP_DIR=/tmp/whisplay-hardness

Skill Tools

Skill tools are added alongside runCommand when HARDNESS_SKILL_TOOL_ENABLED is not set to false.

HARDNESS_SKILL_TOOL_ENABLED=true

# Optional comma- or colon-separated skill roots.
# Each skill is a directory containing SKILL.md.
HARDNESS_SKILL_DIRS=./skills

# Maximum characters returned from a SKILL.md file.
HARDNESS_SKILL_RETURN_CHARS=8000

Example Prompts

  • "Check whether the chatbot service is running."
  • "Show the last 50 lines of the chatbot log."
  • "Check the current IP address."
  • "List the local skills you can use."
  • "Read the display debugging skill."

Provider Notes

Some LLM providers expose tools by default, while others require an explicit flag:

Provider Common setting
OpenAI-compatible OPENAI_ENABLE_TOOLS=true
Ollama OLLAMA_ENABLE_TOOLS=true
OpenRouter OPENROUTER_ENABLE_TOOLS=true
Perplexity PERPLEXITY_ENABLE_TOOLS=true with a tool-capable model such as sonar-pro

If the provider returns a tool/function-calling error, confirm that the selected model supports tools and that the provider-specific tool flag is set correctly.

Clone this wiki locally