|
| 1 | +### [Promptfoo](https://github.com/promptfoo/promptfoo) |
| 2 | + |
| 3 | +> Handle: `promptfoo`<br/> |
| 4 | +> URL: [http://localhost:34233](http://localhost:34233)<br/> |
| 5 | +
|
| 6 | + |
| 7 | + |
| 8 | +[](https://npmjs.com/package/promptfoo) |
| 9 | +[](https://npmjs.com/package/promptfoo) |
| 10 | +[](https://github.com/promptfoo/promptfoo/actions/workflows/main.yml) |
| 11 | + |
| 12 | +[](https://discord.gg/gHPS9jjfbs) |
| 13 | + |
| 14 | +`promptfoo` is a tool for testing, evaluating, and red-teaming LLM apps. |
| 15 | + |
| 16 | +With promptfoo, you can: |
| 17 | + |
| 18 | +- **Build reliable prompts, models, and RAGs** with benchmarks specific to your use-case |
| 19 | +- **Secure your apps** with automated [red teaming](https://www.promptfoo.dev/docs/red-team/) and pentesting |
| 20 | +- **Speed up evaluations** with caching, concurrency, and live reloading |
| 21 | +- **Score outputs automatically** by defining [metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs) |
| 22 | +- Use as a [CLI](https://www.promptfoo.dev/docs/usage/command-line), [library](https://www.promptfoo.dev/docs/usage/node-package), or in [CI/CD](https://www.promptfoo.dev/docs/integrations/github-action) |
| 23 | +- Use OpenAI, Anthropic, Azure, Google, HuggingFace, open-source models like Llama, or integrate custom API providers for [any LLM API](https://www.promptfoo.dev/docs/providers) |
| 24 | + |
| 25 | +#### Starting |
| 26 | + |
| 27 | +```bash |
| 28 | +# [Optional] Pre-pull the image |
| 29 | +harbor pull promptfoo |
| 30 | +``` |
| 31 | + |
| 32 | +You'll be running Promptfoo CLI most of the time, it's available as: |
| 33 | + |
| 34 | +```bash |
| 35 | +# Full name |
| 36 | +harbor promptfoo --help |
| 37 | + |
| 38 | +# Alias |
| 39 | +harbor pf --help |
| 40 | +``` |
| 41 | + |
| 42 | +Whenever the CLI is called, it'll also automatically start local Promptfoo backend. |
| 43 | + |
| 44 | +```bash |
| 45 | +# Run a CLI command |
| 46 | +harbor pf --help |
| 47 | + |
| 48 | +# Promptfoo backend started |
| 49 | +harbor ps # harbor.promptfoo |
| 50 | +``` |
| 51 | + |
| 52 | +Promptfoo backend serves all recorded results in the web UI: |
| 53 | + |
| 54 | +```bash |
| 55 | +# Open the web UI |
| 56 | +harbor open promptfoo |
| 57 | +harbor promptfoo view |
| 58 | +harbor pf o |
| 59 | +``` |
| 60 | + |
| 61 | +#### Usage |
| 62 | + |
| 63 | +Most of the time, your workflow will be centered around creating prompts, assets, writing an eval config, running it and then viewing the results. |
| 64 | + |
| 65 | +Harbor will run `pf` CLI from where you call Harbor CLI, so you can use it from any folder on your machine. |
| 66 | + |
| 67 | +```bash |
| 68 | +# Ensure a dedicated folder for the eval |
| 69 | +cd /path/to/your/eval |
| 70 | + |
| 71 | +# Init the eval (here) |
| 72 | +harbor pf init |
| 73 | + |
| 74 | +# Edit the configuration, prompts as needed |
| 75 | +# Run the eval |
| 76 | +harbor pf eval |
| 77 | + |
| 78 | +# View the results |
| 79 | +harbor pf view |
| 80 | +``` |
| 81 | + |
| 82 | +> [!NOTE] |
| 83 | +> If you're seeing any kind of file system permission errors you'll need to ensure that files written from within a container are [accessible to your user](../docs/1.-Harbor-User-Guide#file-system-permissions). |
| 84 | +
|
| 85 | +#### Configuration |
| 86 | + |
| 87 | +Harbor pre-configures `promptfoo` to run against `ollama` out of the box (must be started before `pf eval`). Any other providers can be configured via: |
| 88 | + |
| 89 | +- env vars (see [`harbor env`](../docs/3.-Harbor-CLI-Reference#harbor-env)) |
| 90 | +- directly in promptfooconfig files (see [Providers reference](https://www.promptfoo.dev/docs/providers/) in the official documentation) |
| 91 | + |
| 92 | +```bash |
| 93 | +# For example, use vLLM API |
| 94 | +harbor env promptfoo OPENAI_BASE_URL $(harbor url -i vllm) |
| 95 | +``` |
| 96 | + |
| 97 | +Promptfoo is a very rich and extensive tool, we recommend reading through excellent [official documentation](https://www.promptfoo.dev/docs/intro) to get the most out of it. |
| 98 | + |
| 99 | +Harbor comes with two (basic) built-in examples. |
| 100 | + |
| 101 | +##### Promptfoo hello-world |
| 102 | + |
| 103 | +```bash |
| 104 | +# Navigate to eval folder |
| 105 | +cd $(harbor home)/services/promptfoo/examples/hello-promptfoo |
| 106 | + |
| 107 | +# Start ollama and pull the target model |
| 108 | +harbor up ollama |
| 109 | +harbor ollama pull llama3.1:8b |
| 110 | + |
| 111 | +# Run the eval |
| 112 | +harbor pf eval |
| 113 | + |
| 114 | +# View the results |
| 115 | +harbor pf view |
| 116 | +``` |
| 117 | + |
| 118 | +##### Promptfoo temp-test |
| 119 | + |
| 120 | + |
| 121 | + |
| 122 | +Evaluate a model across a range of temperatures to see if there's a sweet spot for a given prompt. |
| 123 | + |
| 124 | +```bash |
| 125 | +# Navigate to eval folder |
| 126 | +cd $(harbor home)/services/promptfoo/examples/temp-test |
| 127 | + |
| 128 | +# Start ollama and pull the target model |
| 129 | +harbor up ollama |
| 130 | +harbor ollama pull llama3.1:8b |
| 131 | + |
| 132 | +# Run the eval |
| 133 | +harbor pf eval |
| 134 | + |
| 135 | +# View the results |
| 136 | +harbor pf view |
| 137 | +``` |
0 commit comments