Skip to content

Commit 5ee2d4c

Browse files
committed
Update readme
1 parent 40a2821 commit 5ee2d4c

File tree

1 file changed

+84
-0
lines changed

1 file changed

+84
-0
lines changed

tests/llm-usage/README.md

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Track language model usage and costs
2+
3+
## Summary
4+
5+
Sample web app that demonstrates how to use Dev Proxy to monitor and track LLM usage and costs.
6+
7+
![GitHub Actions summary for 'Test Dev Proxy LLM Usage with Playwright #38' by garrytrinder. Status: Success. Duration: 3m 40s. Includes LLM usage report for gpt-4-1-2025-04-14 showing token usage and cost in development environment.](/assets/llm-usage-github.png)
8+
9+
The sample showcases:
10+
11+
- **LLM Cost Tracking**: Monitor token usage and costs for OpenAI API calls
12+
- **Functional API Integration**: Serve test data using CrudApiPlugin
13+
- **Latency Simulation**: Add realistic delays to API responses
14+
- **VS Code Integration**: Use Dev Proxy Toolkit for local development and testing
15+
- **End-to-End Testing**: Playwright tests that verify the complete AI pipeline
16+
- **CI/CD Integration**: GitHub Actions workflow with automated testing
17+
18+
## Compatibility
19+
20+
![Dev Proxy v1.0.0-beta.8](https://img.shields.io/badge/devproxy-v1.0.0-green.svg)
21+
22+
## Contributors
23+
24+
* [Garry Trinder](https://github.com/garrytrinder)
25+
26+
## Version history
27+
28+
Version|Date|Comments
29+
-------|----|--------
30+
1.0|July 28, 2025|Initial release
31+
32+
## Minimal path to awesome
33+
34+
1. Clone this repository (or [download this solution as a .ZIP file](https://pnp.github.io/download-partial/?url=https://github.com/pnp/proxy-samples/tree/main/samples/llm-usage) then unzip it)
35+
1. Open the repository in Visual Studio Code
36+
1. Install the [Dev Proxy Toolkit](https://marketplace.visualstudio.com/items?itemName=garrytrinder.dev-proxy-toolkit) extension
37+
38+
### Run locally (with local AI)
39+
40+
> [!NOTE]
41+
>
42+
> For Dev Proxy to simulate AI responses, you will need to run a local model on your machine which is accessible via an OpenAI API compatible endpoint, .e.g. Ollama. By default, this sample uses the Llama 3.2 model, change the `languageModel.model` property in the `.devproxy/simulate-ai.json` file to use a different model. Ensure that the model is running before starting Dev Proxy.
43+
44+
1. Start debug session in Visual Studio Code by pressing <kbd>F5</kbd>
45+
1. Wait for the process to complete
46+
1. Open the markdown file that is created in the root of the project to view the LLM usage report for the run.
47+
48+
### Run locally (with cloud AI)
49+
50+
1. Generate a fine-grained personal access token with `models:read` permission granted.
51+
1. Update the `apiKey` variable value in `js/env.js` with your token.
52+
1. Open **Run and Debug** panel and select **🧪 Run tests, cloud AI & local API** debug configuration
53+
1. Start the debug session by pressing <kbd>F5</kbd>
54+
1. Wait for the process to complete
55+
1. Open the markdown file that is created in the root of the project to view the LLM usage report for the run.
56+
57+
### Run in GitHub Actions
58+
59+
> [!NOTE]
60+
>
61+
> Enable GitHub Actions in your repository settings before running the workflow.
62+
63+
1. Push the **main** branch to your GitHub repository
64+
1. Open a browser and navigate to your repository
65+
1. Open the **Actions** tab in your repository
66+
1. Trigger the **Test Dev Proxy LLM Usage with Playwright** workflow manually
67+
1. Wait for the workflow to complete
68+
1. View the usage report in the job summary
69+
70+
## Help
71+
72+
We do not support samples, but this community is always willing to help, and we want to improve these samples. We use GitHub to track issues, which makes it easy for community members to volunteer their time and help resolve issues.
73+
74+
You can try looking at [issues related to this sample](https://github.com/pnp/proxy-samples/issues?q=label%3A%22sample%3A%20llm-usage%22) to see if anybody else is having the same issues.
75+
76+
If you encounter any issues using this sample, [create a new issue](https://github.com/pnp/proxy-samples/issues/new).
77+
78+
Finally, if you have an idea for improvement, [make a suggestion](https://github.com/pnp/proxy-samples/issues/new).
79+
80+
## Disclaimer
81+
82+
**THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.**
83+
84+
![](https://m365-visitor-stats.azurewebsites.net/SamplesGallery/proxy-samples-llm-usage)

0 commit comments

Comments
 (0)