Planner

.vscode/settings.json

@@ -21,6 +21,8 @@
         "gitmoji",
         "groundedness",
         "groundtruth",
+        "GTEMT",
+        "GTMEs",
         "inputspec",
         "inputspecgen",
         "intentgen",
@@ -36,6 +38,7 @@
         "OKERR",
         "ollama",
         "openaievals",
+        "PPGT",
         "promptid",
         "promptpex",
         "Prompty",

docs/src/content/docs/dev/index.md

@@ -9,7 +9,7 @@ This section provides documentation for developers who want to use PromptPex in
 ## Bring Your Own Inference Library
 
 PromptPex is workflow of LLM prompts that implement the test generation process.
-The template are stored in a [markdown-ish, framework agnostic, template format](/promptpex/configuration/prompt-format/).
+The template are stored in a [markdown-ish, framework agnostic, template format](/promptpex/reference/prompt-format/).
 
 - [prompts directory](https://github.com/microsoft/promptpex/tree/main/src/prompts)
 

docs/src/content/docs/reference/implementation-plan.md

@@ -0,0 +1,85 @@
+---
+title: Implementation Plan
+sidebar:
+    order: 30
+description: An implementation plan for a coding agent to implement the PromptPex framework in any codebase.
+---
+
+Hi there fellow Agent,
+
+The goal of this page is to help you implement the PromptPex framework in your codebase.
+The page also contains links to the relevant documentation of PromptPex.
+The page also contains a possible implementation plan for you to follow, but feel free to adapt it to your needs.
+
+Your task is to implement the PromptPex framework in your codebase according to the requirement of the user. Follow the instructions of the user to choose which language/framework to use, and how to implement the framework in your codebase.
+
+## IMPORTANT
+
+When implementing PromptPex, it is very important to reuse the existing prompts as is. Do not change the prompts unless you need to adapt the syntax for your template engine.
+
+We strongly recommend keep the prompts in separate files and load them
+at runtime, rather than hardcoding them in your codebase.
+
+Whenever possible, use the original `.prompty` files from the `/src/prompts` directory.
+
+## Phase 1: Test Generation
+
+PromptPex is a test generate framework for prompts. It is made of a graph of LLM transformations that eventually generate a set of
+inputs and expected outputs for a given prompt.
+
+- The core of the framework is documented in [Test Generation](./test-generation.md).
+
+## Phase: Validate Test Generation
+
+Once you have implemented the test generation, you should validate it on a prompt.
+
+- generate examples of prompty files (you can use this example as a prompt test case)
+
+```markdown
+---
+tags:
+    - unlisted
+---
+
+system:
+You need to categorize a joke as funny or not.
+Respond with "funny" or "not funny".
+
+user:
+{{joke}}
+```
+
+- for each prompty file, create a test case (using the test framework of your choice) that checks that your implementation is able to run and generate test cases.
+
+- run the tests cases and fix the code until all tests pass.
+
+## Phase 3: Additional features
+
+You will find additional features for promptpex generation that can be useful to add once the basic test generation is working.
+
+- [Groundtruth](./groundtruth.md): how to generate groundtruth for prompts.
+- [Test Samples](./test-samples.md): how to integrate existing test samples into the test generation process.
+- [Test Expansion](./test-expansion.md): how to morph tests into more complex longer texts.
+- [Scenarios](./scenarios.md): how to receive custom set of inputs instructions from the users and use it to guide the generation of tests.
+
+### Notes
+
+You can assume that the secrets are already set in the environment or in a `.env` file
+that can be loaded using a library.
+
+## Reference
+
+You can read the following page to understand the PromptPex framework and how to use it in your codebase:
+
+- [Glossary](./glossary.md): A glossary of terms used in the PromptPex framework.
+- [Test Generation](./test-generation.md): The core of the framework, how to generate tests for prompts.
+- The prompts are `.prompty` files in the `/src/prompts` directory.
+- The **.prompty** format is documented in [./prompt-format.md].
+
+## Reference implementation
+
+The GenAIScript reference implementation is in the `/src/genaiscript` directory. PromptPex starts in `src/genaiscript/src/promptpex.mts`.
+
+It is implemented using [GenAIScript](https://microsoft.github.io/genaiscript/).
+
+**Following the patterns and habits of the the target framework/language you are generating**. The reference implementation is a good starting point but you should adapt it to the target framework/language you are generating.