Merge pull request #125 from anticomputer/anticomputer/jinja-templating

Migrate template system to Jinja2

Author: kevinbackhouse

README.md

@@ -174,15 +174,15 @@ Every YAML files used by the Seclab Taskflow Agent must include a header like th
 
 ```yaml
 seclab-taskflow-agent:
-  version: 1
+  version: "1.0"
   filetype: taskflow
 ```
 
-The `version` number in the header should always be 1. It means that the
+The `version` number in the header is currently 1. It means that the
 file uses version 1 of the seclab-taskflow-agent syntax. If we ever need
 to make a major change to the syntax, then we'll update the version number.
 This will hopefully enable us to make changes without breaking backwards
-compatibility.
+compatibility. Version can be specified as an integer, float, or string.
 
 The `filetype` determines whether the file defines a personality, toolbox, etc.
 This means that different types of files can be stored in the same directory.
@@ -294,10 +294,10 @@ server_params:
   url: https://api.githubcopilot.com/mcp/
   #See https://github.com/github/github-mcp-server/blob/main/docs/remote-server.md
   headers:
-    Authorization: "{{ env GITHUB_AUTH_HEADER }}"
+    Authorization: "{{ env('GITHUB_AUTH_HEADER') }}"
   optional_headers:
-    X-MCP-Toolsets: "{{ env GITHUB_MCP_TOOLSETS }}"
-    X-MCP-Readonly: "{{ env GITHUB_MCP_READONLY }}"
+    X-MCP-Toolsets: "{{ env('GITHUB_MCP_TOOLSETS') }}"
+    X-MCP-Readonly: "{{ env('GITHUB_MCP_READONLY') }}"
 ```
 
 You can force certain tools within a `toolbox` to require user confirmation to run. This can be helpful if a tool may perform irreversible actions and should require user approval prior to its use. This is done by including the name of the tool (function) in the MCP server in the `confirm` section:
@@ -345,7 +345,7 @@ taskflow:
         Finally, why are apples and oranges healthy to eat?
 
       # taskflows can set temporary environment variables, these support the general
-      # "{{ env FROM_EXISTING_ENVIRONMENT }" pattern we use elsewhere as well
+      # "{{ env('FROM_EXISTING_ENVIRONMENT') }}" pattern we use elsewhere as well
       # these environment variables can then be made available to any stdio mcp server
       # through its respective yaml configuration, see memcache.yaml for an example
       # you can use these to override top-level environment variables on a per-task basis
@@ -492,12 +492,12 @@ Files of types `taskflow` and `toolbox` allow environment variables to be passed
 server_params:
   ...
   env:
-    CODEQL_DBS_BASE_PATH: "{{ env CODEQL_DBS_BASE_PATH }}"
+    CODEQL_DBS_BASE_PATH: "{{ env('CODEQL_DBS_BASE_PATH') }}"
     # prevent git repo operations on gh codeql executions
     GH_NO_UPDATE_NOTIFIER: "disable"
 ```
 
-For `toolbox`, `env` can be used inside `server_params`. A template of the form `{{ env ENV_VARIABLE_NAME }}` can be used to pass values of the environment variable from the current process to the MCP server. So in the above, the MCP server is run with `GH_NO_UPDATE_NOTIFIER=disable` and passes the value of `CODEQL_DBS_BASE_PATH` from the current process to the MCP server. The templated paramater `{{ env CODEQL_DBS_BASE_PATH }}` is replaced by the value of the environment variable `CODEQL_DBS_BASE_PATH` in the current process.
+For `toolbox`, `env` can be used inside `server_params`. A template of the form `{{ env('ENV_VARIABLE_NAME') }}` can be used to pass values of the environment variable from the current process to the MCP server. So in the above, the MCP server is run with `GH_NO_UPDATE_NOTIFIER=disable` and passes the value of `CODEQL_DBS_BASE_PATH` from the current process to the MCP server. The templated parameter `{{ env('CODEQL_DBS_BASE_PATH') }}` is replaced by the value of the environment variable `CODEQL_DBS_BASE_PATH` in the current process.
 
 Similarly, environment variables can be passed to a `task` in a `taskflow`:
 
@@ -514,9 +514,9 @@ taskflow:
         MEMCACHE_BACKEND: "dictionary_file"
 ```
 
-This overwrites the environment variables `MEMCACHE_STATE_DIR` and `MEMCACHE_BACKEND` for the task only. A template `{{ env ENV_VARIABLE_NAME }}` can also be used.
+This overwrites the environment variables `MEMCACHE_STATE_DIR` and `MEMCACHE_BACKEND` for the task only. A template `{{ env('ENV_VARIABLE_NAME') }}` can also be used.
 
-Note that when using the template `{{ env ENV_VARIABLE_NAME }}`, `ENV_VARIABLE_NAME` must be the name of an environment variable in the current process.
+Note that when using the template `{{ env('ENV_VARIABLE_NAME') }}`, `ENV_VARIABLE_NAME` must be the name of an environment variable in the current process.
 
 ## Import paths
 

doc/GRAMMAR.md

@@ -133,10 +133,10 @@ Often we may want to iterate through the same tasks with different inputs. For e
     agents:
       - seclab_taskflow_agent.personalities.c_auditer
     user_prompt: |
-      The function has name {{ RESULT_name }} and body {{ RESULT_body }} analyze the function.
+      The function has name {{ result.name }} and body {{ result.body }} analyze the function.
 ```
 
-In the above, the first task fetches functions in the code base and creates a json list object, with each entry having a `name` and `body` field. In the next task, `repeat_prompt` is set to true, meaning that a task is created for each individual object in the list and the object fields are referenced in the templated prompt using `{{ RESULT_<fieldname> }}`. In other words, `{{ RESULT_name }}` in the prompt is replaced with the value of the `name` field of the object etc. For example, if the list of functions fetched from the first task is:
+In the above, the first task fetches functions in the code base and creates a json list object, with each entry having a `name` and `body` field. In the next task, `repeat_prompt` is set to true, meaning that a task is created for each individual object in the list and the object fields are referenced in the templated prompt using `{{ result.fieldname }}`. In other words, `{{ result.name }}` in the prompt is replaced with the value of the `name` field of the object etc. For example, if the list of functions fetched from the first task is:
 
 ```javascript
 [{'name' : foo, 'body' : foo(){return 1;}}, {'name' : bar, 'body' : bar(a) {return a + 1;}}]
@@ -152,7 +152,7 @@ etc.
 
 Note that when using `repeat_prompt`, the last tool call result of the previous task is used as the iterable. It is recommended to keep the task that creates the iterable short and simple (e.g. just make one tool call to fetch a list of results) to avoid wrong results being passed to the repeat prompt.
 
-The iterable can also contain a list of primitives like string or number, in which case, the template `{{ RESULT }}` can be used in the `repeat_prompt` prompt to parse the results instead:
+The iterable can also contain a list of primitives like string or number, in which case, the template `{{ result }}` can be used in the `repeat_prompt` prompt to parse the results instead:
 
 ```yaml
   - task:
@@ -173,7 +173,7 @@ The iterable can also contain a list of primitives like string or number, in whi
       agents:
         - seclab_taskflow_agent.personalities.assistant
       user_prompt: |
-        What is the integer value of {{ RESULT }}?
+        What is the integer value of {{ result }}?
 ```
 
 Repeat prompt can be run in parallel by setting the `async` field to `true`:
@@ -185,7 +185,7 @@ Repeat prompt can be run in parallel by setting the `async` field to `true`:
     agents:
       - seclab_taskflow_agent.personalities.c_auditer
     user_prompt: |
-      The function has name {{ RESULT_name }} and body {{ RESULT_body }} analyze the function.
+      The function has name {{ result.name }} and body {{ result.body }} analyze the function.
 ```
 
 An optional limit can be set to limit the number of asynchronous tasks via `async_limit`. If not set, the default value (5) is used.
@@ -198,7 +198,7 @@ An optional limit can be set to limit the number of asynchronous tasks via `asyn
     agents:
       - seclab_taskflow_agent.personalities.c_auditer
     user_prompt: |
-      The function has name {{ RESULT_name }} and body {{ RESULT_body }} analyze the function.
+      The function has name {{ result.name }} and body {{ result.body }} analyze the function.
 ```
 
 Both `async` and `async_limit` have no effect when used outside of a `repeat_prompt`.
@@ -211,7 +211,7 @@ At the moment, we do not support nested `repeat_prompt`. So the following is not
     agents:
       - seclab_taskflow_agent.personalities.c_auditer
     user_prompt: |
-      The function has name {{ RESULT_name }} and body {{ RESULT_body }} analyze the function.
+      The function has name {{ result.name }} and body {{ result.body }} analyze the function.
   - task:
     repeat_prompt: true
     ...
@@ -233,7 +233,7 @@ For example:
       agents:
         - seclab_taskflow_agent.personalities.assistant
       user_prompt: |
-        What kind of fruit is {{ RESULT }}?
+        What kind of fruit is {{ result }}?
 ```
 
 The string `["apple", "banana", "orange"]` is then passed directly to the next task.
@@ -349,7 +349,7 @@ taskflow:
       agents:
         - examples.personalities.fruit_expert
       user_prompt: |
-        Tell me more about {{ GLOBALS_fruit }}.
+        Tell me more about {{ globals.fruit }}.
 ```
 
 Global variables can also be set or overridden from the command line using the `-g` or `--global` flag:
@@ -422,10 +422,10 @@ A reusable taskflow can also have a templated prompt that takes inputs from its
       agents:
         - examples.personalities.fruit_expert
       user_prompt: |
-        Tell me more about {{ INPUTS_fruit }}.
+        Tell me more about {{ inputs.fruit }}.
 ```
 
-In this case, the template parameter `{{ INPUTS_fruit }}` is replaced by the value of `fruit` from the `inputs` of the user, which is apples in this case:
+In this case, the template parameter `{{ inputs.fruit }}` is replaced by the value of `fruit` from the `inputs` of the user, which is apples in this case:
 
 ```yaml
   - task:
@@ -437,9 +437,9 @@ In this case, the template parameter `{{ INPUTS_fruit }}` is replaced by the val
 
 ### Reusable Prompts
 
-Reusable prompts are defined in files of `filetype` `prompts`. These are like macros that get replaced when a templated parameter of the form `{{ PROMPTS_<import-path> }}` is encountered.
+Reusable prompts are defined in files of `filetype` `prompts`. These are like macros that get included using Jinja2's `{% include %}` directive.
 
-Tasks can incorporate templated prompts which are then replaced by the actual prompt. For example:
+Tasks can incorporate reusable prompts using the include directive. For example:
 
 Example:
 
@@ -449,8 +449,8 @@ Example:
         - examples.personalities.fruit_expert
       user_prompt: |
         Tell me more about apples.
-        
-        {{ PROMPTS_examples.prompts.example_prompt }}
+
+        {% include 'examples.prompts.example_prompt' %}
 ```
 and `examples.prompts.example_prompt` is the following: