Benchmark template rendering times

A concern is that rendering the template this way could be much slower,
assuming Jinja2 makes extensive use of caching. We thus write a
comparative benchmark between using Jinja2 templates or simply
concatenating strings.

Author: rlouf

docs/reference/template.md

@@ -25,6 +25,11 @@ allows to easily compose complex prompts.
     Prompt functions are opinionated when it comes to prompt rendering. These opinions are meant to avoid common prompting errors, but can have unintended consequences if you are doing something unusual. We advise to always print the prompt before using it. You can also [read the
     reference](#formatting-conventions) section if you want to know more.
 
+
+!!! note "Performance"
+
+    Prompt templates introduce some overhead compared to standard Python functions, although the rendering time is still very reasonable. In the unlikely scenario where rendering templates are a bottleneck you can replace them with functions that use standard string manipulation.
+
 ## Your first prompt
 
 The following snippet showcases a very simple prompt. The variables between

pyproject.toml

@@ -20,7 +20,7 @@ packages = ["prompts"]
 write_to = "prompts/_version.py"
 
 [project.optional-dependencies]
-test = ["pre-commit", "pytest"]
+test = ["pre-commit", "pytest", "pytest-benchmark"]
 docs = [
     "mkdocs",
     "mkdocs-material",

tests/test_templates.py

@@ -1,3 +1,6 @@
+import random
+import string
+
 import pytest
 
 import prompts
@@ -192,3 +195,42 @@ def simple_prompt_name(query: str):
     assert simple_prompt("test") == "test"
     assert simple_prompt["gpt2"]("test") == "test"
     assert simple_prompt["provider/name"]("test") == "name: test"
+
+
+def test_benchmark_template_render(benchmark):
+
+    @prompts.template
+    def test_tpl(var0, var1):
+        prompt = var0
+        return prompt + """{{var1}} test"""
+
+    def setup():
+        """We generate random strings to make sure we don't hit any potential cache."""
+        length = 10
+        var0 = "".join(
+            random.choice(string.ascii_uppercase + string.digits) for _ in range(length)
+        )
+        var1 = "".join(
+            random.choice(string.ascii_uppercase + string.digits) for _ in range(length)
+        )
+        return (var0, var1), {}
+
+    benchmark.pedantic(test_tpl, setup=setup, rounds=500)
+
+
+def test_benchmark_template_function(benchmark):
+
+    def test_tpl(var0, var1):
+        return var0 + f"{var1} test"
+
+    def setup():
+        length = 10
+        var0 = "".join(
+            random.choice(string.ascii_uppercase + string.digits) for _ in range(length)
+        )
+        var1 = "".join(
+            random.choice(string.ascii_uppercase + string.digits) for _ in range(length)
+        )
+        return (var0, var1), {}
+
+    benchmark.pedantic(test_tpl, setup=setup, rounds=500)