Add runtime test of all the examples (#135)

* Add runtime test of all the examples

* Remove uninteresting example

* Create run-examples action

* Do not execute test_examples_run on build

Author: mandel

.github/workflows/build.yml

@@ -39,7 +39,7 @@ jobs:
     - name: pre-commit checks
       run: pre-commit run -a
     - name: run tests
-      run: py.test -v --capture=tee-sys tests
+      run: py.test -v --capture=tee-sys --ignore=tests/test_examples_run.py tests
 
   viewer:
     name: Build PDL live viewer

.github/workflows/run-examples.yml

@@ -0,0 +1,41 @@
+---
+name: Run examples
+
+on:
+  workflow_dispatch:
+
+jobs:
+  tests:
+    name: Exeution tests
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ['3.11', '3.12']
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Cache pip
+      uses: actions/cache@v4
+      with:
+        # This path is specific to Ubuntu
+        path: ${{ env.pythonLocation }}
+        # Look to see if there is a cache hit for the setup file
+        key: ${{ runner.os }}-pip-new3-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
+        restore-keys: |
+          ${{ runner.os }}-pip-new3
+          ${{ runner.os }}-new3
+    - name: Install dependencies
+      run: pip install --upgrade --upgrade-strategy eager .[all]
+    - name: pip list packages
+      run: pip list
+    - name: show pip dependencies
+      run: |
+        pip install pipdeptree
+        pipdeptree -fl
+    - name: run tests
+      run: py.test -v --capture=tee-sys tests/test_examples_run.py

examples/hello/test.pdl

@@ -111,7 +111,7 @@ defs:
                     Act: {"name": "Finish", "arguments": {"topic": "2.869047619047619"}}
       - call: react_inner
         args:
-          context: []
+          pdl_context: []
           examples: ${ examples }
           question: ${ question }
           model: ${ model }

examples/react/react_fun.pdl

@@ -111,7 +111,7 @@ defs:
                     Act: {"name": "Finish", "arguments": {"topic": "2.869047619047619"}}
       - call: react_inner
         args:
-          context: []
+          pdl_context: []
           examples: ${ examples }
           question: ${ question }
           model: ${ model }

examples/talk/react_fun.pdl

@@ -44,6 +44,8 @@ class InterpreterConfig(TypedDict, total=False):
     role: RoleType
     """Default role.
     """
+    cwd: Path
+    """Path considered as the current working directory for file reading."""
 
 
 def exec_program(
@@ -127,7 +129,7 @@ def exec_str(
 
 
 def exec_file(
-    prog: str,
+    prog: str | Path,
     config: Optional[InterpreterConfig] = None,
     scope: Optional[ScopeType] = None,
     output: Literal["result", "all"] = "result",
@@ -144,6 +146,10 @@ def exec_file(
         Return the final result.
     """
     program, loc = parse_file(prog)
+    if config is None:
+        config = InterpreterConfig()
+    if config.get("cwd") is None:
+        config["cwd"] = Path(prog).parent
     result = exec_program(program, config, scope, loc, output)
     return result
 
@@ -284,18 +290,21 @@ def main():
     else:
         batch = 1
 
-    config = InterpreterConfig(
-        yield_result=stream_result, yield_background=stream_background, batch=batch
-    )
     pdl_file = Path(args.pdl)
     if args.trace == "*_trace.json":
         trace_file = str(pdl_file.with_suffix("")) + "_trace.json"
     else:
         trace_file = args.trace
+    config = InterpreterConfig(
+        yield_result=stream_result,
+        yield_background=stream_background,
+        batch=batch,
+        cwd=pdl_file.parent,
+    )
     pdl_interpreter.generate(
         pdl_file,
         args.log,
-        InterpreterState(**config, cwd=pdl_file.parent),
+        InterpreterState(**config),
         initial_scope,
         trace_file,
     )

src/pdl/pdl.py

@@ -1337,9 +1337,21 @@ def process_input(
     read, block = process_expr_of(block, "read", scope, loc)
     if read is not None:
         file = state.cwd / read
-        with open(file, encoding="utf-8") as f:
-            s = f.read()
-            append_log(state, "Input from File: " + str(file), s)
+        try:
+            with open(file, encoding="utf-8") as f:
+                s = f.read()
+                append_log(state, "Input from File: " + str(file), s)
+        except Exception as exc:
+            if isinstance(exc, FileNotFoundError):
+                msg = f"file {str(file)} not found"
+            else:
+                msg = f"Fail to open file {str(file)}"
+            raise PDLRuntimeError(
+                message=msg,
+                loc=loc,
+                trace=ErrorBlock(msg=msg, location=loc, program=block),
+                fallback="",
+            ) from exc
     else:
         message = ""
         if block.message is not None:

src/pdl/pdl_interpreter.py

@@ -0,0 +1,39 @@
+Question: Noah charges $60 for a large painting and $30 for a small painting.
+Last month he sold eight large paintings and four small paintings.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 8 large paintings and 4 small paintings last month.
+He sold twice as many this month.
+8 large paintings x $60 = << 8*60= 480 >> 480
+4 small paintings x $30 = << 4*30= 120 >> 120
+So he sold << 480+120= 600 >> 600 paintings last month.
+Therefore he sold << 600*2= 1200 >> this month.
+The answer is $1200.
+
+Question: Noah charges $10 for a small vase and $30 for a large vases
+Last month he sold five large vases and three small vases.
+If he sold three times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 large vases and 3 small vases last month.
+He sold three times as many this month.
+5 large vases x $30 = << 5*30= 150 >> 150
+3 small vases x $10 = << 3*10= 30 >> 30
+So he sold << 150+30= 180 >> 180 vases last month.
+Therefore he sold << 180*3= 540 >> this month.
+The answer is $540.
+
+Question: Noah charges $10 for a large box and $5 for a small box.
+Last month he sold four large boxes and two small boxes.
+If he sold half as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 4 large boxes and 2 small boxes last month.
+He sold half as many this month.
+4 large boxes x $10 = << 4*10= 40 >> 40
+2 small boxes x $5 = << 2*5= 10 >> 10
+So he sold << 40+10= 50 >> 50 boxes last month.
+Therefore he sold << 50/2= 25.0 >> this month.
+The answer is 25
+

tests/results/examples/arith/Arith-simple.result

@@ -0,0 +1,71 @@
+Question: Noah charges $60 for a large painting and $30 for a small painting.
+Last month he sold eight large paintings and four small paintings.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 8 large paintings and 4 small paintings last month.
+He sold twice as many this month.
+8 large paintings x $60 = << 8*60= 480 >> 480
+4 small paintings x $30 = << 4*30= 120 >> 120
+So he sold << 480+120= 600 >> 600 paintings last month.
+Therefore he sold << 600*2= 1200 >> this month.
+The answer is $1200.
+
+Question: Noah charges $10 for a small vase and $30 for a large vases
+Last month he sold five large vases and three small vases.
+If he sold three times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 large vases and 3 small vases last month.
+He sold three times as many this month.
+5 large vases x $30 = << 5*30= 150 >> 150
+3 small vases x $10 = << 3*10= 30 >> 30
+So he sold << 150+30= 180 >> 180 vases last month.
+Therefore he sold << 180*3= 540 >> this month.
+The answer is $540.
+
+Question: 
+Noah charges $10 for a small vase and $30 for a large vase.
+Last month he sold five large vases and three small vases.
+If he sold three times as much this month, how much is his sales for this month?
+Answer: Let's think step by step.
+He sold 5 large vases and 3 small vases last month.
+He sold three times as much this month.
+5 large vases x $30 = << 5*30= 150 >> 150
+3 small vases x $10 = << 3*10 = 30 >>  30
+So he sold << 150+30 = 180 >> 180 vases last month.
+Therefore he sold << 180*3  = 540 >>  540 vases this month.
+The answer is 540
+
+
+Question: 
+Noah charges $10 for a small vase and $30 for a large vase.
+Last month he sold five large vases and three small vases.
+If he sold three times as much this month, how much is his sales for this month?
+Answer: Let's think step by step.
+He sold 5 large vases and 3 small vases last month.
+He sold three times as much this month.
+5 large vases x $30 = << 5*30 = 150 >>  150
+3 small vases x $10 = << 3*10   = 30 >>    30
+So he sold << 150+30    = 180 >>    180 vases last month.
+Therefore he sold << 180*3      = 540 >>     540 vases this month.
+The answer is 540
+
+
+Question: 
+Noah charges $10 for a small vase and $30 for a large vase.
+Last month he sold five large vases and three small vases.
+If he sold three times as much this month, how much is his sales for this month?
+Answer: Let's think step by step.
+ He sold 5 large vases and 3 small vases last month.
+He sold three times as much this month.
+5 large vases x $30 = << 5*30  = 150 >>    150
+3 small vases x $10 = << 3*10     = 30 >>       30
+So he sold << 150+30        = 180 >>          180 vases last month.
+Therefore he sold << 180*3             = 540 >>            540 vases this month.
+The answer is 540 
+
+
+Question
+
+

tests/results/examples/arith/arith.result

@@ -0,0 +1,3 @@
+What is APR?
+Annual Percentage Rate (APR) is the total cost of borrowing money, expressed as a percentage. It includes the interest rate and any additional fees or charges associated with the loan. APR is used to compare different loan offers and to understand the true cost of borrowing money over time.
+