Update run-examples tests (#139)

* Update run-examples tests

* Schedule the action to run every day at 1 am

Author: mandel

.github/workflows/run-examples.yml

@@ -2,6 +2,8 @@
 name: Run examples
 
 on:
+  schedule:
+    - cron:  '0 1 * * *'
   workflow_dispatch:
 
 jobs:
@@ -38,4 +40,8 @@ jobs:
         pip install pipdeptree
         pipdeptree -fl
     - name: run tests
+      env:
+        WATSONX_PROJECT_ID: ${{ secrets.WATSONX_PROJECT_ID }}
+        WATSONX_APIKEY: ${{ secrets.WATSONX_APIKEY }}
+        WATSONX_URL: ${{ secrets.WATSONX_URL }}
       run: py.test -v --capture=tee-sys tests/test_examples_run.py

tests/results/examples/demo/3-weather.result

@@ -0,0 +1,2 @@
+
+The weather in Yorktown Heights, New York, USA is currently partly cloudy with a temperature of 15.6°C (60.1°F). The wind speed is 10.1 mph (16.2 kph) from the NNW direction. The humidity is 40%, and the visibility is 16 km (9 miles). The UV index is 3.4, and there is no precipitation. The feels-like temperature is 15.6°C (60.1°F), and the wind chill is 13.7°C (56.6°F). The heat index is 14.4°C (58.0°F), and the dew point is 3.8°C (38.8°F). The pressure is 1017 mb (30.02 in).

tests/results/examples/react/wikipedia.result

@@ -22,8 +22,7 @@ Act:{"name": "Search", "arguments": {"topic": "discoverer of the Hudson River"}}
 The Hudson River runs through the Munsee, Lenape, Mohican, Mohawk, and Haudenosaunee homelands. Prior to European exploration, the river was known as the Mahicannittuk by the Mohicans, Ka'nón:no by the Mohawks, and Muhheakantuck by the Lenape. The river was subsequently named after Henry Hudson, an Englishman sailing for the Dutch East India Company who explored it in 1609, and after whom Hudson Bay in Canada is also named. It had previously been observed by Italian explorer Giovanni da Verrazzano sailing for King Francis I of France in 1524, as he became the first European known to have entered the Upper New York Bay, but he considered the river to be an estuary. The Dutch called the river the North River, and they called the present-day Delaware River the South River, which formed the spine of the Dutch colony of New Netherland. Settlements of the colony clustered around the Hudson, and its strategic importance as the gateway to the American interior led to years of competition between the English and the Dutch over control of the river and colony.
 During the 18th century, the river valley and its inhabitants were the subject and inspiration of Washington Irving, the first internationally acclaimed American author. In the nineteenth century, the area inspired the Hudson River School of landscape painting, an American pastoral style, as well as the concepts of environmentalism and wilderness. The Hudson River was also the eastern outlet for the Erie Canal, which, when completed in 1825, became an important transportation artery for the early 19th century United States.
 Pollution in the Hudson River increased in the 20th century, more acutely by mid-century, particularly with industrial contamination from polychlorinated biphenyls, also known by their acronym PCBs. Pollution control regulations, enforcement actions and restoration projects initiated in the latter 20th century have begun to improve water quality, and restoration work has continued in the 21st century.
-
- Tho: The discoverer of the Hudson River is Henry Hudson. I need to search Henry Hudson, find when he was born.
+Tho: The discoverer of the Hudson River is Henry Hudson. I need to search Henry Hudson, find when he was born.
 Act:{"name": "Search", "arguments": {"topic": "Henry Hudson"}}Obs: Henry Hudson (c. 1565 – disappeared 23 June 1611) was an English sea explorer and navigator during the early 17th century, best known for his explorations of present-day Canada and parts of the Northeastern United States.
 In 1607 and 1608, Hudson made two attempts on behalf of English merchants to find a rumoured Northeast Passage to Cathay via a route above the Arctic Circle. In 1609, he landed in North America on behalf of the Dutch East India Company and explored the region around the modern New York metropolitan area. Looking for a Northwest Passage to Asia on his ship Halve Maen ("Half Moon"), he sailed up the Hudson River, which was later named after him, and thereby laid the foundation for Dutch colonization of the region. His contributions to the exploration of the New World were significant and lasting. His voyages helped to establish European contact with the native peoples of North America and contributed to the development of trade and commerce. 
 On his final expedition, while still searching for the Northwest Passage, Hudson became the first European to see Hudson Strait and the immense Hudson Bay. In 1611, after wintering on the shore of James Bay, Hudson wanted to press on to the west, but most of his crew mutinied. The mutineers cast Hudson, his son, and six others adrift; the Hudsons and their companions were never seen again.

tests/results/examples/talk/9-react.result

@@ -22,8 +22,7 @@ Act:{"name": "Search", "arguments": {"topic": "discoverer of the Hudson River"}}
 The Hudson River runs through the Munsee, Lenape, Mohican, Mohawk, and Haudenosaunee homelands. Prior to European exploration, the river was known as the Mahicannittuk by the Mohicans, Ka'nón:no by the Mohawks, and Muhheakantuck by the Lenape. The river was subsequently named after Henry Hudson, an Englishman sailing for the Dutch East India Company who explored it in 1609, and after whom Hudson Bay in Canada is also named. It had previously been observed by Italian explorer Giovanni da Verrazzano sailing for King Francis I of France in 1524, as he became the first European known to have entered the Upper New York Bay, but he considered the river to be an estuary. The Dutch called the river the North River, and they called the present-day Delaware River the South River, which formed the spine of the Dutch colony of New Netherland. Settlements of the colony clustered around the Hudson, and its strategic importance as the gateway to the American interior led to years of competition between the English and the Dutch over control of the river and colony.
 During the 18th century, the river valley and its inhabitants were the subject and inspiration of Washington Irving, the first internationally acclaimed American author. In the nineteenth century, the area inspired the Hudson River School of landscape painting, an American pastoral style, as well as the concepts of environmentalism and wilderness. The Hudson River was also the eastern outlet for the Erie Canal, which, when completed in 1825, became an important transportation artery for the early 19th century United States.
 Pollution in the Hudson River increased in the 20th century, more acutely by mid-century, particularly with industrial contamination from polychlorinated biphenyls, also known by their acronym PCBs. Pollution control regulations, enforcement actions and restoration projects initiated in the latter 20th century have begun to improve water quality, and restoration work has continued in the 21st century.
-
- Tho: The discoverer of the Hudson River is Henry Hudson. I need to search Henry Hudson, find when he was born.
+Tho: The discoverer of the Hudson River is Henry Hudson. I need to search Henry Hudson, find when he was born.
 Act:{"name": "Search", "arguments": {"topic": "Henry Hudson"}}Obs: Henry Hudson (c. 1565 – disappeared 23 June 1611) was an English sea explorer and navigator during the early 17th century, best known for his explorations of present-day Canada and parts of the Northeastern United States.
 In 1607 and 1608, Hudson made two attempts on behalf of English merchants to find a rumoured Northeast Passage to Cathay via a route above the Arctic Circle. In 1609, he landed in North America on behalf of the Dutch East India Company and explored the region around the modern New York metropolitan area. Looking for a Northwest Passage to Asia on his ship Halve Maen ("Half Moon"), he sailed up the Hudson River, which was later named after him, and thereby laid the foundation for Dutch colonization of the region. His contributions to the exploration of the New World were significant and lasting. His voyages helped to establish European contact with the native peoples of North America and contributed to the development of trade and commerce. 
 On his final expedition, while still searching for the Northwest Passage, Hudson became the first European to see Hudson Strait and the immense Hudson Bay. In 1611, after wintering on the shore of James Bay, Hudson wanted to press on to the west, but most of his crew mutinied. The mutineers cast Hudson, his son, and six others adrift; the Hudsons and their companions were never seen again.

tests/results/examples/weather/weather.result

@@ -0,0 +1,2 @@
+
+The temperature is 15.6 degrees Celsius and the sky is partly cloudy.

tests/test_examples_run.py

@@ -15,12 +15,12 @@
     str(name)
     for name in [
         pathlib.Path("examples") / "demo" / "2-teacher.pdl",  # TODO: check why
+        pathlib.Path("examples") / "talk" / "8-tools.pdl",  # TODO: check why
         pathlib.Path("examples") / "talk" / "11-sdg.pdl",  # TODO: check why
         pathlib.Path("examples") / "teacher" / "teacher.pdl",  # TODO: check why
-        pathlib.Path("examples") / "demo" / "3-weather.pdl",
+        pathlib.Path("examples") / "tools" / "calc.pdl",  # TODO: check why
         pathlib.Path("examples") / "tutorial" / "calling_apis.pdl",
         pathlib.Path("examples") / "cldk" / "cldk-assistant.pdl",
-        pathlib.Path("examples") / "weather" / "weather.pdl",
         pathlib.Path("examples") / "talk" / "10-multi-agent.pdl",
         pathlib.Path("examples") / "gsm8k" / "gsmhard-bugs.pdl",
         pathlib.Path("examples") / "gsm8k" / "math-base.pdl",
@@ -38,11 +38,16 @@
         pathlib.Path("examples") / "granite" / "multi_round_chat.pdl",
         pathlib.Path("examples") / "granite" / "single_round_chat.pdl",
         pathlib.Path("examples") / "joke" / "Joke.pdl",
-        # pathlib.Path("examples") / "react" / "multi-agent.pdl",
+        pathlib.Path("examples") / "react" / "multi-agent.pdl",
         pathlib.Path("examples") / "talk" / "11-sdg.pdl",
         pathlib.Path("examples") / "talk" / "7-chatbot-roles.pdl",
+        pathlib.Path("examples") / "talk" / "8-tools.pdl",
         pathlib.Path("examples") / "teacher" / "teacher.pdl",
+        pathlib.Path("examples") / "tools" / "calc.pdl",
         pathlib.Path("examples") / "tutorial" / "include.pdl",
+        pathlib.Path("examples") / "hello" / "hello-roles-array.pdl",
+        pathlib.Path("examples") / "weather" / "weather.pdl",
+        pathlib.Path("examples") / "demo" / "3-weather.pdl",
     ]
 }
 
@@ -83,6 +88,14 @@ class InputsType:
         pathlib.Path("examples")
         / "hello"
         / "hello-data.pdl": InputsType(scope={"something": "ABC"}),
+        pathlib.Path("examples")
+        / "weather"
+        / "weather.pdl": InputsType(stdin="What is the weather in Yorktown Heights?\n"),
+        pathlib.Path("examples")
+        / "demo"
+        / "3-weather.pdl": InputsType(
+            stdin="What is the weather in Yorktown Heights?\n"
+        ),
     }.items()
 }
 
@@ -134,9 +147,6 @@ def test_valid_programs(capsys, monkeypatch) -> None:
     wrong_results = {}
     for pdl_file_name in pathlib.Path(".").glob("**/*.pdl"):
         scope: ScopeType = {}
-        print(str(pdl_file_name))
-        # if "cldk" in str(pdl_file_name):
-        #     pass
         if str(pdl_file_name) in TO_SKIP:
             continue
         if str(pdl_file_name) in TESTS_WITH_INPUT:
@@ -167,14 +177,16 @@ def test_valid_programs(capsys, monkeypatch) -> None:
                 result_dir_name / result_file_name, "r", encoding="utf-8"
             ) as result_file:
                 expected_result = str(result_file.read())
-            if str(result) != expected_result:
+            if str(result).strip() != expected_result.strip():
                 wrong_results[str(pdl_file_name)] = {
                     "actual": str(result),
                     "expected": str(expected_result),
                 }
         except PDLParseError:
             actual_parse_error |= {str(pdl_file_name)}
-        except PDLRuntimeError:
+        except PDLRuntimeError as exc:
+            if str(pdl_file_name) not in set(str(p) for p in EXPECTED_RUNTIME_ERROR):
+                print(exc)  # unexpected error: breakpoint
             actual_runtime_error |= {str(pdl_file_name)}
     # Parse errors
     expected_parse_error = set(str(p) for p in EXPECTED_PARSE_ERROR)