Update results

Signed-off-by: Jing Chen <[email protected]>

Author: jgchn

docs/contrib.md

@@ -101,7 +101,34 @@ pytest --capture=tee-sys -rfE -s tests/test_examples_run.py --disable-pytest-war
 
 A slight variation in the Python version and OS environment can cause a different LLM response, thus Run Examples might fail because it uses exact string matching for PDL outputs.
 
-When you open a pull request (PR) against the `main` branch, a series of status checks will be executed. Specificially, three Run Examples test will be initiated against the files you have added and modified. If there's any variation, you should manually examine the results produced in the Github Actions environment, then copy and paste the results to a new file, and push another commit to your PR so the CI can pass. Be aware of whitespaces in between sentences, as the CI does an exact string matching. 
+When you open a pull request (PR) against the `main` branch, a series of status checks will be executed. Specificially, three Run Examples test will be initiated against the PDL files you have added and modified as part of the PR. If there's any variation, you should manually examine the results produced in the Github Actions environment, then copy and paste the results to a new file, and push another commit to your PR so the CI can pass. Be aware of whitespaces in between sentences. 
+
+Below is what the CI might look like if the results differ. If the results look reasonable, you should
+1. copy and paste the result for that file 
+2. create a new file in `tests/results` in the format of `tests/results/<path/to/file>.<i>.result` where `<i>` should be incremented from the highest number of `<i>.result` in the folder
+
+If the test fails, look for something similar like the following:
+```
+tests/test_examples_run.py 
+============================================================================
+File that produced wrong result: examples/demo/1-hello.pdl
+Actual result (copy everything below this line):
+✂️ ------------------------------------------------------------
+Hello
+Hello
+-------------------------------------------------------------
+
+============================================================================
+File that produced wrong result: examples/demo/2-model-chaining.pdl
+Actual result (copy everything below this line):
+✂️ ------------------------------------------------------------
+Hello
+Hello
+Did you just say Hello?
+Yes, I did. It's a common greeting, similar to how humans might respond when they first interact with an artificial intelligence like me. How can I assist you today?
+-------------------------------------------------------------
+F
+```
 
 Note: Your PR should always set `update_results: false` before merging, though the CI pipeline will not attempt to write the results to your PR even if you forget to set this field to `false`. 
 

tests/results/examples/react/demo.6.result

@@ -0,0 +1,23 @@
+How many years ago was the discoverer of the Hudson River born? Keep in mind we are in 2025. When searching for a birthday for a person, simply ask for the name of that person.
+Thought: I need to find out who discovered the Hudson River and then calculate how many years ago their birthdate was.
+Action:
+[{"name": "Search", "arguments": {"topic": "Henry Hudson"}}]
+Observation: Henry Hudson (c. 1565 – disappeared 23 June 1611) was an English sea explorer and navigator during the early 17th century, best known for his explorations of present-day Canada and parts of the Northeastern United States.
+In 1607 and 1608, Hudson made two attempts on behalf of English merchants to find a rumoured Northeast Passage to Cathay via a route above the Arctic Circle. In 1609, he landed in North America on behalf of the Dutch East India Company and explored the region around the modern New York metropolitan area. Looking for a Northwest Passage to Asia on his ship Halve Maen ("Half Moon"), he sailed up the Hudson River, which was later named after him, and thereby laid the foundation for Dutch colonization of the region. His contributions to the exploration of the New World were significant and lasting. His voyages helped to establish European contact with the native peoples of North America and contributed to the development of trade and commerce. 
+On his final expedition, while still searching for the Northwest Passage, Hudson became the first European to see Hudson Strait and the immense Hudson Bay. In 1611, after wintering on the shore of James Bay, Hudson wanted to press on to the west, but most of his crew mutinied. The mutineers cast Hudson, his son, and six others adrift; what then happened to the Hudsons and their companions is unknown.
+
+
+Thought: Henry Hudson discovered the Hudson River in 1609. I need to find out when he was born.
+Action:
+[{"name": "Search", "arguments": {"topic": "Henry Hudson birthdate"}}]
+Observation: Page id "henry hudson birthday" does not match any pages. Try another id!
+Thought: I couldn't find a specific birthdate for Henry Hudson, but it's estimated he was born around 1565.
+
+Action:
+[{"name": "Calc", "arguments": {"expr": "2025 - 1565"}}]
+Observation: 460
+Thought: Henry Hudson was born around 1565, so in 2025, it has been approximately 460 years.
+
+Answer: It has been about 460 years since the birth of Henry Hudson.Action:
+[{"name": "Finish", "arguments": {"topic": "460"}}]
+

tests/test_examples_run.py

@@ -146,7 +146,7 @@ def __init__(self, monkeypatch: MonkeyPatch) -> None:
             self.expected_runtime_error = content["expected_runtime_error"]
 
             for filename, inputs_type in content["with_inputs"].items():
-                stdin, scope = None, None 
+                stdin, scope = None, None
                 if "stdin" in inputs_type:
                     stdin = inputs_type["stdin"]
                 if "scope" in inputs_type:
@@ -320,7 +320,9 @@ def test_example_runs(capsys: CaptureFixture[str], monkeypatch: MonkeyPatch) ->
             "\n============================================================================"
         )
         print(f"File that produced wrong result: {file}")
-        print(f"Actual result (copy everything below this line):\n✂️ ------------------------------------------------------------\n{actual}\n-------------------------------------------------------------")
+        print(
+            f"Actual result (copy everything below this line):\n✂️ ------------------------------------------------------------\n{actual}\n-------------------------------------------------------------"
+        )
 
     assert (
         len(background.failed_results.unexpected_parse_error) == 0