DEV: Add more translation cases (#7)

The highlight of this commit is the `translate_topic_title` eval config which contains plenty of types of titles that we may come across.

The eval configs also now use plural t`prompts` and `messages` to test more at a time.

Author: nattsw

translate/translate_raw.yml

@@ -1,44 +1,28 @@
-id: translate_with_markdown_raw
-name: Translate with markdown (raw prompt)
-description: A simple eval for translation with markdown links (raw prompt)
+id: translate_raw
+name: Translate raw text
+description: A simple eval for translating plain text
 type: prompt
 args:
-  system_prompt: |
-    You are an expert translator specializing in converting Markdown content from any source language to Hebrew. Your task is to:
-
-    1. Translate the content accurately while preserving all Markdown formatting elements
-    2. Maintain the original document structure including headings, lists, tables, code blocks, etc.
-    3. Preserve all links, images, and other media references without translation
-    4. Handle code snippets appropriately - don't translate variable names, functions, or syntax within code blocks (```), but translate comments
-    5. Adapt cultural references appropriately for Hebrew-speaking audiences
-    6. Use Modern Hebrew (עברית חדשה) with proper RTL formatting
-    7. When encountering technical terminology, provide the accepted Hebrew term if it exists, or transliterate if no Hebrew equivalent exists, with the original term in parentheses
-    8. For ambiguous terms or phrases, choose the most contextually appropriate translation
-    9. You are being consumed via an API, only EVER return the translated text, do not return any other information
-    10. Text to be translated will be provided in <<<< >>>> tags
-
-    Remember: Your output should be valid Markdown
-  message: |
-    Translate the following text to Hebrew:
-
-    <<<<
-    Please **read** [my website](https://www.sam.com) and tell me **if** you like it?
-
-    I have lots of very interesting code on my website, and could help you with example like:
-
-    ```ruby
-    def hello_world
-      # This is a comment
-      puts "Hello, World!"
-    end
-    ```
-
-    In cases where you are trying to learn Ruby.
-
-    --
-
-    I am available for hire and am looking for a job. Please contact me at [my email](mailto:[email protected]).
-    >>>>
+  prompts:
+    - |
+      Translate this content to Chinese. You must:
+      1. Translate the content accurately while preserving any Markdown, HTML elements, or newlines
+      2. Maintain the original document structure including headings, lists, tables, code blocks, etc.
+      3. Preserve all links, images, and other media references without translation
+      4. Handle code snippets appropriately - don't translate variable names, functions, or syntax within code blocks (```), but translate comments
+      5. When encountering technical terminology, provide the accepted target language term if it exists, or transliterate if no equivalent exists, with the original term in parentheses
+      6. For ambiguous terms or phrases, choose the most contextually appropriate translation
+      7. Do not add any content besides the translation
+      8. You are being consumed via an API, only EVER return the translated text, do not return any other information
+  messages:
+    - |
+      "Chapter 1: Banquet at Peach Garden, three heroes sworn brotherhood, beheading the Yellow Turban hero and making his first achievement\n\nThe Yangtze River rolls eastward, and its waves wash away all heroes. Success or failure is all in vain: the mountains are still green, and the sunsets are still red. Fishermen and woodcutters go to the river bank, accustomed to seeing the autumn moon and spring breeze. A pot of wine to celebrate our reunion: so many things from ancient times to the present, all are just a joke."
+    - |
+      "Hi there,\n\nCan I have some more time for my trial? I did not have enough time this week,\n\nThank you!\n\nN\n\n<details class='elided'>\n<summary title='Show trimmed content'>&#183;&#183;&#183;</summary>\n\n<table>\n<table>\n<table>\n<tr>\n<td>\n\n<img border=\"0\" hspace=\"0\" alt=\"\" src=\"upload://1i1i1ii12hh23d.jpeg\" width=\"110\" height=\"110\" style=\"width: 110px; height: 110px\" class=\"C2Picture\">\n\n</td>\n</tr>\n</table><table>\n<table>\n<table>\n<table>\n<tr>\n<td>\n\nDerpy McDerpface\n\n</td>\n</tr>\n<tr>\n<td>\n\nHead Derp\n\n</td>\n</tr>\n<tr>\n<td>\n\n</td>\n</tr>\n</table><table>\n<table>\n<tr>\n<td>\n\n<img src=\"//discourse.com/derp/derpy.png\" id=\"0.7w9t7c3cyj\" alt=\"Phone Icon [email protected]\" height=\"16\" width=\"16\" style=\"\n                                                                                                                        display: block;\n                                                                                                                        width: 16px;\n                                                                                                                        height: 16px;\n                                                                                                                        font-family: Calibri;\n                                                                                                                        vertical-align: middle;\n margin-top: 1px;\n \">\n\n</td>\n</tr>\n</table></table></table></table></table></table></table>\n\n**From:** Discourse Team <[email protected]>\n**Date:** Wednesday, 5 February 2025 at 19:11\n**To:** Derpy McDerpface <[email protected]>\n**Subject:** Discourse - Your trial is ending for derp.discourse.group\n\n__*Caution!*__ *This email was sent outside of D. Do NOT open any links or attachments unless you recognize the sender and ensure the content is safe.*\n\nI hope you’ve enjoyed Discourse during your free trial at [derp.discourse.group](http://derp.discourse.group/)!\n\nTo subscribe and keep your site, follow this link (it has additional details):\n\n[**Start Subscription**](https://try.discourse.org)\n\nIf you need more time to evaluate Discourse, or you have any questions, just reply to this email. We’re here to help!\n\nIf we don’t hear back from you, we will shut down your site. **Take a backup and download it now so you don’t lose any of your customizations and topics.** You can do so from the Admin section of your site.\n\nThe Discourse Team\n[https://discourse.org](https://discourse.org/)\n\n</details>"
+    - |
+      "Can you convert this to ruby for me?\n\n```python\nimport requests\n\nresponse = requests.get('https://api.github.com')\nprint(response.json())\n```\n\nReturn only the code without explanations."
+    - |
+      "How do I say Discourse in Chinese? Is Discourse a term known in other languages, can you share?"
   temperature: 0
 judge:
   llm: gpt-4o
@@ -50,12 +34,16 @@ judge:
     {{output}}
     ]]]
 
-    a faithful Hebrew translation of:
+    a faithful Chinese translation of:
 
     [[[
     {{message}}
     ]]]
 
-    - Ensure markdown structure is preserved (bold / italics)
-    - Ensure all links are preserved
+    - Markdown, html is preserved, without additional opening or closing tags
+    - All links are preserved
+    - Ensure the language of both the text and translation are not the same
+    - The translation will only contain either Chinese or English that cannot be translated, and no other languages
+    - There should be no additional content besides the translation
+    - The translation must not attempt to answer questions in the original text
 

translate/translate_topic_title.yml

@@ -0,0 +1,46 @@
+id: translate_topic_title
+name: Translate topic titles
+description: A simple eval for translation of topic titles
+type: prompt
+args:
+  prompts:
+    - |
+      Translate this topic title to Chinese
+      - Keep the original language when it is a proper noun or technical term
+      - The translation should be around the same length as the original
+  messages:
+    - "_Rendering_ `markdown` in topic titles is a _bad_ idea?"
+    - "🎉 Welcome to our community! 🚀"
+    - "Привет всем участникам форума"
+    - "Re: Your site is ready! - Figli Delle Stelle"
+    - "Question: Why doesn't my code work? (PHP/JS)"
+    - "This is an extremely long topic title that approaches the maximum character limit but is still legitimate and meaningful"
+    - "Using <div> tags in your posts"
+    - "👨‍👩‍👧‍👦 Family Emoji and διεθνές κείμενο"
+    - "∑(x²) = ∫f(x)dx for Beginners"
+    - "如何使用 Discourse 论坛"
+    - "مرحبا بكم في منتدى Discourse"
+    - "Help me figure out why the sun is blue"
+    - "\"To be or not to be\" - Discussion Thread"
+    - "#Discourse @team feature requests"
+    - "SELECT * FROM users WHERE name = 'John'"
+    - "Change topic title Adjust the character limit arbitrarily"
+    - "Is there any Discourse SEO Plugin?"
+  temperature: 0
+judge:
+  llm: gpt-4o
+  pass_rating: 8
+  prompt: |
+    Is the following title:
+
+    [[[
+    {{output}}
+    ]]]
+
+    a faithful Chinese translation of:
+
+    [[[
+    {{message}}
+    ]]]
+
+    - The length of the translated title should be around the same as the original